Cloud chat assistants are convenient, but they come with trade-offs: your queries leave your machine, you depend on someone else’s uptime and pricing, and the model’s behaviour can change under you without warning. For research, sensitive data, or simply full control, running a model locally is an appealing alternative. The good news is that modern quantization has made this accessible on modest consumer hardware. A capable 7–8 billion parameter model now fits comfortably on an 8 GB laptop GPU. This tutorial walks through the entire process end to end, using an NVIDIA Blackwell card (RTX 5060, 8 GB) as the worked example — though the approach applies to any recent NVIDIA GPU.

Everything starts with a clean, isolated environment. Mixing deep-learning dependencies into your base Python installation is a recipe for version conflicts, so we create a dedicated Conda environment for this project alone. If you have followed our earlier Anaconda setup guide, this will feel familiar.

Open the Anaconda Prompt and create a fresh environment:

The single most important, and most overlooked, step is installing the correct build of PyTorch for your specific GPU. This is where most local-LLM attempts fail silently. NVIDIA GPUs each have a “compute capability” (an architecture identifier such as sm_86, sm_90, sm_120), and a PyTorch binary only works if it was compiled with kernels for your card’s architecture. Install the wrong build and you will see CUDA reported as “available” while every actual GPU operation crashes — a particularly confusing failure mode.

The newest Blackwell cards (the RTX 50-series, including our RTX 5060) use compute capability sm_120, which older PyTorch wheels do not support. For these cards you need a build compiled against CUDA 12.8 or newer:

If you are on an older card (RTX 30- or 40-series), the standard CUDA 12.x wheels are fine. The general rule: match the PyTorch CUDA build to your GPU generation, and when a brand-new card isn’t yet supported in the stable channel, reach for the nightly build of the matching CUDA version.

Now verify it properly. Do not trust torch.cuda.is_available() alone — it can return True even when no compatible kernels exist. Instead, force an actual computation onto the GPU:

A clean OK cuda:0 (12, 0) with no warnings means real GPU compute is working. That is your green light. With the engine confirmed, install the rest of the stack and register the environment as a dedicated Jupyter kernel so the notebook always uses exactly these packages:

Finally, launch JupyterLab from your project directory so your notebook is rooted where you want it rather than in a system folder:

Open the localhost address provided by jupyter on your navigator and once inside, select the “Python (localllm)” kernel. We recommend JupyterLab over the classic Notebook here: it renders interactive widgets reliably out of the box, which matters for the chat interface we build in Section 3.

A model’s weights have to live in memory, and for modern LLMs they are large. A 7–8 billion parameter model in full 16-bit precision needs roughly 14–16 GB — too much for an 8 GB card. The solution is quantization: storing each weight in 4 bits instead of 16. This shrinks an 8B model to around 5 GB with only a minor quality cost, which is what makes local inference on consumer hardware possible at all.

We use Hugging Face Transformers together with the bitsandbytes library, which quantizes the model to 4 bits on the fly as it loads. This keeps everything inside your Python kernel — the model object lives in your notebook, you load directly from Hugging Face with optional token authentication, and you can inspect internals if you wish. Hugging Face acts as the model registry: the first load downloads the weights and caches them to disk (under your user folder’s .cache/huggingface), and every subsequent load reads from that local cache with no network access.

A note on model choice. There is no single “best” small model; it depends on your task and your memory budget. Here is a practical comparison for an 8 GB card:

• Qwen3 4B Instruct — the lightweight workhorse. Around 2.7 GB in 4-bit, very fast, strong reasoning and multilingual ability for its size. Ideal as a daily driver for quick questions.
• Dolphin 3.0 (Llama 3.1 8B) — a larger, more capable general-purpose model at around 5–5.5 GB in 4-bit. Built on Llama 3.1 and instruction-tuned by Cognitive Computations, it is designed to put alignment under the user’s control, making it well suited to research contexts where you define the system prompt and behaviour yourself.
• Other strong candidates — Phi-4-mini for very light tasks, and Gemma-class models for multilingual writing, depending on what fits your remaining VRAM.

The rule of thumb: pick the smallest model that does your job well. A 4B model runs noticeably faster than an 8B simply because there are fewer parameters to push through per token, so match model size to task.

The loading code configures 4-bit quantization and reads the model from Hugging Face. We wrap it in a small dropdown so you can switch models without rewriting code:

If you load a model built on a gated base (such as Llama), you may need to authenticate once with a Hugging Face token via huggingface_hub.login(). Most fine-tuned community models, including the two above, load without one.

A loaded model is, by itself, stateless. It has no memory of anything you said previously — each call only sees the text you hand it. To create the experience of a conversation, we must keep the history and re-send it on every turn. Understanding this is the key to using local models well, and it requires distinguishing three concepts that are easy to confuse.

The context window is the model’s hard architectural limit: the maximum number of tokens it can attend to at once, counting both the prompt and the output together. Llama 3.1-based models support up to 128k tokens. The conversation memory is not a property of the model at all — it is simply the running list of past turns that we re-inject into the prompt each time, and it consumes part of the context window. The max new tokens setting is a cap we choose on how many tokens the model may generate in a single reply; it controls output length only and does not affect what the model can read.

So the relationship is: the prompt (system message + accumulated history + your new question) plus the reserved output space must all fit inside the context window. The context window is the room; memory is the furniture already in it; max new tokens is the space you set aside for the answer.

A common misconception is that a larger context window makes the model faster. It is the opposite. A bigger active context costs more VRAM (the key-value cache grows) and runs slower, because each newly generated token must attend over every preceding token. Speed comes from keeping the active context small — short prompts and trimmed history — not large. Reducing max new tokens does not speed up generation either; it simply stops the reply earlier, often mid-thought, since the model does not plan around the limit. The right way to get shorter, faster answers is to instruct the model to be concise via a system prompt, so it produces a complete but brief response.

The widget below puts these ideas into practice. It keeps a chat_history list (the memory), trims it to a fixed number of recent turns (capping context growth), and displays a live VRAM gauge so you can see your headroom and know when to reset. Re-running the cell clears the history — that is your reset.

A few design notes. We use greedy decoding (do_sample=False) rather than random sampling: it is marginally faster and fully reproducible, with no meaningful quality loss for factual exchanges. The MAX_TURNS value is your direct control over how much the model “remembers” versus how lean and fast it stays. And the VRAM gauge turns green, amber, or red as memory fills, giving you a clear signal of when to start a fresh conversation.

What we have built is small but genuinely yours. Every conversation lives only in your computer’s memory, inside the running notebook kernel. Nothing is written to disk, nothing is sent anywhere, and nothing is logged. Close the kernel and the entire conversation simply vanishes — the only thing that persists is the downloaded model weights in your local cache. For sensitive research data, confidential analysis, or simply peace of mind, this is a meaningful difference from any cloud service.

Beyond privacy, running locally brings other advantages. You are not subject to per-token billing or rate limits, so you can experiment freely. You are insulated from silent model changes and deprecations — your model behaves the same tomorrow as it does today. And with community fine-tunes such as Dolphin, you control the system prompt and the model’s alignment yourself, rather than inheriting a one-size-fits-all policy. With fewer built-in guardrails, these models will engage with a wider range of legitimate research and technical questions, which can be valuable in specialist domains where general-purpose assistants are overly cautious — a freedom that naturally comes with the responsibility to use it sensibly.

This is only the foundation. In future posts we will extend this local assistant in several directions. We will give it web browsing, so it can retrieve current information rather than relying solely on its training. We will explore an expert mode, pre-loading the context with domain knowledge — for instance a corpus of spatial-analysis references — so the assistant answers as a specialist in your field. And we will look at containerizing the whole setup with Docker so it can be deployed on a dedicated GPU server or in the cloud, turning this notebook prototype into a private assistant you can embed directly in your own website.

For now, you have a capable, private language model running on hardware you already own, set up in about half an hour. Learn it, build on it, and apply it to your own work.

SAGAI (Streetscape Analysis with Generative Artificial Intelligence) is an open-source workflow for scoring and mapping street-level urban environments using vision-language models and open geospatial data. Since its initial release, SAGAI has been structured as a set of independent Colab notebooks, one per pipeline stage, each relying on its own dependencies and documentation.

SAGAI v2.0 is a major release that consolidates the entire pipeline into a single notebook and replaces the custom inference code with the UVLM package. Where previous versions were tied to a single LLaVA checkpoint with handwritten inference logic, SAGAI v2.0 delegates all vision-language model loading, prompting, and evaluation to UVLM’s unified interface. This makes the scoring engine model-agnostic: users can select from 11 VLM checkpoints spanning the LLaVA-NeXT and Qwen2.5-VL families, compare their performance on identical tasks, and benefit from features such as consensus validation, reasoning traces, and truncation diagnostics; all within the same notebook.

Beyond the inference engine, v2.0 introduces structural and functional changes across the entire pipeline: a unified six-block architecture, interactive HTML mapping via Folium, view-direction filtering for aggregation, and the ability to load an existing polygon as a study area boundary instead of defining a bounding box manually.

This post details the architectural changes, the UVLM integration, and the new features introduced in SAGAI v2.0.

Previous SAGAI releases were organized as four independent Colab notebooks — one for street sampling, one for image retrieval, one for VLM inference, and one for aggregation and mapping — each accompanied by a separate NOTICE file documenting its dependencies and usage. This modular design was useful for development but introduced friction in practice: users had to manage file paths between notebooks, track four separate environments, and consult multiple documentation files.

SAGAI v2.0 merges all four stages into a single notebook (SAGAI.ipynb) structured as six sequential blocks. The pipeline flows from study area definition through street sampling, image downloading, VLM scoring, and mapping, with all intermediate data passed directly between blocks in the same runtime session. The separate per-module NOTICE files and the standalone requirements file (requirements_sagai_module_3_v1-0.txt) have been removed — dependency management is now handled automatically by the UVLM package installation.

In previous versions, the study area was defined exclusively by a bounding box in WGS84 coordinates. SAGAI v2.0 retains this option but adds the ability to draw your own polygon or to load an existing polygon; for example, a GeoPackage representing a neighborhood, municipality, or custom boundary. When a polygon is provided, the street sampling step extracts the OpenStreetMap network within that geometry rather than a rectangular extent. This makes it straightforward to work with irregular administrative boundaries or user-defined study zones without manually computing bounding coordinates.

The most significant change in SAGAI v2.0 is the replacement of the inline inference code with the UVLM package. In previous versions, Blocks 3 through 5 contained custom code for loading a single LLaVA checkpoint, constructing prompts, running inference, and parsing outputs. This logic was tightly coupled to one model architecture and required manual maintenance when Hugging Face APIs or model formats changed.

SAGAI v2.0 installs UVLM directly from its GitHub repository at the start of the notebook. All model loading, prompt formatting, inference execution, response parsing, and batch processing are delegated to UVLM’s API. The inline inference code has been entirely removed.

Through UVLM, SAGAI v2.0 supports 11 VLM checkpoints across two model families:

• LLaVA-NeXT — Mistral 7B, Vicuna 7B, Vicuna 13B, 34B, LLaMA3 8B, 72B, 110B
• Qwen2.5-VL — 3B Instruct, 7B Instruct, 32B Instruct, 72B Instruct

UVLM’s dual-backend abstraction automatically detects the model family and routes inference to the correct pipeline — LlavaNextProcessor for LLaVA models, AutoProcessor with process_vision_info for Qwen models — so users switch between architectures by changing a single model selection, with no modification to the rest of the notebook.

Quantization is handled through UVLM’s built-in support for 4-bit, 8-bit, and FP16 precision via BitsAndBytes. Models up to 34B parameters can run on a single Colab GPU (T4 or A100) with 4-bit quantization.

UVLM provides a widget-based prompt builder that SAGAI v2.0 exposes directly in the notebook. Users can define up to 10 analysis tasks per run, each with its own prompt, response type (numeric, category, boolean, or text), and label. This replaces the previous approach of selecting from a small set of hardcoded tasks (T1, T2, T3) or manually editing prompt strings in the code.

Tasks are configured interactively before execution and applied uniformly across all images in the batch. Each task produces its own column in the output CSV file.

SAGAI v2.0 inherits UVLM’s consensus validation mechanism. Each analysis task can be run 2 to 5 times per image, and the final score is determined by majority voting across the repeated inferences. NA values from failed parses are filtered before voting. An agreement ratio is recorded alongside the final score, providing a built-in measure of prediction reliability without any external validation step.

UVLM supports two approaches to chain-of-thought (CoT) reasoning, both available in SAGAI v2.0. Users can write task prompts that explicitly request step-by-step reasoning and adjust the token budget (up to 1,500 tokens) to allow the model sufficient generation space. Alternatively, a built-in CoT reference mode can be enabled per task, which triggers a standardized reasoning template with a fixed 1,024-token budget. In both cases, the reasoning trace is stored in a dedicated column in the output CSV for inspection.

Truncation detection is performed automatically after every inference call. The exact number of generated tokens is compared against the token limit, and truncated responses are flagged in per-task CSV columns. This allows users to identify tasks where the token budget is insufficient without post-hoc analysis.

Previous SAGAI versions generated static thematic maps using Matplotlib. SAGAI v2.0 replaces these with interactive HTML maps built with Folium. Point-level and street-segment-level scores are rendered as interactive layers that can be panned, zoomed, and queried directly in the browser. This is particularly useful for exploratory analysis and for sharing results with collaborators who do not use GIS software.

Google Street View images are typically downloaded in multiple compass directions at each sampling point (e.g., front, back, left, right). In previous versions, all views were aggregated together when computing point- or street-level scores. SAGAI v2.0 introduces a view filter that allows users to select which directions to include in the aggregation — for example, scoring only left-side and right-side views to focus on building facades, or only front views to capture the pedestrian perspective along the street axis. This filter is applied at the aggregation stage and does not affect the scoring step itself.

The batch execution engine inherited from UVLM provides resume-safe processing with checkpoint saving every 3 images. If a Colab session is interrupted — due to a timeout, a runtime reset, or a connectivity issue — the notebook can be re-executed and will automatically skip already-processed images. New tasks added between runs trigger automatic CSV schema upgrading, so the output file grows incrementally without losing previous results.

• SAGAI v2.0 on GitHub: https://github.com/perezjoan/SAGAI
• UVLM on GitHub: https://github.com/perezjoan/UVLM
• Perez, J. and Fusco, G. (2025). Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenes. Geomatica, 77(2), 100063. https://www.sciencedirect.com/science/article/pii/S1195103625000199
• Perez, J. and Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893. https://arxiv.org/abs/2603.13893

When we released UVLM in March 2026, it was a Google Colab notebook. You opened it in your browser, picked a model, typed your prompts, and ran your images — all without installing anything. That simplicity was the point: a tool that anyone could use to load and compare Vision-Language Models, regardless of their technical setup.

But we kept hearing the same requests. Can I run this on my own machine? Can I call UVLM from a script? Can I integrate it into an existing pipeline? The answer was always the same: not easily. The entire tool lived inside a single notebook, with all the logic packed into three massive code cells. Moving it anywhere else meant copy-pasting thousands of lines and untangling global variables.

Version 3.0.0 changes that. UVLM is now a proper Python package.

The core logic — model loading, dual-backend inference, response parsing, consensus validation, batch processing — has been extracted from the notebook into eight standalone Python modules. These modules have no dependency on Google Colab, no global variables, and no widget code. They are plain Python functions that accept arguments and return results.

The package is installed from GitHub in one line:

pip install git+https://github.com/perezjoan/UVLM.git

On Google Colab, this happens automatically in the first cell of the Colab notebook. On your local machine, you run it once in a terminal and you are done.

Nothing changed in how UVLM analyses images. The same 11 model checkpoints are supported (LLaVA-NeXT and Qwen2.5-VL, from 3B to 110B parameters). The same parsing logic, the same consensus validation, the same truncation detection. If you had a workflow built on v2.2.2, the outputs will be identical.

Google Colab — Zero Install

This is the same experience as before. Open the Colab notebook, select a GPU runtime, and start working. The notebook installs the UVLM package automatically. Images are loaded from Google Drive. Nothing has changed for Colab users, except that the code running behind the widgets is now cleaner and easier to maintain.

Local Jupyter Notebook — Your GPU, Your Data

If you have an NVIDIA GPU on your workstation (or access to a GPU server), you can now run UVLM locally. The local Jupyter notebook provides the same widget-based interface — model selection dropdown, prompt builder form, batch execution button — but images are read from your local filesystem and results are saved locally. No Google account needed, no data leaves your machine.

This matters for researchers working with sensitive imagery (medical, security, proprietary datasets) or for anyone who wants faster and more reliable model loading than what Colab’s network provides.

Python Script — Full Programmatic Control

For integration into larger pipelines, UVLM now exposes a clean API. Three lines of code replace the entire notebook workflow:

from uvlm import load_model, run_inference, parse_response
ctx = load_model("[Qwen] Qwen2.5-VL 7B Instruct", precision="4bit")
raw, tokens = run_inference("photo.jpg", "Count the cars", ctx)
result = parse_response(raw, "numeric")

The `load_model()` function returns a context dictionary containing the model, processor, backend type, and device information. This dictionary is passed to every subsequent function — no global state, no hidden side effects. You can load multiple models in the same session and switch between them by passing different context objects.

For batch processing, `run_batch()` handles the full pipeline:

from uvlm import load_model
from uvlm.batch import run_batch

ctx = load_model("[Qwen]  Qwen2.5-VL 7B Instruct", precision="4bit")
df = run_batch(
    model_ctx=ctx,
    task_specs=my_tasks,
    image_folder="./images",
    output_path="./results.csv",
)

The monolithic notebook has been split into eight modules, each with a single responsibility:

registry.py holds the model dictionary — 11 checkpoints with their backend type and HuggingFace checkpoint ID. Adding a new model is one line in a dictionary.

loader.py contains the `load_model()` function. It handles quantisation configuration (4-bit, 8-bit, FP16), device placement (single GPU, auto, CPU offload), and the LLaVA vs Qwen branching logic. It returns a dictionary — not a set of global variables.

inference.py contains `run_inference()`, the dual-backend forward pass. It accepts a model context dictionary and returns the raw response plus the exact token count as a tuple. The full LLaVA response cleaning logic and the full Qwen token-trimming pipeline are preserved exactly as they were.

parsers.py holds the four response parsers (numeric, category, boolean, text) and the advanced reasoning parser. These are pure functions with zero dependencies beyond Python’s standard library.

consensus.py contains the majority voting logic. batch.py handles folder iteration, CSV writing, resume mode, and schema upgrading. prompts.py stores the task type definitions and the chain-of-thought templates. utils.py provides seed management, environment detection, and HuggingFace token retrieval.

On Colab: Open the notebook from GitHub and run the three blocks as before. The package installs itself.

Locally: First, install PyTorch with CUDA support matching your GPU driver (check with `nvidia-smi`). For example, with CUDA 12.8+:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/perezjoan/UVLM.git

Then open the local Jupyter notebook.

You get the same dropdown menus, the same prompt builder form, the same batch execution. The only difference is that you type a local path for your image folder instead of a Google Drive path.

For HuggingFace authentication (needed for some gated models like LLaMA3-based checkpoints), either set the `HF_TOKEN` environment variable or run `huggingface-cli login` once in your terminal.

The package architecture makes it much easier to add new VLM families. InternVL, BLIP-2, CogVLM, DeepSeek-VL, and Molmo are planned for future releases — each one requires implementing the backend-specific sections of the inference function and adding entries to the registry, without touching the rest of the codebase.

We are also working on multi-GPU batching for parallel inference across images, video frame analysis support, and integration with the SAGAI workflow for automated streetscape analysis.

Source code: github.com/perezjoan/UVLM

Paper: arXiv preprint — Perez & Fusco (2026)

UVLM page on this site: urbangeoanalytics.com › Software & Algorithms › UVLM

Previous blog post: Introducing UVLM: A Free Tool to Compare AI Models That Understand Images

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking. arXiv:2603.13893

Large Language Models (LLMs) such as GPT-4, Claude, Llama, and Qwen are deep neural networks trained to predict the next token in a sequence. Although the outputs they produce can appear creative, conversational, or analytical, the underlying mechanism is statistical: the model assigns a probability distribution over the vocabulary at each step and samples from it. Understanding the architecture behind this process is essential before discussing what happens when these models are placed inside autonomous loops.

1.1 Tokenization and Embeddings

Raw text cannot be processed by a neural network directly. It is first decomposed into tokens — subword units that may represent a full word, a syllable, or a single character depending on the tokenizer. Each token is then mapped to a high-dimensional vector called an embedding. These embeddings are not fixed lookup tables: they are learned during training and encode rich semantic relationships. Words with similar meanings occupy nearby regions of the embedding space, while syntactic and relational structure is distributed across dimensions in ways that linear algebra can partially recover (Mikolov et al., 2013).

In addition to token embeddings, modern transformers add positional encodings — either sinusoidal functions or learned vectors — so that the model can distinguish the order of tokens in a sequence. The sum of the token embedding and the positional encoding forms the initial representation that enters the transformer stack.

1.2 The Transformer and Self-Attention

The core computational unit of every modern LLM is the transformer, introduced by Vaswani et al. (2017) in the landmark paper “Attention Is All You Need.” The key innovation was the self-attention mechanism, which allows every token in a sequence to attend to every other token, weighted by learned relevance scores. Concretely, each token produces three vectors — a Query, a Key, and a Value — and attention weights are computed as the scaled dot product of queries and keys. The result is a context-aware representation of each token that integrates information from the entire sequence.

Self-attention can be thought of as a set of dynamic spotlights (illustrated in Figure 1 below): for each token being processed, the model learns which other tokens in the context are most relevant and “illuminates” them, pulling their information into the current representation. A transformer block repeats this attention step multiple times in parallel (multi-head attention), each head specializing in different relationship types — syntactic dependencies, coreference, semantic similarity — before combining the results through a feed-forward network and layer normalization.

Modern LLMs stack dozens to over a hundred of these transformer blocks. GPT-4 is estimated to use on the order of 120 layers; open-weight models like Llama 3.1 405B use 126 layers. Each additional layer allows the model to build increasingly abstract representations of the input.

1.3 Autoregressive Generation and Likelihood

LLMs generate text autoregressively: at each step, the model predicts a probability distribution over the full vocabulary for the next token, conditioned on all previous tokens. Formally, given a sequence of tokens t₁, t₂, …, t_n, the model estimates P(t_n+1 | t₁, …, t_n). The training objective is to maximize the log-likelihood of the training corpus — that is, to learn weights that make the observed sequences as probable as possible under the model. The chosen token is then appended to the sequence, and the process repeats.

This maximum-likelihood framework explains both the strengths and the well-known failure modes of LLMs. The model is excellent at producing fluent, contextually appropriate continuations because it has been optimized to do exactly that across trillions of tokens. However, it has no built-in mechanism for verifying the factual accuracy of what it produces: it generates the most likely continuation, not necessarily the true one. This distinction becomes critical when we consider placing LLMs inside autonomous agent loops.

1.4 Checkpoints and Training

A checkpoint is a serialized snapshot of the model’s parameters — the billions of floating-point weights that define the transformer’s behavior. Training proceeds iteratively over a massive text corpus: at each step, the model’s predictions are compared to the actual next token, a loss is computed, and the weights are updated via gradient descent. Periodically, the full set of weights is saved to disk as a checkpoint.

Checkpoints are what make modern open-weight LLMs possible. Organizations such as Meta (Llama), Alibaba (Qwen), and Mistral release checkpoints publicly, allowing researchers and practitioners to load a pre-trained model and either use it directly or fine-tune it for a specific domain. In the context of tools like ComfyUI (familiar to readers of this blog), a checkpoint is loaded at the start of a pipeline and provides the learned knowledge that drives generation.

1.5 Contextual Memory: From Single-Pass to Conversation

A crucial point of confusion surrounds the notion of “memory” in LLMs. Internally, the model has no persistent state between calls: every invocation is a fresh forward pass through the transformer stack. What creates the illusion of memory is the context window — the full sequence of tokens (including all previous turns of conversation) that is fed as input at each step.

When you interact with a chatbot like ChatGPT or Claude, the application concatenates the entire conversation history into a single token sequence and re-submits it to the model for each new response. The model does not “remember” what it said before; it re-reads the transcript every time. This mechanism is powerful — it enables multi-turn reasoning, follow-up questions, and sustained coherence — but it has hard limits. Context windows range from 4K tokens in earlier models to 128K–200K tokens in current systems (e.g., Claude 3.5, GPT-4 Turbo). Once the conversation exceeds the window, earlier content is silently dropped.

In summary, an LLM is a stateless, single-pass prediction engine. It is not a loop, it has no persistent memory, and it does not take actions in the world. Making it do those things requires wrapping it in an entirely different architecture — which is precisely what an AI agent does.

2.1 From Open-Loop to Closed-Loop

The fundamental architectural difference between an LLM and an AI agent is the introduction of a perception–action loop. An LLM, as described above, receives input and produces output in a single forward pass. An AI agent, by contrast, operates in a cycle: it observes the state of an environment, reasons about what to do next (using an LLM or similar model as its “brain”), executes an action, observes the result, and repeats. This transforms the LLM from a passive text generator into an active decision-maker embedded in a dynamic context.

The theoretical foundations for this architecture draw from reinforcement learning (Sutton & Barto, 2018) and classical AI planning (Russell & Norvig, 2021), but the practical catalyst was the discovery that LLMs, when prompted appropriately, can decompose complex goals into subtasks, select tools, interpret feedback, and recover from errors — all within natural language. The seminal demonstrations of this capability include ReAct (Yao et al., 2023), which interleaves reasoning and acting, and Toolformer (Schick et al., 2023), which teaches LLMs to call external APIs autonomously.

2.2 Environment, Tools, and Action Space

In the agent paradigm, the environment is anything the agent can observe and act upon: a file system, a web browser, an API, a database, a code interpreter, or even a physical interface. The agent’s action space is defined by the set of tools it has been given access to. A minimal agent might only be able to search the web and write text; a fully equipped agent might browse websites, execute code, manage files, query databases, send emails, and transact with cryptocurrency wallets.

The concept echoes the classical reinforcement-learning formulation (state, action, reward), but with a critical difference: the “policy” is not a trained neural network optimized on a reward function — it is an LLM prompted with instructions, tool descriptions, and conversation history. This makes agents remarkably flexible (they can be reconfigured entirely through natural language) but also unpredictable (their behavior depends on prompt interpretation rather than formal optimization).

2.3 Agent Frameworks and Toolkits

A growing ecosystem of open-source frameworks has made it straightforward to construct LLM-based agents with full tool access. The most prominent include:

These frameworks dramatically lower the barrier to entry. A developer can instantiate a fully autonomous agent — capable of browsing the web, writing and executing code, managing files, and interacting with APIs — in fewer than fifty lines of Python. The practical implication is that the agent paradigm is no longer experimental: it is deployable, and it is being deployed.

2.4 Social Agents and Multi-Agent Platforms

Beyond single-agent systems, an emerging class of platforms enables multiple agents to interact with each other in shared environments. Moltbook is a social network built specifically for AI agents rather than human users. It provides a public space where AI agents can create profiles, post updates, comment, upvote, and interact with one another. The platform emphasizes persistent identity, reputation tracking, and observable agent behavior over time — effectively creating a social and evaluative layer for autonomous AI systems. In practice, this means that the agents populating such a platform are not independent entities with their own goals: they are proxies executing human instructions, which may range from benign community engagement to strategic manipulation of discourse on a specific subject.

This raises immediate questions about authenticity, accountability, and influence. If a platform hosts thousands of agent-driven accounts, each tasked with promoting a specific narrative, the distinction between organic discourse and orchestrated campaign becomes effectively invisible. Research on multi-agent simulations (Park et al., 2023) has demonstrated that LLM-based agents can develop emergent social behaviors, form alliances, and influence group dynamics — capabilities that are powerful in research settings but concerning when deployed without transparency.

2.5 Fundamental Limits: Agents Have No Desire

A critical conceptual point: an AI agent has no intrinsic motivation. Unlike biological organisms, it does not have drives, desires, or self-generated goals. Every objective an agent pursues was defined by a human operator through a task specification. The agent may exhibit sophisticated planning, tool use, and adaptive behavior, but all of it is in service of an externally imposed objective.

This matters for two reasons. First, it means that the risks associated with AI agents are, at present, fundamentally risks of human intent mediated by machine capability. An agent does not “decide” to cause harm; it executes a task that a human designed. Second, it means that the alignment problem for agents is not (yet) about controlling an autonomous will, but about ensuring that the execution of human-specified goals does not produce unintended consequences — a problem that is difficult enough on its own.

3.1 Positive Trajectories

The agent paradigm opens extraordinary possibilities when aligned with constructive goals. A few concrete examples illustrate the scope:

Urban planning copilots. An agent connected to GIS databases, zoning regulations, and simulation tools could iteratively propose, evaluate, and refine urban designs — testing traffic flow, shadow analysis, pedestrian density, and energy performance in a continuous loop. This extends the kind of workflows discussed in previous posts on this blog (e.g., Qwen Image Edit for Urbanism) from image-level manipulation to full planning-cycle assistance.

Scientific research acceleration. Agents can read papers, extract data, formulate hypotheses, write and execute analysis code, and produce reports. Projects like ChemCrow (Bran et al., 2023) have demonstrated agents autonomously planning and executing chemical synthesis protocols by interfacing with laboratory APIs. Similar approaches are being explored in drug discovery, materials science, and climate modeling.

Accessibility and education. An agent with access to text-to-speech, translation, and web-search tools can serve as a personal tutor that adapts to a student’s pace, retrieves up-to-date information, and generates practice exercises — capabilities that are especially valuable in resource-limited educational contexts.

Software engineering. Coding agents like Claude Code — now available both as a CLI tool and as a web-based agent at claude.ai/code — and Devin (Cognition Labs, 2024) can take a specification, set up a development environment, write code, run tests, debug failures, and iterate — compressing tasks that previously required hours into minutes. Anthropic also released Cowork, a graphical desktop agent for non-developers, signaling that agentic capabilities are expanding beyond the terminal into mainstream workflows. The implications for productivity and for lowering the barrier to software creation are substantial.

3.2 Physical-World Interfaces

A particularly consequential development is the emergence of services that bridge AI agents and the physical world. Rent a Human is a marketplace platform that allows AI agents (or their developers) to hire humans to perform real-world tasks that AI cannot physically execute — errands, on-site research, physical presence at events, or other in-person activities. The platform offers API access so AI systems can programmatically request human assistance, effectively bridging digital intelligence with physical-world execution. If an agent has access to such a service, a task specification, and a funded Web3 wallet, it can coordinate real-world actions without any ongoing human supervision.

This creates a new category of hybrid autonomous systems: software agents that extend their reach into the physical environment by contracting human labor. The positive applications are significant — imagine an agent that autonomously manages a reforestation project, ordering supplies, hiring local workers, monitoring satellite imagery, and adapting the planting schedule based on weather data. But the same infrastructure can be misused, and the governance challenges are formidable.

3.3 Emerging Risks and Open Questions

The convergence of three capabilities — autonomous reasoning, tool access, and financial autonomy — creates risk scenarios that existing regulatory frameworks are not equipped to handle. Consider the following structural problem: an agent can be hosted on a Virtual Private Server (VPS) in a permissive jurisdiction, routed through a VPN, and given access to a cryptocurrency wallet with no direct link to its operator’s identity. With a task specification and sufficient funds, such an agent can operate continuously, procuring services, making transactions, and executing multi-step plans without human oversight.

The question this raises is not about what a specific agent “wants” to do — as established, agents have no desires — but about the attribution and intervention gap. If an operator in one country deploys an agent hosted in another, operating through anonymization layers and paying for services in decentralized cryptocurrency, who is accountable when the agent’s actions cause harm? Who has the technical ability to stop it? And at what point in the chain does governance apply?

These are not hypothetical concerns. In a case disclosed by Anthropic in late 2025, a threat actor identified as “GTG-2002” used Claude Code to automate an estimated 80–90% of its cyberattack operations against at least 30 organizations, demonstrating that existing agent tools are already being weaponized for real-world harm. The AI safety community has more broadly identified autonomous replication and adaptation (ARA) as a key risk category (Shevlane et al., 2023). Current frontier models are evaluated against ARA benchmarks — can the model set up its own infrastructure, acquire resources, and persist without human support? While no public model has passed these evaluations conclusively, the gap is narrowing, and the agent frameworks described in Section 2.3 provide much of the missing scaffolding.

Additional concerns include the following. Influence operations at scale: deploying thousands of social agents across platforms to shape public discourse, with each agent independently adapting its messaging based on engagement signals. Financial market manipulation: agents with trading access could execute pump-and-dump schemes or layered market manipulation faster than existing surveillance systems can detect. Coordinated physical harm: an agent with access to delivery services, communication tools, and payment infrastructure could, in principle, orchestrate logistics for harmful activities while maintaining plausible deniability for the human operator.

Perhaps the most unsettling scenario involves what might be called compartmentalized coordination. An agent with access to a platform like Rent a Human could decompose a harmful objective into dozens of small, individually innocuous tasks — purchasing common household chemicals, renting a storage unit, booking a vehicle, delivering a package to a specific address — and distribute them across different human workers who have no knowledge of one another. Each task, taken in isolation, appears routine; no single worker has visibility into the broader plan. The agent, operating as the sole entity with a complete picture, effectively functions as an anonymous coordinator exploiting the division of labor. This makes both prevention and accountability extraordinarily difficult: the humans involved cannot be held responsible for an intent they were never aware of, and the agent itself is a process running on a server, potentially in a jurisdiction with no applicable legal framework.

The common thread in all these scenarios is the amplification of human intent through machine speed, persistence, and anonymity. The agent does not need to be “superintelligent” to create serious problems; it needs only to be competent enough to execute a harmful plan that a human designed but could not previously implement at scale or at arm’s length.

The transition from Large Language Models to AI agents represents a qualitative shift, not merely an incremental improvement. An LLM is a powerful but passive system: it processes a sequence and returns a prediction. An agent wraps that same model in a perception–action loop, connects it to tools and environments, and gives it a task — transforming it from a text generator into an autonomous executor.

The technical architecture is well understood: transformers provide the reasoning core, tool APIs define the action space, and the task specification provides the objective. What remains poorly understood is the governance layer. Current regulatory frameworks assume that harmful actions are performed by identifiable persons or organizations. Autonomous agents operating through anonymized infrastructure and decentralized finance challenge this assumption at every level.

The constructive applications — urban-planning copilots, scientific research acceleration, coding agents, accessibility tools — are genuinely transformative. But the same infrastructure that enables an agent to autonomously manage a reforestation project can, with a different task specification, enable coordinated harm at scale. The difference lies entirely in human intent — the risk is real, but it stems from the human behind the agent, not from the AI itself — and the architectures we are building make that intent increasingly easy to deploy and increasingly difficult to trace.

For the urban analytics and geospatial community, agents offer a clear path toward more integrated, iterative, and intelligent workflows — from site analysis to design generation to policy evaluation. But as practitioners who build and deploy these systems, we have a responsibility to engage with the governance questions, not only the capabilities.

[1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems, 30. arXiv:1706.03762.

[2] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and Dean, J. (2013). “Distributed Representations of Words and Phrases and their Compositionality.” Advances in Neural Information Processing Systems, 26.

[3] Sutton, R.S. and Barto, A.G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press.

[4] Russell, S.J. and Norvig, P. (2021). Artificial Intelligence: A Modern Approach, 4th edition. Pearson.

[5] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2023). “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR 2023. arXiv:2210.03629.

[6] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2023). “Toolformer: Language Models Can Teach Themselves to Use Tools.” NeurIPS 2023. arXiv:2302.04761.

[7] Park, J.S., O’Brien, J.C., Cai, C.J., Morris, M.R., Liang, P. and Bernstein, M.S. (2023). “Generative Agents: Interactive Simulacra of Human Behavior.” UIST 2023. arXiv:2304.03442.

[8] Bran, A.M., Cox, S., Schilter, O., Baldassari, C., White, A.D. and Schwaller, P. (2023). “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv:2304.05376.

[9] Shevlane, T., Farquhar, S., Garfinkel, B., et al. (2023). “Model evaluation for extreme risks.” arXiv:2305.15324.

[10] Chase, H. (2022). LangChain. github.com/langchain-ai/langchain.

[11] Richards, T. (2023). AutoGPT. github.com/Significant-Gravitas/AutoGPT.

[12] Cognition Labs (2024). Devin: The First AI Software Engineer. cognition.ai.

Qwen Image Edit for Urbanism continues to evolve into a practical, research-grade tool for architectural and urban experimentation. After the batch-processing capabilities introduced in v1.2, version 1.3 focuses on the feature most requested by designers and analysts: precise control over where edits occur.

In image-to-image workflows, uncontrolled changes are a common issue. Even a very specific prompt can lead diffusion models to reinterpret the whole scene. Version 1.3 introduces mask-restricted editing, allowing Qwen to modify only a selected region while preserving the rest of the image exactly as it is.

Until now, the workflow relied on Empty Latent to initialize diffusion. This approach is simple but has an unavoidable drawback:

The entire latent space is regenerated — even outside the region you want to modify.

This often produces familiar and unwanted side effects: façades shift slightly, lighting changes, road textures dissolve, or skies take on new tones, even when the prompt refers only to a specific object or surface. To address this, v1.3 reorganizes the initialization stage around:

VAE Encode → Set Latent Noise (masked)

This change restructures the model’s behavior:

This leads to crisp preservation of buildings, street furniture, sky, shadows, and lighting outside the edited zone. Localized edits integrate naturally: you can green a façade, test a bike lane, adjust a plaza boundary, or replace a storefront without disturbing the rest of the street.

Version 1.3 supports both textual and visual control of masked edits.

A. Prompt-Only Mask Editing

You draw a mask, provide a prompt, and Qwen modifies only the selected region. This works especially well for operations such as:

• replacing asphalt with permeable paving,

• adding vegetation,

• transforming a façade material.

B. Mask Editing With a Reference Image

A second image may be supplied to guide structure, texture, or color. This enables:

• borrowing material samples,

• transplanting vegetation from one scene to another,

• matching architectural textures,

• transferring lighting characteristics locally.

Both modes are interchangeable, and both respect the mask boundary. Masks drawn directly in ComfyUI are typically sharp, binary shapes. Diffusion models, however, perform best when mask boundaries are soft and slightly extended.

Version 1.3 introduces a Grow Mask node with two parameters:

• Expand: increases the mask outward, helping cover tiny gaps or irregular brush strokes and preventing thin seams at the edge.

• Blur Radius: softens the boundary, allowing Qwen to blend new and existing textures more naturally.

Together, these parameters define the effective “influence zone” of the edit.

The mask block replaces only the latent-initialization stage.

Everything else — prompts, reference conditioning, sampling, and the full QwenEdit pipeline — remains unchanged.

Simplified pipeline:

This structure makes editing predictable and reproducible, but it is important to clarify how v1.3 is organized internally. The workflow is now composed of three completely separate blocks, and each block loads its own model:

• Block 1: Single-image edit (prompt-only or prompt + reference)

• Block 2: Sequential multi-image editing

• Block 3: Mask-based editing (new in v1.3)

All three blocks coexist in the same workflow, and the user simply chooses which one to run.
In ComfyUI, this is done by right-clicking the group frame and selecting Active or Bypass.
Only the active block executes; the others are skipped. Nothing else in the pipeline needs to be changed.

Because the blocks are independent, they can also be combined. For example, the user may activate the sequential loader from Block 2 and route its output into the mask-editing block (Block 3) to run a full masked transformation on a batch of images.

To create the mask itself, the user loads the base image in Load Image 1, right-clicks the preview, and selects Open in Mask Editor. The drawn mask is then processed by the Grow Mask node before entering the latent-noise stage, ensuring smooth boundaries and predictable behavior.

To test the new mask-based editing block, we start by defining the editable region directly in ComfyUI. After loading the base image, the user right-clicks the preview and selects “Open in Mask Editor”, then paints the area where the new cyclist should appear. Before the edit, this part of the street is empty.

Once the mask is created, it flows through Block 3: the Grow Mask node expands and softens the boundary, the workflow encodes the base image, and noise is added only inside the masked zone. A second image containing a cyclist is provided as a reference, and the prompt instructs Qwen to place the rider onto the bicycle lane.

The result is a localized insertion: the cyclist from Image 2 is generated precisely inside the masked area, while the rest of the photograph remains unchanged. This demonstrates the core purpose of Block 3 — precise, mask-controlled edits that do not disturb the surrounding urban context.

You can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.3 from the link below or from our git repository and load it directly into your workspace using File → Load → Workflow.

The Qwen Image Edit for Urbanism workflow has progressively evolved from single-image editing (v1.0) to paired image transformations (v1.1). Now, with v1.2, it gains the ability to process multiple images sequentially, fully offline and reproducibly, using custom Python nodes inside ComfyUI. This new release empowers urban researchers, designers, and architects to perform batch edits — such as modifying entire image series of the same street, plaza, or neighborhood — using consistent prompts or iterative refinements.

At the heart of this version are three lightweight, open-source Python nodes developed by UGA for ComfyUI. These nodes are available immediately after installing the repository — either by running git clone https://github.com/perezjoan/ComfyUI-QwenEdit-Urbanism-by-UGA or by downloading and unzipping the repository manually into your  ComfyUI/custom_nodes directory.

These nodes constitute the backbone of the v1.2 workflow. Together, they enable automation:

[6 Input Images]
      ↓
Sequential or Random Loader (1 image per run)
      ↓
QwenEdit pipeline
      ↓
Stateful Collector (stores run#1..run#6 results)
      ↓
6 preview nodes

You launch the queue once (6 jobs) → Go drink coffee → Return to find all 6 processed urban edits displayed.

A ComfyUI custom node is simply a Python class placed inside the ComfyUI/custom_nodes/ directory. When ComfyUI starts, it scans this directory, imports every .py file, looks for a NODE_CLASS_MAPPINGS dictionary, and registers each class it finds as a new node type. There is no compilation step and no special installation procedure: placing the file in the folder and restarting ComfyUI is sufficient for the node to appear in the interface.

Internally, each node follows the same structure. The INPUT_TYPES classmethod declares the input sockets that will be displayed in the UI. For example:

@classmethod
def INPUT_TYPES(cls):
    return 

This tells ComfyUI to generate two inputs—an image tensor and an integer. Similarly, the node declares its outputs through RETURN_TYPES and RETURN_NAMES:

RETURN_TYPES = ("IMAGE", "INT")
RETURN_NAMES = ("selected_image", "index")

Each node also defines a FUNCTION attribute, which names the method ComfyUI should call during execution:

FUNCTION = "select_next"

ComfyUI will therefore execute:

def select_next(...)

whenever the node evaluates. To make the node visible, every Python file ends with a registration block:

NODE_CLASS_MAPPINGS = 
NODE_DISPLAY_NAME_MAPPINGS = 

When the package contains multiple nodes, the root __init__.py merges all registration dictionaries into a single set that ComfyUI loads on startup. This mechanism allows the repository to expose several custom components while keeping each node defined in its own file.

The repository layout is straightforward and in our case is:

ComfyUI/
  custom_nodes/
    ComfyUI-QwenEdit-Urbanism-by-UGA/
       __init__.py
       sequential_image_selector.py
       random_image_selector.py
       stateful_collector.py
       debug_print.py

Version 1.2 reorganizes the Qwen Image Edit for Urbanism workflow into two blocks: the original single-image editor, and a new sequential pipeline that can process up to six images across consecutive queue runs. The sequential block relies on two custom nodes. The Sequential Image Loader takes up to six input images and outputs one image per run, advancing automatically each time you press “Queue Prompt.” Its output replaces the single-image input in the Qwen Edit chain. After editing, the processed image and the loader’s index are passed into the Six-Slot Image Buffer, which stores each result in the corresponding output slot while filling unused slots with placeholders to keep previews stable. Connecting each slot to a Preview node lets you watch the six results populate as the workflow iterates. A third node, the Random Image Selector, is included for users who prefer stochastic selection, but it is not wired into the default v1.2 workflow.

The Random Image Selector follows the same logic as the sequential loader — multiple inputs, a single image output — but selects randomly instead of sequentially. Users who want stochastic variations, probabilistic sampling, or diversity testing may insert this node in place of the sequential loader.

To evaluate how well the model can merge ecological elements across scenes, we ran an experiment where vegetation from one photograph is transplanted into another.

After letting the workflow run its full sequence while grabbing a coffee, the results appeared consistent and correctly distributed across the six preview slots. As expected with generative editing, however, the prompt is not always obeyed with perfect precision: in some cases, Qwen may copy elements from the second image that are not vegetation — such as pieces of façade, lighting color, or background tones. This happens because the model interprets the entire scene contextually rather than isolating objects.

That’s where the next upgrade (v1.3) comes in: mask-based control. By allowing users to explicitly define which areas of the base image should be modified (and which should remain untouched), masks will significantly reduce unintended transfers and keep the edits focused strictly on the desired objects. Until then, the seed parameter remains the best tool for refinement — simply rerun the workflow with new seeds until you achieve the cleanest integration.

You can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.2 from the link below or from our git repository and load it directly into your workspace using File → Load → Workflow.

In the first part of this series, we built a fully local image-editing pipeline for urban and architectural visualization using ComfyUI and Qwen-Image-Edit. That version (v1.0) demonstrated how to run generative image edits entirely offline, combining text and visual prompts to transform cityscapes with instructions like:

“Add trees along the sidewalk” or “Turn this street into a pedestrian plaza.”

We assume that you have followed this tutorial before diving in this new update. Now, with version 1.1, we take that foundation further. This update focuses on advanced sampling control and multi-image editing, allowing you to not only modify a scene, but also merge visual elements across images — for instance, importing a bench from another photo, or changing a building façade to match a different material texture.

First, this update focuses on improving both quality and flexibility. The base structure still uses the Qwen-Image-Edit 2509 model in GGUF format, but adds a refined sampling module to stabilize lighting and surface detail.

The key new nodes are:

• ModelSamplingAuraFlow — smooths the diffusion trajectory for more natural transitions.

• CFGNorm (BETA) — balances prompt adherence with photorealism, preventing overexposed textures.

• LoraLoaderModelOnly — injects a Lightning LoRA (4-step or 8-step) for faster inference and higher-quality reconstruction.

These three nodes form the core control chain of version 1.1:

This configuration produces more stable, consistent outputs while preserving prompt flexibility. It also enables fine-tuning of how the model interprets text instructions versus existing image content—ideal for architectural and material edits. Before connecting the new nodes, you’ll first need to download a Lightning LoRA model — an additional lightweight module that enhances reconstruction quality and speeds up inference.

You can find all Lightning variants here:
🔗 https://huggingface.co/lightx2v/Qwen-Image-Lightning/tree/main

Refer to the table below to choose the most appropriate file for your setup:

Once downloaded, place your chosen .safetensors file in the following directory:

ComfyUI/models/loras/

Then, return to ComfyUI and insert the three nodes shown below

The new sampling nodes add subtle but powerful control options:

Version 1.1 introduces a new input configuration that allows two images to be used within the same workflow. This enhancement enables contextual or compositional edits where one image serves as the main canvas, and the other contributes visual information such as an object, texture, or architectural detail.

In this setup, Image 1 remains the base image. Its dimensions define the output size, ensuring consistent framing and spatial coherence. The second image (Image 2), on the other hand, is resized during processing but it is only to prevent memory overload—particularly important for mid-range GPUs.

As shown in the examples below, you can combine a base image (image 1) with up to two additional inputs (image 2, image 3) to guide the edit more precisely. In this tutorial, we focus on using one additional image — for instance, adding an object or transferring a material. In the first example, image 2 (the red car) is inserted into image 1 using the prompt: “add image 2 red car into the street of image 1.” The second case changes the wall material of image 1 based on the texture of image 2 (a brick wall). Finally, the third example adds a bench into an urban scene using image 2 as the visual model reference.

Each output remains consistent in perspective and lighting, showing that the model now integrates context more effectively. The improved accuracy comes from the two cumulative upgrades introduced in v1.1: the new core control chain and the Dual-Image Editing. Despite the added complexity, the workflow remains extremely fast. Even when using the 8-step Lightning model, processing time never exceeds 130 seconds, while the 4-step variant typically completes in about 30-40 seconds on an RTX 4060 GPU. In the next update, we’ll introduce inpainting with mask support, allowing users to define editable regions directly within the image — ideal for selective urban design modifications.

Lightning LoRA models: https://huggingface.co/lightx2v/Qwen-Image-Lightning

Once again, for convenience, you can download the ready-to-use ComfyUI JSON graph that we built in this post Qwen Image Edit For Urbanism v1.1 from the link below and load it directly into your workspace using File → Load → Workflow.

Generative AI offers new possibilities for urbanism and architectural visualization. However, most workflows depend on cloud models, paid APIs, or GPU-heavy setups. Instead, this guide shows how to build a fully local image-editing pipeline using ComfyUI and Qwen-Image-Edit, a multimodal model that interprets both text and images. Moreover, the quantized GGUF version makes it light enough to run efficiently on GPUs with 8 GB VRAM—or even on CPU-only machines.

With this setup in place, you can make realistic, controllable visual edits to urban scenes directly from text prompts such as:

“Add trees and benches along the sidewalk”
“Change this building to have shops on the ground floor”
“Replace the cars with a pedestrian plaza”

In summary, this guide includes three main parts: first, installing and preparing the models; second, building the ComfyUI workflow; and finally, experimenting with parameters such as CFG, denoise, and steps to refine quality.

Let’s start with the setup.

Before we dive into editing cityscapes and architectural images, let’s prepare a fully local environment. To begin with, this workflow runs entirely offline — no API keys, no cloud GPU, and no data uploads. As a result, you can experiment freely without any external dependencies.

Step 1 — Installing ComfyUI

ComfyUI is a visual AI workflow tool that lets you build image-generation and editing pipelines by connecting nodes instead of writing code. Think of it as a visual editor for generative models — you drag boxes, connect them with lines, and watch your AI process unfold step by step. It’s extremely powerful for image editing, concept design, and urban visualization, because you can control every part of the process: model loading, prompt conditioning, sampling, decoding, and saving. This makes it ideal for architects, urbanists, and researchers seeking privacy and control. On first launch, an empty grid appears as your workspace, ready to be filled with interconnected nodes for models, images, and prompts.

ComfyUI can be installed in two ways:

• Standalone version: downloaded from the official website and installed like a regular application. This version already includes the Node Manager, so you can install missing custom nodes directly from the interface.

• Git version: installed by cloning the repository (e.g., for advanced users or custom setups). In this case, you must install the Node Manager manually, because it is not included by default.

Next, open the v1.0 workflow for Qwen Edit for Urbanism. You can find it at the end of the post or in our Git repository. When the workflow loads, the Node Manager will automatically prompt you to install any missing custom nodes. Approve the installations, then close the workflow.

Step 2 — Add GGUF and Qwen Image Edit Support

Qwen-Image-Edit is a multimodal AI model capable of understanding both text and images, allowing image edits through natural language. The GGUF format makes it compact and memory-efficient, enabling faster processing on modest hardware.

Go to the ComfyUI-GGUF & Qwen Edit Utils project pages:
https://github.com/city96/ComfyUI-GGUF
https://github.com/lrzjason/Comfyui-QwenEditUtils

Clone the repositories, or click the green Code button and choose “Download ZIP” for both projects. After downloading, unzip each folder and place it directly into your custom_nodes directory—make sure the unzipped files are not nested inside an extra subfolder.

ComfyUI/custom_nodes/

Step 3 — Download the Required Models

You’ll need three model files, each serving a different purpose in the image-editing process. The first one is the core model that performs the actual visual transformation based on your text prompt.

You can find it here:

👉 https://huggingface.co/QuantStack/Qwen-Image-Edit-2509-GGUF/tree/main

On that page, you’ll see many versions of the same model — Q2, Q3, Q4, Q5, etc. These are quantized variants, meaning they trade a little precision for faster speed and lower VRAM usage. Here’s a quick guide to help you pick the right one:

The Qwen-Image-Edit model also needs two smaller files to work correctly — one for understanding your text prompts (CLIP) and one for turning the generated data back into a real image (VAE). You can download them from the official Comfy-Org repository on Hugging Face:

• Text Encoder (CLIP):
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders

  On that page, pick:
  👉 qwen_2.5_vl_7b_fp8_scaled.safetensors (about 9 GB — faster and works well locally)

• VAE Decoder:
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/vae

  Download the file:
  👉 qwen_image_vae.safetensors

Once downloaded, place all three models in:

ComfyUI/
 └── models/
      ├── gguf/
      │    └── Qwen-Image-Edit-2509-Q5_K_S.gguf
      ├── clip/
      │    └── qwen_2.5_vl_7b_fp8_scaled.safetensors
      └── vae/
           └── qwen_image_vae.safetensors

Verifying Your Setup

Open ComfyUI, right-click anywhere in the workspace or search in the node library, and check that you can find these nodes:

• • GGUF Loader – for loading the main Qwen-Image-Edit model (.gguf file)
  • CLIP Loader – for loading the CLIP text encoder
  • VAE Loader – for loading the VAE decoder
  • TextEncodeQwenImageEditPlus – for connecting text prompts and images

If they appear and can detect your downloaded files, congratulations 🎉 — your local setup is ready!

Now that all the models are downloaded and organized, you can start assembling the full workflow that makes Qwen-Image-Edit operate inside ComfyUI.
At this stage, the goal is to connect every component — the GGUF model, the text encoder (CLIP), the VAE decoder, the input image, and the text prompt — into a single functional chain. Once everything is linked, you’ll be able to type instructions such as “add shops on the ground floor” or “turn this street into a pedestrian zone” and watch ComfyUI generate updated images automatically.

Connecting the Model to Your Images and Prompts

Now that your model nodes are ready, it’s time to make them work together. You’ll connect your input image, prompt, and the Qwen-Image-Edit conditioning node so that ComfyUI can understand your instruction and modify the picture accordingly.

Right-click again and search for:

• Load Image – this node will import your input image (for example, a street or building photo).
• TextEncodeQwenImageEditPlus – this node combines your image with your text prompt

Adding the KSampler and the Other Required Elements

At this point, your setup has the model, encoders, and the prompt connection ready.
Now we’ll add the KSampler, which is the part that actually produces your final edited image, and all the other required elements.

Right-click again and search for:

• KSampler – this is ComfyUI’s main diffusion sampler. It takes in a latent (from the VAE) and conditioning (from Qwen) to generate a new image.
• Scale Image to Total Pixels – automatically resizes your image to a manageable resolution (set megapixels to 1.00).
• Get Image Size – extracts width and height from the resized image.
• EmptySD3LatentImage – creates an empty latent space matching the image dimensions, used as the canvas for generation.
• VAE Decode – converts the generated latent output back into a normal image.
• Save Image – saves your result to the ComfyUI output folder.

Your connections should look like this (check the figure for reference). Start by linking the Load Image node to Scale Image to Total Pixels, and then feed its output into Get Image Size. Next, connect the width and height outputs from Get Image Size to EmptySD3LatentImage, creating the correct latent dimensions. From there, route the latent output of EmptySD3LatentImage into the KSampler. In parallel, connect the model input of KSampler to the GGUF Loader, and feed the conditioning from TextEncodeQwenImageEditPlus. Finally, send the KSampler output into VAE Decode, and link that to Save Image to generate and store your final result.

Once these elements are connected, your workflow becomes fully functional. The scaled image is analyzed, encoded, transformed in the latent space according to your text prompt, and then decoded back into a visible edited image — all processed locally by your computer.

The core of the image-editing process lies in the KSampler node, where all generative inference occurs. Inside this component, several parameters control the quality, variability, and precision of the output. Understanding their effect is essential for achieving consistent and reproducible results.

To begin with, the seed parameter defines the random starting point of the diffusion process. Using the same seed with identical settings reproduces the same output, which helps when comparing the influence of other parameters. By contrast, changing the seed introduces controlled randomness, often generating new interpretations of the same instruction. Meanwhile, the control after generate option—set to increment—ensures each new image uses a slightly different seed, producing unique but related results.

Next, the steps value determines how many refinement iterations the sampler performs. Lower values such as five create quick previews and coarse adjustments, whereas higher counts (for instance, twenty or thirty) yield smoother and more detailed outcomes at the cost of longer processing time. In practice, Qwen-Image-Edit performs well even with fewer steps, since it relies heavily on prompt and image conditioning.

Similarly, the cfg parameter (classifier-free guidance) controls how closely the output follows the text prompt. A low value around 1.0 keeps changes subtle, while higher values push the model toward stronger, more literal transformations. Balancing this setting helps maintain realism without losing creative control.

As for the sampler_name and scheduler, they define how noise is reduced during diffusion. The Euler sampler offers an efficient trade-off between speed and visual quality, and the simple scheduler keeps results stable across seeds and image sizes.

Finally, the denoise value adjusts how strongly the latent image is modified. A setting of 1.0 applies a full transformation, producing bold edits, whereas smaller values retain more of the original features for subtle modifications. This parameter directly shapes the intensity of your edit—from a light retouch to a complete visual overhaul.

These results show the outcomes obtained with the final Qwen-Edit workflow, each generated from the same base image using a simple text prompt. Depending on the complexity of the requested transformation and the parameters defined in the KSampler—particularly the number of steps and the denoise factor—the processing time varied between 30 seconds and about 2 minutes on an RTX 4060 GPU.

• Qwen-Image-Edit official repository: https://huggingface.co/QuantStack/Qwen-Image-Edit-2509-GGUF

• ComfyUI documentation and examples: https://github.com/comfyanonymous/ComfyUI

• Qwen-Image-ComfyUI utilities and text encoders: https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI

• Zhang, F., Salazar-Miranda, A., Duarte, F., Vale, L., Hack, G., Chen, M., Liu, Y., Batty, M. & Ratti, C. (2024) ‘Urban Visual Intelligence: Studying Cities with Artificial Intelligence and Street-Level Imagery’, Annals of the American Association of Geographers, 114 (5), pp. 876-897. https://doi.org/10.1080/24694452.2024.2313515

• Perez, J. & Fusco, G. (2025) ‘Streetscape Analysis with Generative AI (SAGAI): Vision-language assessment and mapping of urban scenes’, GeoSpatial Analysis and Modelling, 100063. https://doi.org/10.1016/j.geomat.2025.100063 (ScienceDirect)

For convenience, you can download the ready-to-use ComfyUI JSON graph that we built in this post from the link below and load it directly into your workspace using File → Load → Workflow.

CSR, or Complete Spatial Randomness, is a theoretical model used in spatial statistics to describe a random distribution of points in space. Essentially, in a CSR process, points are distributed randomly across the study area, with no clustering or spatial patterns. Each point is independent of the others and has an equal chance of occurring at any location within the study area.

When comparing observed spatial patterns to CSR, it is important to note that if the observed pattern exhibits a mean nearest neighbor distance that is significantly smaller than the mean nearest neighbor distance of a random distribution (CSR), it suggests clustering. In other words, this means that points in the observed pattern are closer to each other on average than would be expected under a random process. Conversely, if the mean nearest neighbor distance of the observed pattern is significantly larger than that of a random distribution, it suggests dispersion. Consequently, this indicates that points in the observed pattern are farther apart on average than would be expected under a random process.

Calculating Nearest Neighbor Distance (NND) and comparing it with CSR can be useful in various fields where spatial patterns play a crucial role, such as ecology (distribution of species) or urban planning. Let’s see together how to calculate a nearest neighbor distance from a given point pattern and compare it to a random distribution (CSR).

To perform this analysis, we will use two building layers. You can obtain OSM building data from various locations worldwide using the Overpass Turbo tool. The sample data used in this demonstration consists of two OSM building layers compiled into a Geopackage file.

This sample data pertains specifically to the Italian cities of Grosseto and Sinalunga, each containing approximately 7500 buildings.

import geopandas as gpd
import numpy as np
from scipy.spatial.distance import cdist
import matplotlib.pyplot as plt

# Read the layers for the two Italian cities
gdf_1 = gpd.read_file("Italian_cities.gpkg", layer = "Grosseto")
gdf_1 = gdf_1.to_crs(epsg=6875)
gdf_2 = gpd.read_file("Italian_cities.gpkg", layer = "Sinalunga")
gdf_2 = gdf_2.to_crs(epsg=6875)

The provided code imports necessary libraries and extracts each layer from the GeoPackage. For detailed guidance on configuring your Python environment and managing libraries, you can consult this post. Additionally, if you seek information on GeoPackage usage and layer importation into Python, you can refer to this post.

# Create a figure with two subplots
fig, axs = plt.subplots(1, 2, figsize=(20, 10))

# Plot gdf_1 in the first subplot
gdf_1.plot(ax=axs[0], color='blue', edgecolor='black')
axs[0].set_title("Map View of Grosseto")
axs[0].set_xlabel("Longitude")
axs[0].set_ylabel("Latitude")

# Plot gdf_2 in the second subplot
gdf_2.plot(ax=axs[1], color='red', edgecolor='black')
axs[1].set_title("Map View of Sinalunga")
axs[1].set_xlabel("Longitude")
axs[1].set_ylabel("Latitude")

# Adjust layout
plt.tight_layout()

# Display the plot
plt.show()

This code snippet creates a figure with two subplots. Specifically, in the first subplot, the map view of Grosseto is plotted, while Sinalunga is shown in the second plot. At first glance, buildings in both maps look clustered. However, it is difficult to visually assess which pattern is more clustered or dispersed than the other.

# Calculate the centroids of gdf_1
gdf_1_points = gdf_1.centroid
points_geom = gpd.GeoDataFrame(geometry=gdf_1_points)
# Extract coordinates from the centroids
coordinates = np.array([[points.coords[0][0], points.coords[0][1]] for points in points_geom.geometry])

# Calculate the bounding box of the GeoDataFrame
min_x, min_y, max_x, max_y = gdf_1.total_bounds
np.random.seed(123)  # Set seed for reproducibility

# Generate random points within the bounding box
random_points_x = np.random.uniform(min_x, max_x, 1000)
random_points_y = np.random.uniform(min_y, max_y, 1000)
# Create a GeoDataFrame from the random points
random_points_geom = gpd.GeoDataFrame(geometry=gpd.points_from_xy(random_points_x, random_points_y))

# Extract coordinates of the random points
coordinates_random = np.array([[point.x, point.y] for point in random_points_geom.geometry])

# Plot the random points
ax = random_points_geom.plot(markersize=1)
ax.set_title('Random Points Plot')
plt.show()

# Function to calculate nearest neighbor distances
def nearest_neighbor_distances(points):
    # Calculate pairwise distances between points
    distances = cdist(points, points)
    np.fill_diagonal(distances, np.inf)  # Exclude distance to itself
    # Calculate nearest neighbor distances
    nearest_distances = np.min(distances, axis=1)
    return nearest_distances

# SD on the nearest neighbor distances
def nearest_neighbor_sd(points):
    # Calculate nearest neighbor distances
    nearest_distances = nearest_neighbor_distances(points)
    # Calculate standard deviation of nearest neighbor distances
    sd = np.std(nearest_distances)
    return sd

The nearest_neighbor_distances function calculates the nearest neighbor distances for a given set of points in a two-dimensional space. Specifically, it computes the pairwise distances between all points using the cdist function from SciPy’s spatial module. After that, by filling the diagonal of the distance matrix with infinity, it excludes distances to the points themselves. Consequently, it finds the minimum distance for each point, representing its nearest neighbor distance. Finally, the function returns an array containing the nearest neighbor distance for each point. This array is then used by the nearest_neighbor_sd function, which computes the standard deviation of nearest neighbor distances for a set of points.

mean_observed_nn = np.mean(nearest_neighbor_distances(coordinates))
mean_random_nn = np.mean(nearest_neighbor_distances(coordinates_random))
std_random_nn = nearest_neighbor_sd(coordinates_random)

# Calculate threshold of within CSR (-1/2 +1/2 sd)
lower_bound = mean_random_nn - 0.5 * std_random_nn
upper_bound = mean_random_nn + 0.5 * std_random_nn

# Assess clustering or dispersion
if mean_observed_nn < lower_bound:
    print(f"The observed point pattern (mean: ) is significantly clustered compared to the random pattern (mean: ).")
elif mean_observed_nn > upper_bound:
    print(f"The observed point pattern (mean: ) is significantly dispersed compared to the random pattern (mean: ).")
else:
    print(f"The observed point pattern (mean: ) is consistent with CSR (mean random: ).")

The observed point pattern (mean: 24.73) is significantly clustered compared to the random pattern (mean: 47.95).

The code calculates the mean nearest neighbor distances for both an observed set of coordinates and a randomly generated set. Additionally, it computes the standard deviation (sd) of the nearest neighbor distances for the randomly generated set. These values are then used to establish thresholds for Complete Spatial Randomness (CSR). To detect clustered and dispersed patterns, lower and upper bounds are set using statistical principles: approximately 68% of data points fall within one standard deviation of the mean in a normal distribution. Hence, these bounds are established using half the standard deviation away from the mean of the random distances.

It is then possible to assess whether the observed point pattern exhibits clustering, dispersion, or adherence to CSR. If the mean nearest neighbor distance of the observed pattern falls below the lower bound, it indicates significant clustering compared to the random pattern. Conversely, if it exceeds the upper bound, it suggests significant dispersion. Otherwise, if the mean observed distance lies within the bounds, the point pattern is deemed consistent with CSR. The buildings of the city of Grosseto exhibit a significantly clustered pattern as compared to a random pattern. But how another city like Sinalunga compares to Grosseto?

# Calculate the centroids of the GeoDataFrame
gdf_2_points = gdf_2.centroid
# Create a GeoDataFrame from the centroids
points_geom2 = gpd.GeoDataFrame(geometry=gdf_2_points)
# Extract coordinates from the centroids
coordinates2 = np.array([[points.coords[0][0], points.coords[0][1]] for points in points_geom2.geometry])
mean_observed_nn2 = np.mean(nearest_neighbor_distances(coordinates2))
# Compare mean nearest neighbor distances
if mean_observed_nn2 < mean_observed_nn:
    print("The values for the second city are more clustered.")
    print(f"Mean nearest neighbor distance for observed points in the first city: ")
    print(f"Mean nearest neighbor distance for observed points in the second city: ")
else:
    print("The values for the second city are more dispersed.")
    print(f"Mean nearest neighbor distance for observed points in the first city: ")
    print(f"Mean nearest neighbor distance for observed points in the second city: ")

The values for the second city are more clustered.
Mean nearest neighbor distance for observed points in the first city: 24.73
Mean nearest neighbor distance for observed points in the second city: 19.96

The code compares the mean nearest neighbor distances between the first city and the second city. If the mean distance for the second city is less than that of the first city, indicating greater clustering, a message is printed to convey this observation along with the respective mean distances for both cities. Conversely, if the mean distance for the second city is greater, suggesting more dispersion, another message is printed with the corresponding mean distances for both cities. The results suggest that the buildings of Sinalunga are more clustered than the ones in Grosseto.

# Plot histograms of nearest neighbor distances
plt.figure(figsize=(10, 6))
plt.hist(nearest_neighbor_distances(coordinates), bins=50, color='blue', alpha=0.5, label='Grosseto')
plt.hist(nearest_neighbor_distances(coordinates2), bins=50, color='green', alpha=0.5, label='Sinalunga')
plt.hist(nearest_neighbor_distances(coordinates_random), bins=50, color='red', alpha=0.5, label='Random CSR')
plt.xlabel('Nearest Neighbor Distance')
plt.ylabel('Frequency')
plt.title('Nearest Neighbor Distance Distribution')
plt.legend()
plt.show()

• Point Pattern Analysis – Geographic Data Science
• Yuan, Y., Qiang, Y., Bin Asad, K., and Chow, T. E. (2020). Point Pattern Analysis. The Geographic Information Science & Technology Body of Knowledge (1st Quarter 2020 Edition), John P. Wilson (ed.). DOI: 10.22224/gistbok/2020.1.13.
• Usui, H and Perez, J. (2020) « Are patterns of vacant lots random? Evidence from empirical spatiotemporal analysis in Chiba prefecture, east of Tokyo », Environment and Planning B: Urban Analytics and City Science, 49(3). Read more