A vision-language model loader for Python: one interface for 24 models across five families, to turn image collections into data and compare models with identical prompts.

UVLM in under a minute

Latest release: v4.2.0 (September 2026). Decoding is now greedy by default, so the same call gives the same answer every time. v4.1 added vision_budget, an optional control of each family’s vision-token budget, and every model now runs its native image preprocessing by default.

v4.0.0 (August 2026), UVLM’s first major version, added Gemma 4 as a fifth model family (E2B, E4B and 12B Instruct) and moved to Transformers 5; the registry now stands at 24 checkpoints across 5 families, from 1B to 110B parameters. Environments that must stay on Transformers 4.x can pin the last compatible release: pip install git+https://github.com/perezjoan/UVLM.git@v3.2.0. Release notes: v4.0.0, v3.2.0, v3.1.0, and VERSIONS.txt on GitHub for every version.

What is UVLM?

UVLM is an open-source Python package that provides a unified interface for loading, configuring, and evaluating multiple Vision-Language Model (VLM) architectures on custom image analysis tasks. It abstracts the substantial architectural differences between VLM families behind a single inference function, enabling researchers and practitioners to compare models using identical prompts and evaluation protocols, without writing model-specific code.

VLMs can interpret images and respond to arbitrary natural language queries about their content: counting objects, classifying scenes, estimating measurements, detecting features. But each model family requires its own processor classes, tokenisation logic, generation configuration, and output parsing. UVLM solves this by routing every inference call through a single backend-agnostic function, regardless of the underlying model.

Two uses stand out: turning an image collection into a table of data (one question per column, one row per image), and benchmarking several models on the same images with identical prompts, to choose the right one for a task.

Since v3.0.0 (April 2026), UVLM is a pip-installable package that runs anywhere Python runs: Google Colab, local Jupyter notebooks, or plain scripts. No GPU of your own is needed on Colab; for local use, any NVIDIA GPU with 8+ GB of VRAM handles models up to 7B parameters in 4-bit quantisation.

Architecture of UVLM, the unified vision-language model loader

Installation

Google Colab, zero install. Open the Colab notebook and everything is set up automatically.

Local install, one command: pip install git+https://github.com/perezjoan/UVLM.git

PyTorch with CUDA must be installed separately to match your GPU driver, for example pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128 for CUDA 12.8+.

How It Works

UVLM follows the same three-step workflow across all deployment modes:

1. Load a modelPick one of 24 checkpoints (7 LLaVA-NeXT, 4 Qwen2.5-VL, 4 Qwen3-VL, 6 InternVL3.5, 3 Gemma 4), from 1B to 110B parameters: family first, then model.
2. Define tasksUp to 10 tasks, through a form in notebooks or a list of dictionaries in scripts: a column name, a prompt (role + task + theory + format), a type (numeric, category, boolean, text) and optional consensus validation. Up to 1,500 tokens for custom reasoning.
3. Run the analysisPoint to an image folder (Google Drive on Colab, any local path otherwise). The batch engine writes results to CSV as it goes, resumes after an interruption, accepts new tasks between runs, saves every 3 images and flags truncated answers.
UVLM benchmark results comparing vision-language models on 120 street-level photographs

Three Deployment Modes

Google ColabOpen the notebook, select a GPU runtime and start working; the package installs itself in the first cell and images load from Google Drive. Best for: quick experiments, no setup, free GPU.
Local Jupyter notebookThe same form-based interface on your own hardware, with images read from local folders: your data never leaves your machine. Best for: sensitive data, faster model loading, reliable GPU access.
Python scriptFull programmatic control in three lines of code (below). Best for: pipeline integration, automation, batch scripts.
from uvlm import load_model, run_inference, parse_response
ctx = load_model(“[Qwen] Qwen2.5-VL 7B Instruct”, precision=”4bit”)
raw, tokens = run_inference(“photo.jpg”, “Count the cars”, ctx)
result = parse_response(raw, “numeric”)

Key Features

Multi-backend abstractionRoutes each call to the right pipeline (LLaVA-NeXT, Qwen2.5-VL, Qwen3-VL, InternVL3.5 or Gemma 4): five families, three architecturally distinct pipelines, one interface.
Multi-task prompt builderModular prompts (role + task + theory + format), four response types with type-specific parsers.
Consensus validationMajority voting across 2–5 repeated inferences per task, NA-aware filtering, agreement ratio tracking.
Flexible reasoningUp to 1,500 tokens for your own chain-of-thought prompts; a built-in reference mode at 1,024 tokens for standardised benchmarking.
Truncation detectionExact token counting from the model output; a per-task _truncated column and console warnings.
Quantization supportFP16, 8-bit and 4-bit via BitsAndBytes. Most models up to 14B run on T4 or L4 GPUs; 32B–38B models need an A100 (see the table).
Modular packageEight Python modules with clean APIs and no global state: easy to extend with new model families.

Supported Models

LLaVA-NeXT

Model Params Min GPU
Mistral 7B 7B T4 (free Colab)
Vicuna 7B 7B T4
Vicuna 13B 13B L4
34B 34B A100
LLaMA3 8B 8B T4
72B 72B Multi-GPU
110B 110B Multi-GPU

Qwen2.5-VL

Model Params Min GPU
3B Instruct 3B T4
7B Instruct 7B T4
32B Instruct 32B A100
72B Instruct 72B Multi-GPU

Qwen3-VL

Model Params Min GPU
2B Instruct 2B T4 (free Colab)
4B Instruct 4B T4
8B Instruct 8B T4
32B Instruct 32B A100

InternVL3.5

Model Params Min GPU
1B 1B T4 (free Colab)
2B 2B T4
4B 4B T4
8B 8B T4
14B 14B L4
38B 38B A100

Gemma 4

Model Params Min GPU
E2B Instruct ≈2.3B effective (≈5.1B raw) 8 GB, FP16 with CPU offload
E4B Instruct ≈4.5B effective (≈8B raw) L4 / A100
12B Instruct 12B A100

Gemma 4 E-series models use Per-Layer Embeddings: small effective parameter counts but large raw checkpoints. On 8 GB GPUs, run E2B in FP16 (the embedding tables offload to CPU RAM); 4-bit is not usable when offload is required.

Get Started

Open the notebook in Google Colab, select a GPU runtime, load a model, define your tasks and run; or install the package locally. The v3.0.0 blog post explains the move from notebook to Python package.

Citations & Publications

If you use UVLM in your work, please cite:

Perez, J. & Fusco, G. (2026). UVLM: A Modular Python Package for Unified Vision–Language Model Loading, Inference and Comparison. Software 2026, 5(3), 30. https://www.mdpi.com/2674-113X/5/3/30

Read the publication


Funding & Partners

UVLM is developed within emc2 (Evolutive Meshed Compact City), a research project of the Driving Urban Transitions (DUT) Partnership, co-funded by the European Union, with the support of CNRS.

emc2, Evolutive Meshed Compact CityDUT, Driving Urban TransitionsCNRSCo-funded by the European Union

© 2026 Joan Perez (Urban Geo Analytics) and Giovanni Fusco (CNRS). UVLM is open source, released under the Apache 2.0 license.