Cloud chat assistants are convenient, but they come with trade-offs: your queries leave your machine, you depend on someone else’s uptime and pricing, and the model’s behaviour can change under you without warning. For research, sensitive data, or simply full control, running a model locally is an appealing alternative. The good news is that modern quantization has made this accessible on modest consumer hardware. A capable 7–8 billion parameter model now fits comfortably on an 8 GB laptop GPU. This tutorial walks through the entire process end to end, using an NVIDIA Blackwell card (RTX 5060, 8 GB) as the worked example — though the approach applies to any recent NVIDIA GPU.

Everything starts with a clean, isolated environment. Mixing deep-learning dependencies into your base Python installation is a recipe for version conflicts, so we create a dedicated Conda environment for this project alone. If you have followed our earlier Anaconda setup guide, this will feel familiar.

Open the Anaconda Prompt and create a fresh environment:

The single most important, and most overlooked, step is installing the correct build of PyTorch for your specific GPU. This is where most local-LLM attempts fail silently. NVIDIA GPUs each have a “compute capability” (an architecture identifier such as sm_86, sm_90, sm_120), and a PyTorch binary only works if it was compiled with kernels for your card’s architecture. Install the wrong build and you will see CUDA reported as “available” while every actual GPU operation crashes — a particularly confusing failure mode.

The newest Blackwell cards (the RTX 50-series, including our RTX 5060) use compute capability sm_120, which older PyTorch wheels do not support. For these cards you need a build compiled against CUDA 12.8 or newer:

If you are on an older card (RTX 30- or 40-series), the standard CUDA 12.x wheels are fine. The general rule: match the PyTorch CUDA build to your GPU generation, and when a brand-new card isn’t yet supported in the stable channel, reach for the nightly build of the matching CUDA version.

Now verify it properly. Do not trust torch.cuda.is_available() alone — it can return True even when no compatible kernels exist. Instead, force an actual computation onto the GPU:

A clean OK cuda:0 (12, 0) with no warnings means real GPU compute is working. That is your green light. With the engine confirmed, install the rest of the stack and register the environment as a dedicated Jupyter kernel so the notebook always uses exactly these packages:

Finally, launch JupyterLab from your project directory so your notebook is rooted where you want it rather than in a system folder:

Open the localhost address provided by jupyter on your navigator and once inside, select the “Python (localllm)” kernel. We recommend JupyterLab over the classic Notebook here: it renders interactive widgets reliably out of the box, which matters for the chat interface we build in Section 3.

A model’s weights have to live in memory, and for modern LLMs they are large. A 7–8 billion parameter model in full 16-bit precision needs roughly 14–16 GB — too much for an 8 GB card. The solution is quantization: storing each weight in 4 bits instead of 16. This shrinks an 8B model to around 5 GB with only a minor quality cost, which is what makes local inference on consumer hardware possible at all.

We use Hugging Face Transformers together with the bitsandbytes library, which quantizes the model to 4 bits on the fly as it loads. This keeps everything inside your Python kernel — the model object lives in your notebook, you load directly from Hugging Face with optional token authentication, and you can inspect internals if you wish. Hugging Face acts as the model registry: the first load downloads the weights and caches them to disk (under your user folder’s .cache/huggingface), and every subsequent load reads from that local cache with no network access.

A note on model choice. There is no single “best” small model; it depends on your task and your memory budget. Here is a practical comparison for an 8 GB card:

• Qwen3 4B Instruct — the lightweight workhorse. Around 2.7 GB in 4-bit, very fast, strong reasoning and multilingual ability for its size. Ideal as a daily driver for quick questions.
• Dolphin 3.0 (Llama 3.1 8B) — a larger, more capable general-purpose model at around 5–5.5 GB in 4-bit. Built on Llama 3.1 and instruction-tuned by Cognitive Computations, it is designed to put alignment under the user’s control, making it well suited to research contexts where you define the system prompt and behaviour yourself.
• Other strong candidates — Phi-4-mini for very light tasks, and Gemma-class models for multilingual writing, depending on what fits your remaining VRAM.

The rule of thumb: pick the smallest model that does your job well. A 4B model runs noticeably faster than an 8B simply because there are fewer parameters to push through per token, so match model size to task.

The loading code configures 4-bit quantization and reads the model from Hugging Face. We wrap it in a small dropdown so you can switch models without rewriting code:

If you load a model built on a gated base (such as Llama), you may need to authenticate once with a Hugging Face token via huggingface_hub.login(). Most fine-tuned community models, including the two above, load without one.

A loaded model is, by itself, stateless. It has no memory of anything you said previously — each call only sees the text you hand it. To create the experience of a conversation, we must keep the history and re-send it on every turn. Understanding this is the key to using local models well, and it requires distinguishing three concepts that are easy to confuse.

The context window is the model’s hard architectural limit: the maximum number of tokens it can attend to at once, counting both the prompt and the output together. Llama 3.1-based models support up to 128k tokens. The conversation memory is not a property of the model at all — it is simply the running list of past turns that we re-inject into the prompt each time, and it consumes part of the context window. The max new tokens setting is a cap we choose on how many tokens the model may generate in a single reply; it controls output length only and does not affect what the model can read.

So the relationship is: the prompt (system message + accumulated history + your new question) plus the reserved output space must all fit inside the context window. The context window is the room; memory is the furniture already in it; max new tokens is the space you set aside for the answer.

A common misconception is that a larger context window makes the model faster. It is the opposite. A bigger active context costs more VRAM (the key-value cache grows) and runs slower, because each newly generated token must attend over every preceding token. Speed comes from keeping the active context small — short prompts and trimmed history — not large. Reducing max new tokens does not speed up generation either; it simply stops the reply earlier, often mid-thought, since the model does not plan around the limit. The right way to get shorter, faster answers is to instruct the model to be concise via a system prompt, so it produces a complete but brief response.

The widget below puts these ideas into practice. It keeps a chat_history list (the memory), trims it to a fixed number of recent turns (capping context growth), and displays a live VRAM gauge so you can see your headroom and know when to reset. Re-running the cell clears the history — that is your reset.

A few design notes. We use greedy decoding (do_sample=False) rather than random sampling: it is marginally faster and fully reproducible, with no meaningful quality loss for factual exchanges. The MAX_TURNS value is your direct control over how much the model “remembers” versus how lean and fast it stays. And the VRAM gauge turns green, amber, or red as memory fills, giving you a clear signal of when to start a fresh conversation.

What we have built is small but genuinely yours. Every conversation lives only in your computer’s memory, inside the running notebook kernel. Nothing is written to disk, nothing is sent anywhere, and nothing is logged. Close the kernel and the entire conversation simply vanishes — the only thing that persists is the downloaded model weights in your local cache. For sensitive research data, confidential analysis, or simply peace of mind, this is a meaningful difference from any cloud service.

Beyond privacy, running locally brings other advantages. You are not subject to per-token billing or rate limits, so you can experiment freely. You are insulated from silent model changes and deprecations — your model behaves the same tomorrow as it does today. And with community fine-tunes such as Dolphin, you control the system prompt and the model’s alignment yourself, rather than inheriting a one-size-fits-all policy. With fewer built-in guardrails, these models will engage with a wider range of legitimate research and technical questions, which can be valuable in specialist domains where general-purpose assistants are overly cautious — a freedom that naturally comes with the responsibility to use it sensibly.

This is only the foundation. In future posts we will extend this local assistant in several directions. We will give it web browsing, so it can retrieve current information rather than relying solely on its training. We will explore an expert mode, pre-loading the context with domain knowledge — for instance a corpus of spatial-analysis references — so the assistant answers as a specialist in your field. And we will look at containerizing the whole setup with Docker so it can be deployed on a dedicated GPU server or in the cloud, turning this notebook prototype into a private assistant you can embed directly in your own website.

For now, you have a capable, private language model running on hardware you already own, set up in about half an hour. Learn it, build on it, and apply it to your own work.

QGIS, a leading open-source Geographic Information System (GIS), provides an interesting suite of tools for geospatial analysis. Have you ever wondered about controlling QGIS with a Python script ? About calling a specific tool within QGIS from Python? This capability can be useful to integrate processing already available in QGIS within a script of your own without opening the QGIS software directly. In this blog post, we’ll explore how to call QGIS from a Python script in the Jupyter Notebook. Let’s dive in.

If you’re new to Python or Jupyter Notebook, check out our introductory guide to set up a Python environnement using Anaconda. In this tutorial, we assume that you are using the Anaconda distribution of Python. The first step is to open your Anaconda prompt windows and create a Conda environment that includes both QGIS and Jupyter Notebook. On Windows, click on the Start Menu and type “Anaconda Prompt” in the search bar and open it. Within the command prompt, run the line below:

Now, you can activate the newly created Conda environment using the following command, also in the Anaconda prompt windows. Once you activate the qgis_jupyter environment, you’ll have access to QGIS and Jupyter Notebook within the same environment, allowing you to use QGIS functionality directly from your Jupyter Notebooks. If you have an issue with the code above, it means that you need to install QGIS in your newly created environment. Thus, run this conda install -c conda-forge qgis line of code before proceeding further.

And finally, open a Jupiter Notebook within the environnement you just activated

Now that your notebook is open, you can run the code snippet below to set up the interaction with QGIS. Note that even if you don’t need to open the QGIS standalone, you need to have QGIS installed on your machine to interact with it. That’s why we need to specify in the code below the installation directory path of QGIS and initialize it. This ensures that the Python interpreter can locate the necessary QGIS libraries. Additionally, the script initializes the next processing algorithms that will be performed.

The layer contains the building footprints of Grosseto. Provided you put the GeoPackage in the folder where your notebook is located, you can already import and plot the layer, as shown below. For this example, we are working on a subset of 100 buildings.

Now, let’s execute a straightforward algorithm, such as « dissolve, » by invoking the QGIS tool directly within the notebook and visualize the outcome to verify its success. Ensure to specify the location of the Geopackage in the code, along with defining the input and output layers. Subsequently, call the algorithm using the native:dissolve identifier, and adjust its parameters as necessary. The result, named Dissolved_Subset_Grosseto is saved as a new layer in the GeoPackage and ploted in the notebook.

As you can see below the buildings have been perfectly dissolved using the native QGIS algorithm.

This code snippet below demonstrates a workflow in Python using the buffer QGIS algorithm. First, we reproject the input layer to a Cartesian Coordinate Reference System (CRS) suitable suitable for the buffer function. Subsequently, the script iterates through a series of distances, executing the buffer algorithm four times with varying distances (10, 20, 50, and 100 meters). Each buffer operation generates a new layer with the GeoPackage.

Let’s map the different buffers. The code below import the layers we just created with a loop, and plot them in reverse order in order to visualize the overlap using Matplotlib.

Feel free to engage with this post by commenting, asking questions, or providing feedback.

In this guide you’ll find clear instructions on setting up Python with Anaconda for spatial analysis. Then, we’ll cover installing Python alongside Anaconda and adding essential dependencies like GeoPandas via the Anaconda Prompt. Lastly, we’ll explore using the Jupyter Notebook for practical application. By the end, you’ll be ready to start your journey in Python-based spatial analysis, AI, and machine learning.

Anaconda is a powerful tool for managing Python environments. Indeed, Anaconda simplifies package management through its integrated package manager, conda. In addition, it comes bundled with a set of pre-installed libraries commonly used in data science and spatial analysis, such as NumPy, Pandas or Matplotlib. Therefore, this eliminates the need for manual installation and ensures immediate access to these libraries. Anaconda is also cross-platform compatible (available for Windows, macOS, and Linux) thus providing consistent Python environments across different operating systems. Furthermore, it seamlessly integrates with popular development environments like Jupyter Notebook, known amongst other things for its user-friendly interface. Visit the Anaconda website and download the Anaconda distribution related to your operating system (Windows, macOS, or Linux). Once Anaconda is installed, you’ll have access to the Anaconda Navigator, Anaconda Prompt, and other useful tools for managing Python environments.

Even if the Python distribution installed by Anaconda comes with numerous pre-installed libraries, additional dependencies will be required to enable performing advanced manipulations, and this is especially true for manipulating spatial data. So, let’s install the following three dependencies to manipulate spatial data : GeoPandas : allow spatial operations on geometric types ; Pyogrio : interoperability between spatial data formats and Contextily : retrieve tile maps from the internet.
Then, on Windows, click on the Start Menu and type “Anaconda Prompt” in the search bar and open it. This will open a new command prompt window with Anaconda enabled. Within the command prompt, run the following commands one by one to install the aforementioned dependencies.

conda install -c conda-forge geopandas 
conda install -c conda-forge pyogrio 
conda install -c conda-forge contextily

If you’ve previously installed the libraries, executing these lines will not only update the libraries themselves but also their dependencies. Additionally, any other dependencies not encompassed within the Anaconda distribution of Python can be installed using the same commands. For instance, for machine learning purposes, you can install ‘XGBoost’ (eXtreme Gradient Boosting) using these lines as well.

Now that you’ve installed Python and the required dependencies, you can open the Anaconda Navigator and launch the Jupyter Notebook from the Home tab, as follows:

Jupyter Notebook is an open-source web application that allows users to create and share documents containing live code, visualizations, and narrative text. Upon launching the notebook, you’ll be directed to the Jupyter Notebook explorer in your default web browser, where you can create folders and new notebooks with a simple right-click action, as demonstrated below.

Afterward, you can rename your notebook (indicated by arrow 1 below). In a notebook, users can compose and execute code in segmented blocks. The figure below illustrates one such block (indicated by arrow 2 below).

Your notebook is automatically linked to the files within the folders where it resides. Let’s add some data to this folder and import it into the notebook (or Python environment). Begin by downloading the dataset provided below. This dataset comprises a GeoPackage file containing two building layers corresponding to two small cities in Italy: Grosseto and Sinalunga. For further insight related the GeoPackage format, you can refer to this post. Then, once you have downloaded, place it in the same directory as your notebook.

Then, you can run the code below to import a layer in Python using GeoPandas. In this example, we are importing a layer of building related to the italian city of Grosseto.

Finally, you can run the code below to plot the building layer with a basemap from OSM using contextily.

The map above was generated using the Python code provided — demonstrating how to overlay geospatial data with a custom basemap using contextily. Fell free to provide feedbacks on our blog posts by contacting us.