Running Models Locally

I have another laptop with fairly decent specs, so when I have nothing to do I run some simple small models on it.

This is the configuration I recorded earlier:

=== Hardware Configuration ===
CPU: Intel(R) Core(TM) Ultra 5 125H
Cores: 18 logical cores
Memory: 30Gi
GPU: NVIDIA GeForce RTX 4050 Laptop GPU
VRAM: 6141 MiB
Disk: nvme0n1 953.9G

VRAM is about 6GB, so I have to be a bit restrained when picking models. Parameter count, quantization method, context length, and image resolution all affect usage; just because a model file can be downloaded does not mean it will necessarily fit during inference.

What follows is the deployment method and the verification order, not measured results from this laptop. Specific speed and VRAM usage have to be recorded under the same input; upstream results cannot be used in their place.

1. First, distinguish the relevant tools

• Ollama: loads models and provides the inference interface.
• Open WebUI: provides the chat interface and connects to an inference service that is already running.
• nvidia-smi: checks the driver, VRAM, and processes in an NVIDIA environment.

Get Ollama working on its own first, then connect the interface. That way, when something goes wrong, it is easier to tell whether the model failed to load or the interface failed to connect.

ollama --version
nvidia-smi

Installation and system support follow the official Ollama documentation; if macOS has no NVIDIA GPU, there is no need to run nvidia-smi.

2. Qwen3-VL-2B-Instruct

This is a vision-language model that can accept text and images at the same time. Start with the small Instruct version, which is suitable for beginning with screenshot descriptions and simple document recognition.

Ollama’s model page lists this exact tag and the minimum version requirement: Qwen3-VL requires Ollama 0.12.7 or later. Do not just write qwen3-vl here; otherwise, when the default tag changes later, different models will get mixed together.

ollama pull qwen3-vl:2b-instruct
ollama show qwen3-vl:2b-instruct
ollama run qwen3-vl:2b-instruct

First send an ordinary text question and confirm that it responds normally, then exit the interaction to try an image.

Sending an image through the local interface

Save the following as vision_request.py. Put a test screenshot in the same directory and name it sample.png. This uses the Python standard library, so no extra SDK needs to be installed.

import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

image = base64.b64encode(Path("sample.png").read_bytes()).decode("ascii")
payload = {
    "model": "qwen3-vl:2b-instruct",
    "messages": [{
        "role": "user",
        "content": "提取图片中能够看清的文字，保持行顺序。看不清的地方标注不确定，不要补写。",
        "images": [image],
    }],
    "stream": False,
    "options": {"num_ctx": 4096, "num_predict": 512},
}
request = Request(
    "http://127.0.0.1:11434/api/chat",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
)
with urlopen(request, timeout=180) as response:
    result = json.load(response)
print(result["message"]["content"])

In the REST interface, images must be passed Base64 data rather than a local file path; SDK usage is not necessarily the same, see Ollama vision. The context and maximum output are set small here to verify the flow first, not because that is the model’s capability limit.

After running it, you can open another terminal and check ollama ps, and in an NVIDIA environment also check nvidia-smi. Do not just look at whether “an answer came out”: compare against the original image and check the numbers, punctuation, and line breaks, especially whether the model guessed at content that was not clear.

3. Then connect Open WebUI

The original notes used the Docker installation method. Docker’s advantage is that the environment is easy to isolate, but it adds a networking problem: localhost inside the container points to the container itself, not to the Ollama on the host machine.

Below is a method that runs on the same machine, to reduce network variables. The terminal commands are written for Bash; the virtual environment activation script is different on Windows.

mkdir local-model-ui
cd local-model-ui
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install open-webui
DATA_DIR="$PWD/data" OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  open-webui serve --host 127.0.0.1 --port 13000

Open http://127.0.0.1:13000, complete the first-time account setup, and select the model you just downloaded from the model list. The data directory is fixed at data in the current project; keep it before upgrading or changing environments.

The official quick start currently supports Python 3.11 and 3.12; this uses 3.11. The port parameters come from the official CLI.

If you stick with Docker, handle the data volume, fixed secret key, and Ollama’s host address according to the same official documentation. Container-to-host networking differs between Docker Desktop and Linux Docker; host.docker.internal cannot be treated as an address that works directly on every system.

4. DeepSeek-OCR 2

OCR 2 is mainly used for document recognition, and its purpose is not exactly the same as that of a general-purpose image-text chat model. Testing first with a single page of clear text, then moving on to tables, multi-column layouts, and scans, makes it easier to pinpoint problems.

You cannot just put an Ollama model tag on it simply because it has DeepSeek in its name. The official repository provides two inference paths, Transformers and vLLM. The upstream reference environment uses Python 3.12.9, PyTorch 2.6.0, and CUDA 11.8, and also involves FlashAttention; these dependencies should be set up in a separate environment, apart from the WebUI environment above.

The specific order is: first prepare the dependencies according to the upstream installation section, download the official model, modify the input and output paths, and then run the single-page script. The Transformers entry point is:

DeepSeek-OCR2-master/DeepSeek-OCR2-hf/run_dpsk_ocr2.py

Run it in the corresponding directory of the repository:

python run_dpsk_ocr2.py

The model comes from the deepseek-ai/DeepSeek-OCR-2 model card. Read carefully how the code loads the model and what the image parameters are before running it. Whether 6GB of VRAM is enough for a given configuration needs to be measured in practice; no “it will definitely run” conclusion is given here.

If an OOM occurs, first record whether the error happened while loading the weights or while processing an image, then reduce the resolution, the number of crops, or the batch size. Lowering the image resolution may also lose small text, so recognition quality has to be checked at the same time.

5. How do you tell whether it counts as working?

Prepare a few images whose answers you can check yourself: an ordinary screenshot, a table with many numbers, and a multi-column document. Keep the original images, prompts, model tags, and run configuration.

Change only one condition at a time, then record:

• Whether any lines of text are missing, and whether the numbers and decimal points are correct.
• Whether table cells, column order, and paragraph order are preserved.
• Whether anything uncertain was filled in anyway.
• The actual run time, peak VRAM, and whether the CPU shared the computation.

Get a result you can reproduce first, then consider connecting a knowledge base, batch PDFs, or an automated pipeline. Otherwise, even if the interface looks usable, you will not necessarily know which inputs cause the model to go wrong.