vLLM D3 Tutorial: Run AI Decision Models Locally with Python

D3 is a family of open-weight decision models that return a probability for every answer instead of generating text. You can classify a support ticket, score an article, or check a video clip with a single request to a model running on your own machine, in as little as 87 milliseconds in our tests.

The family spans 6 models, from 600 million to 27 billion parameters. You can experiment with the smaller models on modest hardware and use the larger ones for more demanding tasks. D3 Edge is so lightweight that it can even run on compatible smartphones.

vLLM supports D3 through its Structured Decisions framework, so you can serve a model locally and call it through a simple application programming interface (API). The wider vLLM Semantic Router project adds routing and decision-making features for AI applications.

In this tutorial, I'll cover setting up D3 locally, the 3 decision types, and your first predictions. Then, we will build 3 practical projects:

1. AI Blog Quality Analyzer: Use the TypeSafe Python software development kit (SDK) and Olostep to classify and evaluate real articles.
2. Image Understanding: Analyze product images, identify categories, and evaluate image quality.
3. Video Understanding: Analyze short videos to detect movement and visual changes.

By the end, you will know how to run D3 locally and use it to build applications that make decisions across text, images, and video. You can also see our guides to running large language models locally and speeding up local inference on an RTX 3090 GPU.

In a nutshell

• D3 is a family of open-weight decision models that return probabilities for choice, yes/no (Noul), and score questions instead of generating text.
• D3 Mini (4B) handles text, images, and video, and runs on a single NVIDIA RTX 3090 graphics processing unit (GPU) with vLLM.
• You serve the model with vllm serve and call it through the TypeSafe Python SDK or the /v1/systemone endpoint.
• We build 3 projects: an article quality analyzer, a product image screener, and a video motion analyzer.
• Responses in our tests took between 87 milliseconds and 1.11 seconds.

What Are D3 Decision Models?

D3 is a family of open-weight AI models built for making decisions rather than generating text. Instead of asking a large language model (LLM) to explain whether a transaction is suspicious, D3 can directly return a probability that it needs review.

Source: vllm-sr/d3-mini · Hugging Face

It supports 3 decision types:

D3 comes in 6 sizes, from 600 million to 27 billion parameters, which suits everything from edge devices to high-end GPUs. You can browse the family in the Decision 3.0 collection on Hugging Face.

The models are evaluated on decision-focused benchmarks such as Typed Decisions, which measure classification accuracy and probability quality.

For this tutorial, we will use D3 Mini (4B) because it supports text, image, and video understanding while remaining small enough to run on a local NVIDIA RTX 3090 GPU.

How to Run D3 Locally With vLLM

vLLM Semantic Router (vLLM SR) is an open-source project in the vLLM ecosystem that helps AI applications make intelligent decisions. It supports model routing, safety checks, semantic caching, and specialized decision models like D3. While vLLM handles model inference, vLLM SR focuses on how requests are evaluated and routed.

In this tutorial, we will use vLLM's new Structured Decisions support to run D3 Mini locally without installing the Semantic Router itself. vLLM serves D3 through the System One API, an endpoint named after fast, intuitive "System 1" thinking. It answers typed questions in a single pass and returns probabilities instead of generated text.

We will connect to the model using the TypeSafe Python SDK and explore how it handles classification, probability estimation, and scoring.

1. Install vLLM and run D3 locally

We will use Ubuntu Linux, Python 3.12, and an NVIDIA RTX 3090 GPU. Windows users can follow along using Ubuntu through Windows Subsystem for Linux 2 (WSL2).

First, install uv. Once the installation is complete, restart your terminal so the uv command is available.

curl -LsSf https://astral.sh/uv/install.sh | sh

Now, create a Python 3.12 environment and activate it:

uv venv --python 3.12 --seed
source .venv/bin/activate

Next, install the vLLM nightly build, which includes the new Structured Decisions support:

uv pip install -U vllm --torch-backend=auto \
  --extra-index-url https://wheels.vllm.ai/nightly

Now start the D3 Mini model with GPU memory settings suitable for local testing:

vllm serve vllm-sr/d3-mini \
  --gpu-memory-utilization 0.75 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --enforce-eager

The first launch downloads the model from Hugging Face. Once loaded, vLLM exposes the local System One API at http://localhost:8000/v1/systemone.

Keep this terminal running. We will access the model from a Jupyter notebook.

2. Connect to D3 using the TypeSafe Python SDK

Instead of manually constructing HTTP requests, we will use the TypeSafe Python SDK to communicate with our locally running model.

Install the required libraries:

%pip install typesafe-sdk olostep

Import the 3 decision types and configure the client to connect to our local vLLM server:

from typesafe_sdk import TypeSafeClient, Choice, Noul, Score

client = TypeSafeClient(
    base_url="http://localhost:8000",
    api_key="local",
    model="vllm-sr/d3-mini"
)

We use a placeholder API key because the local server does not require a hosted TypeSafe credential.

3. Make your first AI decision

Let's start with a simple fraud detection example. We will give D3 an unusual transaction and ask whether it should be flagged for review.

%%time
result = client.system_one(
    state="""
    A customer usually spends $20-100 in the US.
    A new $2,500 transaction was made from another country.
    """,
    questions={
        "suspicious": Noul(
            instructions="Should this transaction be flagged for fraud review?"
        )
    }
)

print(result.nouls["suspicious"].noul)

Output:

0.9990723773071258
CPU times: user 1.89 ms, sys: 2.01 ms, total: 3.91 ms
Wall time: 87.4 ms

D3 returns a probability of approximately 99.9%, indicating that the transaction should be flagged for review. The inference took around 87 milliseconds.

We can also combine all 3 decision types in a single request. This time, we will evaluate a job candidate and ask D3 to identify the best role, check their qualifications, and score their suitability.

%%time
result = client.system_one(
    state="""
    Candidate: 4 years of Python experience, 2 years working
    with machine learning, and strong SQL skills.
    No experience with frontend development.
    Available roles: Data Scientist, Backend Developer,
    Frontend Developer.
    """,
    questions={
        "role": Choice(
            instructions="Which role best matches the candidate?",
            criteria={
                "data_scientist": "Python, SQL, and machine learning",
                "backend": "Python, APIs, and backend systems",
                "frontend": "JavaScript, React, and UI development"
            }
        ),
        "qualified": Noul(
            instructions="Does the candidate meet the basic requirements for the best-matching role?"
        ),
        "fit": Score(
            instructions="How well does the candidate match the best-fitting role?",
            criteria=["Low", "Medium", "High"]
        )
    }
)

print("Best Role:", result.choices["role"].choice)
print("Qualified:", result.nouls["qualified"].noul)
print("Candidate Fit:", result.scores["fit"].score)

Output:

Best Role: data_scientist
Qualified: 0.9446079177060042
Candidate Fit: 1.6421276950866912
CPU times: user 4.18 ms, sys: 3.11 ms, total: 7.29 ms
Wall time: 165 ms

The model correctly identifies Data Scientist as the best match, assigns a 94.5% probability of qualification, and gives a fit score of 1.64 on a 0 to 2 scale. All 3 decisions were returned in just 165 milliseconds.

Now that we have D3 running locally, let's use it to build 3 practical AI applications.

Project 1: Build an AI Blog Quality Analyzer With D3 Mini

Imagine managing a technical publication where dozens of articles need to be reviewed before publishing. Instead of reading every article manually, we can build an AI quality analyzer that uses Olostep to extract website content and our locally running D3 Mini model to classify and score it.

We will test it on a real article, our KTransformers tutorial, which I wrote.

1. Scrape our KTransformers tutorial

First, we will use the Olostep Python SDK to extract the article content as Markdown.

from getpass import getpass
from olostep import Olostep

olostep = Olostep(api_key=getpass("Enter your Olostep API key: "))

url = "https://www.datacamp.com/tutorial/ktransformers-tutorial"
page = olostep.scrapes.create(
    url_to_scrape=url,
    formats=["markdown"]
)

content = page.result.markdown_content
print(content[:1000])

Olostep retrieves the article as Markdown, preserving its headings, text, and code examples for analysis.

2. Categorize and rate the article using D3

Next, we will use Choice to identify the article's category and Score to evaluate its educational quality on a 5-point scale.

%%time
result = client.system_one(
    state=content[:12000],
    questions={
        "category": Choice(
            instructions="What is the main topic of this article?",
            criteria={
                "ai": "Artificial intelligence and LLMs",
                "data_science": "Data science and analytics",
                "programming": "Software development",
                "other": "Other technical topics"
            }
        ),
        "quality": Score(
            instructions=(
                "Rate the article's educational quality based on "
                "clarity, technical depth, practical examples, "
                "and usefulness to readers."
            ),
            criteria=[
                "Very poor: Confusing and lacks useful information",
                "Poor: Limited depth and unclear explanations",
                "Average: Understandable but missing important details",
                "Good: Clear explanations and useful examples",
                "Excellent: Detailed, practical, and easy to follow"
            ]
        )
    }
)

category = result.choices["category"].choice
quality = result.scores["quality"].score + 1

print("Category:", category)
print(f"Quality: {quality:.1f}/5")

Output:

Category: ai
Quality: 4.6/5
CPU times: user 3.29 ms, sys: 2.96 ms, total: 6.25 ms
Wall time: 1.11 s

D3 correctly categorizes the article as AI and gives it a 4.6/5 quality score, completing the analysis in approximately 1.11 seconds.

Project 2: Analyze a Product Image With D3 Mini

In this project, we will build a simple product image screening workflow for an e-commerce store. The goal is to use D3 Mini to identify the product category, check whether the product is clearly visible, and rate the image quality before it gets added to a catalog.

The following code uses the vLLM System One API directly and reads results from the correct answers field.

1. Load a product image

First, we will download a sample product image, display it in the notebook, and convert it into base64 so it can be sent to the model.

import base64
import requests
from IPython.display import Image, display

image_url = "https://images.unsplash.com/photo-1542291026-7eec264c27ff?w=600"
image_data = requests.get(image_url, timeout=20).content

display(Image(data=image_data))

image_base64 = base64.b64encode(image_data).decode()
image_input = f"data:image/jpeg;base64,{image_base64}"

This gives us a ready-to-send image input that D3 Mini can analyze directly.

2. Analyze the image using D3 Mini

Next, we will ask D3 Mini 3 things at once: what category the product belongs to, whether the product is clearly visible, and how suitable the image is for an online store.

%%time
result = client.system_one(
    state="Review this product image for an online store.",
    images=[image_input],
    questions={
        "category": Choice(
            instructions="What product category is shown?",
            criteria={
                "footwear": "Shoes, sneakers, or boots",
                "clothing": "Shirts, jackets, or trousers",
                "electronics": "Electronic devices",
                "other": "Other products"
            }
        ),
        "product_visible": Noul(
            instructions="Is a physical product clearly visible?"
        ),
        "quality": Score(
            instructions="Rate the image suitability for an online store.",
            criteria=["Poor", "Average", "Good", "Excellent"]
        )
    }
)

print("Category:", result.choices["category"].choice)
print("Product Visible:", result.nouls["product_visible"].noul)
print("Image Quality:", result.scores["quality"].score)

Output:

Product Category: footwear
Product Visibility: 97.11%
Image Quality: 2.30/3
CPU times: user 4.91 ms, sys: 952 μs, total: 5.86 ms
Wall time: 282 msPowered by

The model identifies the item as footwear, confirms that the product is clearly visible, and gives the image a solid quality score in just 282 milliseconds.

Project 3: Build a Video Motion Analyzer With D3 Mini

In our final project, we will build a simple video motion analyzer using D3 Mini. Instead of generating lengthy video descriptions, we'll use the model to understand what happens in a short clip, identify the direction of movement, and detect visual changes. This approach can be useful for video monitoring, event detection, and automated content analysis.

1. Download a sample video

We will use the official D3 Mini sample video from Hugging Face. First, install the required library:

%pip install huggingface_hub

Next, download the sample video so we can analyze it locally.

from huggingface_hub import hf_hub_download
from IPython.display import Video, display

video_path = hf_hub_download(
    repo_id="vllm-sr/d3-mini",
    filename="assets/example-video.mp4"
)

The video shows a box moving from left to right while changing color from blue to green.

2. Analyze the video

First, we will convert the video into base64 format so it can be sent to our local vLLM server.

import base64
import requests

with open(video_path, "rb") as f:
    video_base64 = base64.b64encode(f.read()).decode()

video_input = f"data:video/mp4;base64,{video_base64}"

Now, we will use the System One API to ask D3 Mini 2 questions: which direction the square moves and whether it changes color.

%%time
response = requests.post(
    "http://localhost:8000/v1/systemone",
    json={
        "model": "vllm-sr/d3-mini",
        "state": "Analyze the movement of the object in this video.",
        "videos": [video_input],
        "questions": {
            "direction": {
                "type": "choice",
                "instructions": "Which direction does the square move?",
                "criteria": {
                    "left": "From right to left",
                    "right": "From left to right",
                    "still": "The square does not move"
                }
            },
            "color_change": {
                "type": "noul",
                "instructions": "Does the square change color during the video?"
            }
        }
    },
    timeout=120
)
response.raise_for_status()

answers = response.json()["answers"]

print("Movement:", answers["direction"]["choice"])
print(f"Color Change: {answers['color_change']['noul']:.1%}")

Output:

Movement: right
Color Change: 93.7%
CPU times: user 3.01 ms, sys: 5.04 ms, total: 8.05 ms
Wall time: 723 ms

D3 Mini correctly identifies the movement as right and assigns a 93.7% probability to the color change, completing the analysis in just 723 milliseconds.

Final Thoughts

D3 Mini handles classification, yes/no checks, and scoring across text, images, and video on a single NVIDIA RTX 3090 GPU. Setting up D3 locally was surprisingly easy. The only issue I ran into was a CUDA out-of-memory error, which I fixed by reducing GPU memory consumption.

Other than that, everything worked smoothly. If you're already familiar with vLLM, you can probably optimize the setup even further.

What impressed me most was the speed. Wall times ranged from 87 milliseconds for a short text decision to 1.11 seconds for a full article, and the video example finished in 723 milliseconds. All of it ran on my own GPU.

I think we're reaching a point where running decision models on a local server for an entire team is becoming practical. Not every request needs a large reasoning model.

For many everyday tasks, such as classification, routing, scoring, and content filtering, a smaller decision model can do the job. You can then reserve expensive reasoning models for the tasks that actually require deeper analysis or detailed explanations.

Of course, you'd still want to evaluate accuracy and performance on your own workloads before using these models in production. After testing D3 across text, images, and video, my rule of thumb is simple: if the task is a classification, a yes/no check, or a rating, try a D3 model first, and move to a reasoning model only when you need an explanation.

If you're keen to learn more about working with open models, I recommend checking out our Working with Hugging Face course.