Starting from version 0.35, Ollama supports decision models built on the Jev API from TypeSafe. These are not chat models in the usual sense: they do not write text, but answer a set of typed questions about the transferred state. For classification, routing, moderation and triage, this turns out to be faster and more reliable than asking a regular LLM to return JSON and then parsing its response.
In this post, I’ll look at what decision models are, how they work at the HTTP level, what models are already available, and how to build them into existing code so that regular generative models continue to work.
How do decision models differ from regular ones
A regular LLM receives a prompt and generates text token by token. Even if you ask her to answer in one word, generation still occurs inside, and the format of the answer has to be controlled by a diagram, instructions and subsequent verification. The Decision model works differently: you pass state and questions with types, and the model returns probabilities and selected options. No free text, no parsing JSON from the response.
The new endpoint is called /v1/systemone. In one request, you can ask several questions at once, and the model will answer all of them. This is convenient for tasks where you need to make several related decisions: for example, determining the team, the need for a return, and the urgency of the ticket.
What models are available
Now you can run three decision models through Ollama:
- nimble is an open model with 9 billion parameters from Bespoke Labs. It is the largest and most accurate available.
- tev1 is an experimental model with 4 billion parameters from Together AI.
- tev1:0.8b is an experimental model with 0.8 billion parameters from Together AI. The fastest and easiest.
They all work locally, do not require payment and do not send data to an external service. Ollama 0.35 and later are supported.
Installation
First check the Ollama version:
ollama --version
Then download the desired model. For the easiest one:
ollama pull tev1:0.8b
For the full version of tev1:
ollama pull tev1
For nimble:
ollama pull nimble
The nimble model takes up about 9.5 GB of disk space, so it’s a good idea to clear some space before downloading. tev1:0.8b takes up less than a gigabyte.
First request via curl
The simplest example is to determine whether a text is a job advertisement:
curl http://localhost:11434/v1/systemone -d '{
"model": "tev1:0.8b",
"state": {
"message": "Senior Swift developer, Berlin, remote friendly"
},
"questions": {
"isJob": {
"type": "noul",
"instructions": "Does the message advertise a job posting that someone can apply to?"
}
}
}'
The answer looks like this:
{
"model": "tev1:0.8b",
"answers": {
"isJob": {
"type": "noul",
"noul": 0.921
}
},
"usage": {
"input_tokens": 42,
"output_tokens": 0
}
}
The noul field is a probability from 0 to 1. The decision is made by comparison with a threshold, usually 0.5. Unlike the generative model, there is no need to ask for a response in JSON format and then check that the model did not break it.
Types of questions
Decision models support three types of questions.
noul is a binary question. Returns the probability that a statement is true:
{
"questions": {
"refund": {
"type": "noul",
"instructions": "Does the customer explicitly ask for a refund?"
}
}
}
choice – selecting one option from a given set. Returns the selected option, the probabilities of all options and confidence:
{
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "Payments and refunds",
"technical": "Bugs and integrations",
"other": "None of the above"
}
}
}
}
score – rating on a scale. Returns the number, legend, and probabilities for each scale point:
{
"questions": {
"urgency": {
"type": "score",
"instructions": "How urgent is this ticket?",
"criteria": ["Routine", "Soon", "Urgent"]
}
}
}
One request can contain several questions of different types at once. This is what makes decision models convenient for triage: one operation, one state, several independent decisions.
Python without additional libraries
For requests, the standard urllib is sufficient; a separate SDK is not required:
import json
from urllib.request import Request, urlopen
OLLAMA_DECISION_URL = "http://localhost:11434/v1/systemone"
def ask_decision(state, questions, model="tev1:0.8b"):
payload = {
"model": model,
"state": state,
"questions": questions,
}
request = Request(
OLLAMA_DECISION_URL,
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
with urlopen(request, timeout=3600) as response:
return json.loads(response.read().decode())["answers"]
Usage:
answers = ask_decision(
{"message": "Senior Swift developer, Berlin, remote friendly"},
{
"isJob": {
"type": "noul",
"instructions": "Does the message advertise a job posting that someone can apply to?"
}
},
)
probability = answers["isJob"]["noul"]
is_job = probability >= 0.5
If you need an official Python SDK, it is called typesafe-sdk. For localhost, it can be directed to Ollama with the environment variables TYPESAFE_BASE_URL, TYPESAFE_API_KEY and TYPESAFE_DEFAULT_MODEL. But for simple tasks, a direct HTTP request is sufficient.
How to understand that a model supports decision
Decision models cannot respond via /api/generate or /api/chat. If you send such a request, Ollama will return a 400 error with text like:
"tev1:0.8b" does not support generate
This is important if you already have code that works with regular models via /api/generate: if you change the model to a decision model, it will stop working. It’s better to find out the model’s capabilities in advance via /api/show and choose the appropriate calling method:
import json
from urllib.request import Request, urlopen
OLLAMA_SHOW_URL = "http://localhost:11434/api/show"
def supports_decision(model):
request = Request(
OLLAMA_SHOW_URL,
data=json.dumps({"model": model}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urlopen(request, timeout=30) as response:
capabilities = json.loads(response.read().decode()).get("capabilities", [])
except (OSError, TypeError, ValueError):
return False
return "decision" in capabilities
Decision models will have only decision in their capabilities. Regular models will have completion, tools or other features there. If Ollama is not available, it is reasonable to consider the model normal and use the generative path.
How to embed into existing code
The simplest way is not to rewrite the classifier, but to divide it into two strategies. The generative strategy works as before: it collects the prompt, sends it to /api/generate and parses the JSON. The decision strategy sends the state and typed question to /v1/systemone and compares the probability with a threshold.
The choice of strategy occurs in one place, when creating a classifier:
def create_classifier(model):
if supports_decision(model):
return DecisionClassifier(model)
return GenerativeClassifier(model)
This approach has several advantages. Generative models continue to run unchanged. The decision model can be enabled by changing one environment variable. And the entire difference between the two methods is concentrated in two small classes, and is not spread out throughout the application.
For a binary classification, the decision strategy looks something like this:
async def classify(text):
answers = await ask_decision(
{"message": text},
{
"isJob": {
"type": "noul",
"instructions": instructions,
}
},
model,
)
probability = answers["isJob"]["noul"]
return probability >= 0.5, probability
Please note that the decision model does not return an explanation. If your result format has a reason field, you can construct it from the probability: for example, store the probability itself or a string like “the model rated the message at 0.935 as a vacancy.” This is more honest than inventing a reason that the model did not give.
Instructions should be a question
This is the most important practical conclusion. Decision models do not understand long prompts in the style of “you classify messages, answer true if…, false otherwise.” They want a short, direct question.
On the same message about a vacancy, I received the following probabilities:
- Direct question “Is this message a job posting?” — 0.921.
- Long instructions explaining what is considered a vacancy – 0.026.
- Long instructions with a question at the end – 0.391.
The difference is almost thirty times. The model is not mistaken in classification, it simply does not understand what they want from it when the question is hidden inside a paragraph. Therefore, for decision models it is worth keeping separate short instructions, and leaving long prompts for generative models.
It is convenient to create a separate environment variable:
OLLAMA_JOB_CLASSIFIER_DECISION_INSTRUCTIONS=Does the message advertise a job posting that someone can apply to?
And use a regular prompt only for the generative path.
Threshold and probabilities
The threshold of 0.5 is suitable as a default value, but it is useful to include it in the configuration. For different tasks, the cost of a false positive and false negative response is different. For moderation, it is safer to raise the threshold; for searching for vacancies, it is safer to lower it, so as not to miss useful messages.
It is also useful to log the probability itself, and not just the resulting boolean. When the classifier makes an error, the log shows whether it was a confident answer or a decision on the border. For choice questions, it is worth saving probabilities and confidence: they show how much the model fluctuates between options.
Performance and memory
The speed greatly depends on the size of the model and the hardware. On the CPU in my tests I got the following numbers:
- tev1:0.8b – split seconds to solve, easily fits into memory.
- tev1 – 4B, noticeably slower, but still suitable for interactive tasks.
- nimble – 9B in Q8_0, about 10 GB in memory. The first request, including loading the model, took about 137 seconds, subsequent solutions took from 12 to 32 seconds on the CPU.
Nimble requires a lot of RAM. On a machine with 16 GB and Docker/WSL with a limit of about 8 GB, the model does not fit: the system begins to swap and the containers stop responding. Increasing the WSL limit to 12 GB via the .wslconfig file helped:
[wsl2]
memory=12GB
swap=8GB
After the change, you need wsl –shutdown and restart Docker Desktop. But even with 12 GB, the reserve remains small: the model takes up about 10 GB, and there is not much left for the system.
By default, Ollama unloads the model after five minutes of inactivity, freeing up memory. This is convenient: nimble does not keep 10 GB busy all the time, but each new launch after a pause starts loading the model again.
When to use decision models
Decision models are well suited for tasks where you need a quick and cheap answer to a clearly formulated question:
- Classification of messages, tickets and requests.
- Routing by commands or executors.
- Content moderation and security checks.
- Fast binary and multivariate solutions in gaming or interactive applications.
They are poorly suited for tasks where detailed text is needed: explanation, field extraction, summarization, answer generation. For this purpose, the usual generative models remain. The best result is obtained when combined: a decision model makes a decision, and a generative model explains it or extracts data.
Total
Decision models in Ollama are a separate, narrower tool than regular LLMs. They don’t replace generative models, but they take away the routine part of the work: typed responses, probabilities instead of text parsing, and a predictable format. If you already have a classification pipeline, the transition comes down to checking the capabilities of the model, a separate calling strategy, and short instructional questions.
Sources
https://ollama.com/blog/ollama-now-supports-jev-style-decision-models
https://ollama.com/library/nimble
https://ollama.com/library/tev1
https://typesafe.ai
Leave a Reply