Decision Models: Prompt Injection Screening with Classifier Probabilities

Overview

A decision model is a classifier that answers a fixed question and gives a probability for each possible answer. A text model, such as the ones on the Providers tab, writes free text. A decision model can't. That makes its answers easy to record, replay and measure.

GraphnAI ships one question today, Instructions aimed at AI. It screens for indirect prompt injection: text an attacker could write, such as a directory description, before other AI features read it. A text that tells an AI what to do, what to ignore or how to classify something is held for review. Other questions that read attacker-written text wait for this screen.

Each question has a stage: Off, Shadow mode, Advisory or Automated. Every question starts Off, and you choose how far it goes. You set up decision models on the Decision models tab of Admin > AI Provider.

---

Prerequisites

• Role: Admin. Anyone else sees "Admins only" on the page.
• License: A license that includes the AI features, to add, edit or remove models, to save routes or to change a stage. Without it, the tab shows saved settings but locks those buttons. Switching a question off is never locked.
• A decision model to call. Use one on your network, such as Clef 27B on Ollama 0.35 or later, or a hosted decision service. Some runtimes, such as LM Studio, run only the base model and can't act as a decision model.

---

Add a decision model

1. Open Admin > AI Provider and select the Decision models tab.
2. In the Models card, select Add decision model.
3. Fill in the dialog:
   • Name: How this model appears in routes and health. You can't change it later.
   • Runs on: My network (local) for a model on this server or your network. A hosted service for a service you call over the internet.
   • Connection type (hosted only): Standard decision API for TypeSafe and Microsoft. Cloudflare Workers AI, OpenRouter and OpenAI Decisions have their own.
   • Web address: Where the model answers, such as your Ollama address. A hosted service takes its own address, such as https://api.typesafe.ai.
   • Model: The build GraphnAI's limits and thresholds were set for. Local and hosted builds are listed separately.
   • Model name to send: The exact version, for example jev-1.13.0. A name of latest, or one that ends in -latest, :latest or @latest is refused, because it can change without notice.
   • API key: Required for a hosted service. For a local model, only if your server requires one.
   • Key header: How the key is sent. Choose the bearer token option or api-key header. No key is for a local model only.
   • Requests at once: Leave blank to use the default for the model. Local models default to 1, hosted models to 4.
4. Select Test. GraphnAI sends one fixed request with no graph data.
5. Select Add.

More settings holds two fields. Path changes the request path, and only Microsoft Foundry and Cloudflare need it. Their guides give the path. Expected model in answers is for Microsoft Foundry, which answers with the model behind your deployment name.

Read the test result

Check that a model repeats its answers

In the Models table, select Check repeats on a model. The Repeats answers column then reads one of these:

• Gives the same answer every time: GraphnAI reuses the model's answers from a cache.
• Answers vary between calls, so GraphnAI won't reuse its answers: GraphnAI sends every call to the model.
• Not checked yet.

Change or remove a model

Select Edit to change a model, or Remove to delete it. Removing a model also removes it from every route. Leave API key blank in the edit dialog to keep the saved key.

The saved key is sent only to the address you saved it for. If you change the Web address, Connection type, Path or Key header, enter the key again. GraphnAI never follows a redirect from a decision model, so a key can't reach another host. A model marked Re-enter API key has a key the server can't read after a security key change. Select Edit and enter the key again.

---

Choose a model for each task

Role: Admin

A route links a task to a decision model. With no route, the questions for that task aren't asked, and nothing is recorded.

1. In the Routes card, find the task. The tasks are Screening for AI instructions, Graphne, Alert triage, Directory checks after sync, Graph judgments and Choosing a model for each request.
2. Under Decision model, choose a model, or No model.
3. Under Fallbacks, select the models to try next, in the order you select them. The column shows "Add another model to set fallbacks." until you have two models.
4. Under If every model fails, choose Use the deterministic result (the default) or Ask a text model (not calibrated). For the second, choose the text task whose provider answers.
5. Select Save routes.

A text model's answer is never calibrated, so GraphnAI only uses it to order lists.

Pick a model that can screen text

Only some models can screen text. Clef 27B (local), Clef (hosted, Cloudflare), Microsoft Decision 1, TypeSafe Jev and OpenAI decisions can. Clef-flash can't.

In the Screening for AI instructions row, a warning appears before you save when no model can screen:

• Not screening: No model is chosen. Choose Clef 27B or a hosted decision service.
• Can't screen text: None of the chosen models can screen. Choose Clef 27B or a hosted decision service.

Until then, questions that read text an attacker can write aren't answered. A model that can't screen shows "(Can't screen text)" beside its name.

---

Move a question through its stages

Role: Admin

The Questions card lists each question with its purpose, the features that use it (Used by), its stage, a bar of its answers in the last 7 days, and an On switch.

To change a stage:

1. In the Questions card, select Change stage on the question.
2. Choose Move to the next stage up, or any stage down. A question moves up one stage at a time.
3. Add an optional note (up to 500 characters) and confirm. The note is recorded in the audit trail.

A question that can only advise has Move to Automated disabled. Its tooltip reads "This question can only advise. It never acts on its own."

Why a question stays in Shadow mode

A question can't move past Shadow mode until it has a calibration, which maps its probabilities to bands. Until then, the dialog stays open and lists the reasons under "This question can't move up yet:". Shadow mode gives you the records to review in the meantime. Decision Assurance shows how to collect that evidence.

If a question reads attacker-written text, the dialog can also say "Move the screen for AI instructions to Advisory first. This question reads text the screen checks."

Switch a question off and back on

Use the On switch when a question gives odd answers and you want it to stop at once.

1. Turn the switch off on the question's row.
2. Enter a Reason. It's required.
3. Select Switch off.

GraphnAI stops asking the question at once. The row shows "Switched off by name on date", and Change stage is disabled until you switch the question back on. If the setting can't be saved yet, the row says "Switched off now. The setting isn't saved yet, so a restart would undo it. GraphnAI keeps trying to save it." The switch works without a license.

To turn every decision question off, use Turn off AI on the Safety tab. Stop all automation, above the tabs, stops decision questions that automatic work asks. Questions you ask from Graphne still run. The Decision models card says which of the two is on. See AI Safety Controls.

Try a question

1. In the Questions card, select Try it on the question.
2. Enter text in each field. The limit is shown beside the field name.
3. Select Ask.

GraphnAI asks the question once. Tries are recorded but never used to measure the question. The result reads like this:

---

Review recorded answers

Select a question's row to open its records. Use the Outcome list to filter by Answered, Not sure, Held for review, Not asked or Failed. The list covers the last 30 days. GraphnAI keeps each record for 730 days by default.

Select a record to see:

• the probability of each answer, and the Band;
• the Model and its Pinned version;
• Question version, which changes when the question's wording does;
• Calibration, or None;
• Input fingerprint, the first 12 characters of a hash of the input. It lets you tell whether two answers read the same input without storing the input in the list;
• Used by, and how long the model took;
• "Held for review because a field looked like instructions to an AI", with the field names, for a flagged text;
• "Replays to the same result" or "Replay differs", which shows whether asking again gives the same answer.

---

Check model health and cost

The Health card has one row per decision model:

Times show a dash with no calls in the last hour. A model that answers as a different version than the one you pinned shows a warning, and GraphnAI treats its answers as not calibrated until you check the model.

After three errors in a row within a minute, a model pauses for 30 seconds and calls move to the next fallback. See Multi-Provider AI Configuration for how pausing works.

The Cost tab has a Decision models section. It shows this month's decision spend against the monthly budget and the hard ceiling, then Task, Model, Calls, Input tokens and Spend for each model. A model with no price shows "No price set, so its spend shows as $0." Local models usually need one.

---

Configuration

Your GraphnAI administrator sets these limits in the server's settings, and they apply after a restart. A value outside its range is limited to the nearest allowed value.

There is no master switch. With no route for a task, its questions aren't asked.

---

Troubleshooting

"Pin a model version" when you save

The model name is latest or ends in -latest, :latest or @latest. Enter the exact version, for example jev-1.13.0.

The test says the address answered, but not as a decision model

The runtime serves only the base model. Use Ollama 0.35 or later, or a hosted decision service.

"Connected, but ... answered instead of ..."

The service answered with a different model than the one you pinned. Correct Model name to send. For Microsoft Foundry, enter the model behind your deployment name in Expected model in answers.

The key stopped working after you changed the address

The saved key goes only to the address it was saved for. Enter the key again in the edit dialog.

The Status card says "No question can be asked yet"

No route exists. Add a model, then choose it for a task in the Routes card.

The screen question is never answered, or other questions abstain

No model that can screen is chosen for Screening for AI instructions. Choose Clef 27B or a hosted decision service. Questions that read attacker-written text wait for the screen.

A question won't move up

It has no calibration yet, so it stays in Shadow mode. Read the reasons in the dialog.

A model shows "Paused after errors"

It had three errors in a row. Check its address and key with Test. It tries again on its own after 30 seconds.

A model shows "No recent calls"

Nothing was asked in the last hour. Select Test to confirm that it responds.

The first call to a local model is slow

A local model loads on its first call and after 10 idle minutes. The cold timeout is 45 seconds by default.

---

Role requirements