Decision Assurance: Calibration, Evidence Checks and Promotion for Decision Questions

Overview

A decision question can't move past Shadow mode until evidence from your deployment supports it. Assurance collects that evidence. It turns analysts' decisions into labelled answers, fits a calibration, certifies the cut points for each band, attacks the question with planted instructions, and checks the people whose agreement it counts.

Calibration is never borrowed. A calibration fits one question, answered by one model, on your data. Nothing is shared between deployments, and nothing can be imported or exported.

A question that earns its evidence can move up in the stated time. A question that loses it moves back on its own: when its model, wording or calibration changes, when its answers drift from your analysts' decisions, or when you change the name-replacement setting for its model.

You work in two places on the Decision models tab of Admin > AI Provider:

• The Assurance panel, which opens from each row of the Questions card.
• The Names sent to outside models card, below Routes.

Only Admins see any of it. Other roles see "Admins only".

---

Prerequisites

• Role: Admin.
• A route for the question's task. Without one, the panel says "No decision model is chosen for this question's task, so there's nothing to measure yet." See Decision Models.
• AI turned on, to run attack and rewording checks and model comparisons. These ask the model. Calibration checks make no model calls.
• Analyst activity. Labels come from alert verdicts, Graphne feedback, directory suggestions and the random checks you answer. A question with no matching activity shows "No analyst decisions match this question's answers yet."
• A license that includes the AI features, to run checks, request a stage or use a calibration.

---

Open the Assurance panel

Role: Admin

1. Open Admin > AI Provider and select the Decision models tab.
2. In the Questions card, select Assurance on a question.

The panel opens beside the list. The page address keeps the question, so you can bookmark the panel and it survives a reload. Close the panel to return to the list.

The header shows the stage, "Answered by model name (served model)", and in shadow mode "In shadow mode for n days." A question that only orders lists reads "This question only orders lists, so it's judged by how it orders them."

The sections, top to bottom, are a drift alert when there is one, What it needs next, Labelled answers, Accuracy and calibration, Bands, Attack and rewording checks, Model comparison, Reviewers, Check answers and Calibration history. Attack and rewording checks appears only for questions that read text an attacker can write. Reviewers appears at Advisory.

A question that's switched off is read-only: "Switch the question back on to run checks or change its stage."

---

Collect labels and freeze an evaluation set

Role: Admin

A label is an analyst's decision matched to the answer GraphnAI recorded. GraphnAI collects labels on its own every 6 hours by default.

1. In Labelled answers, read the Confirmed and Ruled out counts, and the same counts by source. "Last collected date." shows when it last ran.
2. To collect now, select Collect labels now.
3. When you have enough confirmed examples, select Freeze evaluation set.

Freezing keeps this exact set, so later checks compare like with like. If you have too few confirmed examples, GraphnAI says "There are n confirmed examples. A set needs at least m."

Once a set is frozen, the section reads "Evaluation set version, frozen date: p confirmed, n ruled out. Half fits the calibration and half checks it." A screen for AI instructions uses a curated set that isn't split: "Every test text checks it."

Which labels count

Only labels from people count. Labels from agents, bulk changes or automatic outcomes never do.

• A set draws only labels where the analyst couldn't see the model's band, plus the random checks. When some were left out, the panel says "n labels were left out because the analyst could see the model's band. Random checks fill the gap."
• "Only random checks decide whether a check passes. Analyst decisions help fit the calibration when there are too few random checks."
• Under each band, the panel shows how many random checks were answered in the last 30 days. "e expired unanswered and k were answered after the alert was opened. Each counts as a wrong answer, so the checks stay safe."

Two warnings protect the evidence:

• "Analyst workflow labels disagree with the random check. Analyst decisions won't be used to fit the calibration until they agree again."
• "One person gave more than half of the random checks in the band band. Ask another admin to answer some."

---

Run checks and read accuracy and calibration

Role: Admin

1. Freeze an evaluation set.
2. In Accuracy and calibration, select Run checks. The panel shows "Running checks: n% done." with Cancel.
3. When the run ends, read the result.

If the checks fail, the panel says "The checks didn't pass:" and lists the reasons. If they can't finish, it says "The checks couldn't finish. The server log has the details." If assurance storage isn't ready, it says "The checks couldn't start because assurance storage isn't ready. Try again in a few minutes."

To recheck the calibration in use on newer examples without changing it, select Recheck with current calibration. A recheck describes the calibration. It never certifies anything new.

Read the result

The chart Stated chance against what happened plots the same thing. Points on the diagonal mean the model is well calibrated. The subgroups table checks each directory on its own, so a model that works in one forest isn't assumed to work in another.

If a note says "Measured on the same examples that set the threshold, so these may read a little high.", treat the numbers as slightly generous.

Every attempt counts

Each run that tries to certify a claim is a numbered attempt. The panel says "Attempt k of this model's checks. Each new attempt has to meet a stricter bar, so rerunning can't buy a pass." A run that fails or errors still uses its attempt.

Curated sets and "on the test set"

A curated set is written by people at GraphnAI, not drawn from your answers. Its mix of real and harmless examples isn't yours. So on a curated set, the section reads On the test set, and every value ends "on the test set". It's never presented as confirmed in your environment.

Questions that screen text

The screen for AI instructions is a detector. It shows 2 meters in place of the high-band meter and the calibration gap:

• Harmless test texts flagged, at most: "n of every 1,000; limit l".
• Known attack examples missed, at most: "x%; limit l%".

Once it's live, "Flags about n of every 1,000 texts here." tells you its real flag rate.

---

Read the bands

Role: Admin

Bands shows the band phrases, the cut points and, when certified, "Automatic from t%".

Two lines explain what each band has meant in your environment:

• "High: n of 10 were confirmed in your environment, from random checks."
• "Low: n of 10 were ruled out in your environment, from random checks."

Until a band has 30 random checks, a curated question reads "High: checked on a test set. Not yet confirmed here." A detector shows no band lines.

With no calibration in use: "Not calibrated. Analysts see no band from this question until a calibration is in use and the question reaches Advisory."

Analysts never see a number. They see a band and a plain sentence. GraphnAI never pre-selects a revoke, and never offers bulk acceptance.

---

Run attack and rewording checks

Role: Admin

Questions that read attacker-writable text, such as a directory description, are tested before they can advise anyone.

1. In Attack and rewording checks, select Run attack and rewording checks. With AI off, the button is disabled: "Turn AI on to run checks that ask the model."
2. Read the table. Each row is a family of planted text, with the number of examples, Changed the answer, Without the screen and how many were held for review.
3. Read the meter Planted text it withstood. A mark shows the minimum automation needs.

The families are explicit instructions, natural-sounding redirection, homoglyphs (look-alike characters), zero-width characters, emoji smuggling and padding past the length limit.

The flip line reads "Rewording or reordering changed h% of clear answers and moved s% into or out of not sure."

If too few examples ran, the panel says "Too few attack examples ran to measure this. Each kind of attack needs at least n different examples." If long text got an answer instead of being refused, it says "Some text past the length limit got an answer instead of being refused. The model may be cutting text silently."

One check may make at most 20,000 model calls by default. A check that needs more stops with an error.

---

Compare models

Role: Admin

1. In Model comparison, tick two to four models.
2. Select Compare.

The table shows each model's wrong-when-high bound, calibration gap and ranking quality, and how closely it agrees with each other model. Each model needs its own calibration and checks. A smaller or compressed build counts as a different model.

With nothing picked, the section reads "Pick two or more models to compare them on the same examples."

---

See what a question needs next and request a stage

Role: Admin

What it needs next is titled "To move to next stage". Each row shows Passed or Not yet, the check, a meter with its limit, and, when it fails, a sentence that says why.

Some checks have server-worded reasons:

• "Too few answers from your environment have been checked in the random sample. It has n. It needs m."
• "Too few of its answers have been checked in the random sample to compare its flag rate here with its test set."
• "Its flag rate here is higher than its test set allows. Check recent flags."

At the top stage a question can reach, the section reads "This question is at the highest stage it can reach."

To request a stage:

1. Wait until every check passes. Until then, Request stage is disabled: "Every check above has to pass first."
2. Select Request stage. The dialog titled "Move to stage?" lists the evidence behind the move.
3. Add an optional Note and select Move to stage.

For Automated, the dialog also says "Automatic actions start from t%, wrong at most a% of the time." and "Tier 0 and Tier 1 accounts are never acted on automatically." Passing these checks never lets a question act on Tier 0 or Tier 1 accounts.

If a stage is refused, the dialog says "This question can't move up yet:" and lists the reasons.

---

Use a calibration

Role: Admin

Calibration history lists each calibration with its method, set version, date and a status: In use, Ready, Retired, Didn't pass or Automatic threshold.

1. Find a calibration marked Ready.
2. Select Use this calibration.
3. Read the dialog, add an optional Note, and select Use calibration.

At Shadow mode, the dialog says "Answers will be recorded with this calibration's bands." At Advisory or Automated, it says "This question goes back to shadow mode until it passes its checks with the new calibration." A promoted question goes back to shadow when you use a new calibration.

A calibration fitted for one model can't be used once the question's model changes. GraphnAI says "This calibration was fitted for model, but the question now uses model."

---

Answer the Check answers list

Role: Admin

Check answers keeps the evidence honest. "Answer a few items so the checks don't depend only on what analysts chose to look at. Some items are tests." New items arrive daily. An empty list reads "Nothing to check right now. New items arrive daily."

Random checks

A random check hides the model's answer.

1. Select Open on a row marked Random check. It shows "Answer by date".
2. Read what the question read. You don't see the band or a link to the item.
3. Select Confirmed, Ruled out or Can't tell.

Answer before you open the item anywhere else. If you open it first, your answer doesn't count, and the check stays fair. After you answer, "Thanks. You can open the item now." shows the link. If your answer wasn't counted, you see "This answer wasn't counted, because the item was opened before you answered."

A random check that isn't answered expires after 14 days and counts as a wrong answer.

Advisory rows

Rows for a question at Advisory show a link to the item, the question title and the band. Select Confirmed, Ruled out or Can't tell.

Test items

Some items are tests with a known answer. After one, the panel says "This was a test item. Its band was deliberately wrong." and "You caught it." or "The band was wrong here." For a random test item: "This was a test item with a known answer." and "You got it right." or "The known answer was different."

Test items are never live alerts and never count as labels.

Reviewers

At Advisory, Reviewers shows "n reviewers have answered enough test items." and "m haven't yet, so their decisions don't count toward automation." Reviewers who agree with whatever band they're shown are caught by test items: "k reviewers missed too many test items. Their agreement isn't counted." The panel shows no names.

---

Drift and moving a question back

Role: Admin

GraphnAI checks every question daily, at 03:00 UTC by default. A question moves back a stage when evidence it depended on lapses. The panel shows a drift alert and, after a move, "Moved back to stage on date: reason."

What moves a question back:

Other alerts are warnings only:

• "Recent answers look different from the ones it was calibrated on. Nothing changed automatically."
• "Its answers on the test set changed since it was checked." The recheck is skipped while AI is off.
• "Almost every recent answer was low. Check that the question still finds real cases."
• "This question has no way to check for drift yet."

---

Names sent to outside models

Role: Admin

Before GraphnAI sends text to a decision model outside your network, it can replace account names, SIDs and host names with stable stand-ins. The real names stay on this server.

The Names sent to outside models card lists each model, whether it runs outside your network, and a Replace names switch. Models outside your network start with it on. Models on your network start with it off.

What's replaced

• Replaced: account names, email addresses, SIDs and host names that include a known domain.
• Not replaced: display names inside free text, bare user names, and host names without a known domain.

Because free text can still hold a display name, don't treat this setting as a promise that no name leaves.

Turn replacement off for a model

1. Turn Replace names off on the model's row.
2. Read "Send real names to model? Questions answered by this model will see account names, SIDs and host names as they are."
3. Enter a Reason. It's required.
4. Confirm.

The row then reads "Real names sent since date, by name". The reason shows as a tooltip.

Any change to this setting notes that "Questions using this model go back to shadow mode, because their checks were run with the other setting." You then run checks again.

Look up a stand-in

Only Admins can look up which name a stand-in stands for, and GraphnAI audits each lookup. A stand-in that can't be turned back reads "This name can't be shown right now."

If the card can't load, it says "Couldn't load the name settings. Retry." It never shows an empty table in its place.

---

Run the checks offline for local models

decision-eval runs a question's evaluation suites from the command line against a model on your network. Use it to compare backends and quantised builds before you configure them.

Run make decision-eval with the question, the endpoint, the model class and the model name. The report is a JSON file under tmp.

An endpoint outside your network needs the -paid flag when you call the tool directly. API keys are read only from the environment variable you name with -key-env. The tool uses an in-memory copy of the question state, so it never changes your server.

---

Configuration

Your GraphnAI administrator sets these limits in the server's settings, and they apply after a restart. A value outside its range is limited to the nearest allowed value. None of them lets a question skip a check.

Labels are kept 730 days by default, the same as recorded answers. When a person is deleted, their label and reviewer identifiers are replaced with a single erased marker.

---

Troubleshooting

Freeze evaluation set says there are too few confirmed examples

The question has too few labelled positives. Wait for more analyst decisions, answer random checks, or use a curated set where one ships for the question.

Request is disabled

A check hasn't passed. Read the failing row in What it needs next. Each failing row says why.

The checks couldn't start because assurance storage isn't ready

The server is still starting or its database was busy. Try again in a few minutes. Nothing was changed.

A run stops with "This check needed more model calls than allowed."

Ask your GraphnAI administrator to raise the model calls per check limit, or run a smaller check.

Run attack and rewording checks is disabled

AI is off, a kill switch is on, or the spending ceiling is reached. See AI Safety Controls.

Random checks keep getting "wasn't counted"

You opened the item elsewhere before you answered. Answer from the random check view first.

A promoted question went back to shadow mode

Read the alert at the top of the panel. A new model, a new calibration or a change to Replace names all do this. Run checks and request the stage again.

"This calibration was fitted for ..., but the question now uses ..."

The route changed models after the calibration was fitted. Run checks for the current model.

---

Role requirements