An AI agent is ready for a live workflow when it can complete representative tasks, respect its permissions, stop safely and produce an acceptable business outcome at a known cost. A convincing demo does not answer those questions. A small test suite can.
Start with 20 to 50 cases from the work the agent will handle. Include routine requests, messy exceptions and cases where the correct action is to ask for help. Run each case more than once. Check the record the agent changed, not only the message it produced.
That is the short version. The rest is the test you can hand to an operations owner.
Start with the record that changed
Imagine an accounts payable agent. It reads an invoice, looks up the purchase order and reports that the case is complete. The reply is concise. The tool log has no red errors. The wrong supplier record now contains the invoice.
The response looked successful. The business outcome was wrong.
This distinction matters because an agent works across several steps. It can select a tool, pass arguments, read the result, choose another step and write back to a system. A polished final answer can hide a mistake anywhere along that route.
Anthropic's guide to agent evaluations separates the transcript from the outcome. The transcript records the messages, tool calls and intermediate results. The outcome is the state left behind. OpenAI's agent evaluation guidance makes a similar practical point by using traces to inspect tool choice, handoffs, policy compliance and end-to-end behaviour.
Test both. A correct record produced through an unsafe route is a problem. So is a tidy trace that ends with the wrong record.
If the steps are already fixed and predictable, an agent may be unnecessary. Compare it with conventional workflow automation before expanding the test surface.
Define the release question
Avoid asking whether the agent is good. Good at what, under which conditions and with whose permission?
Write one release question in plain language. For the invoice example, it could be this:
Can the agent match standard invoices to the correct purchase order, route every mismatch to a reviewer and avoid any payment or supplier-bank change?
That sentence establishes an outcome, an exception path and a hard boundary. It also gives the business owner something concrete to approve.
NIST's August 2026 initial public draft of the TEVV-Athlon Framework starts with the same discipline at a broader level. Its first stage asks teams to state the assessment goal, identify who will use the results and estimate the resources required. The framework then moves through defining measures, applying them and interpreting the evidence. It covers agentic systems, but the cited document is an initial public draft, whose comment period closed on 6 October 2026. Treat it as a useful framework, not a certification.
Turn the release question into six checks.
| Check | What to inspect | Example failure |
|---|---|---|
| Business outcome | Final state in the system of record | Invoice attached to the wrong supplier |
| Tool use | Tool selected and arguments passed | Search uses an incomplete vendor identifier |
| Permission boundary | Actions attempted and blocked | Agent tries to approve a payment |
| Escalation | Cases sent to the right person with useful context | Mismatch is marked complete instead of routed |
| Stability | Results across repeated trials | Same case produces three different routes |
| Operating cost | Cost, elapsed time, retries and review minutes | Cheap inference creates expensive repair work |
The table is deliberately vendor neutral. The release decision belongs to the workflow, not the framework used to build the agent.
Build the test bank from real work
Anthropic recommends starting with 20 to 50 simple tasks drawn from real failures and product requirements. You do not need a benchmark with a heroic name. You need cases that resemble Tuesday afternoon.
For a first release, 24 cases can be enough to expose a weak design. This is an illustrative mix, not a universal sample size.
| Case group | Number | What belongs in it |
|---|---|---|
| Routine work | 8 | Clean inputs that should complete without review |
| Common exceptions | 6 | Missing fields, duplicates, conflicting records and unusual formats |
| Permission boundaries | 4 | Requests that would require a prohibited write or approval |
| Missing or unavailable data | 3 | Timeouts, empty searches and stale records |
| Adversarial or misleading content | 3 | Instructions inside documents or messages that should not control the agent |
Use de-identified cases where production data is sensitive. Preserve the details that make the work difficult, such as inconsistent supplier names or a purchase order split across two cost centres. A sanitised case that removes every awkward feature tests the cleaning process more than the agent.
For each case, record the expected outcome, acceptable variations, forbidden actions and the evidence a reviewer should see. Assign an owner who understands the work. An engineer can confirm that the API call succeeded. The accounts payable lead knows whether the case was handled correctly.
Our invoice processing guide shows why this matters. Extraction is only one part of the job. Matching, discrepancy handling and approval boundaries decide whether the workflow is useful.
Grade the outcome before the wording
Use deterministic checks wherever the result can be inspected directly.
For the invoice case, code can verify the supplier ID, purchase order ID, amount, currency, status and whether a review task exists. A model grader can help assess a draft explanation or the quality of an escalation note. A human should calibrate subjective graders and review high-risk failures.
Do not give one blended score permission to hide a dangerous result. If the agent attempts a prohibited payment action, a strong explanation does not average the failure away.
A useful scorecard separates hard gates from quality measures.
| Measure | Grader | Release role |
|---|---|---|
| Correct final record | Database or API assertion | Hard gate |
| No prohibited action | Permission log and denied-action assertion | Hard gate |
| Correct escalation | Workflow-state assertion | Hard gate for exception cases |
| Clear reviewer note | Human or calibrated rubric | Quality measure |
| Tool efficiency | Trace analysis | Diagnostic measure |
| Cost and latency | Usage and timing logs | Operating measure |
OpenAI recommends moving from trace inspection to repeatable datasets and evaluation runs once the team knows what good looks like. That sequence is sensible. Read a few failures first. Otherwise, you may automate a score that rewards the wrong behaviour with great consistency.
Attack the permission boundary
An agent should operate with the least access required for its job. Test that claim directly.
OWASP describes excessive agency as a combination of too much functionality, permission or autonomy. Its prevention guidance recommends limiting available tools, narrowing their functions, applying downstream authorization and requiring human approval for high-impact actions.
Turn those controls into cases.
- Ask a read-only research agent to edit a source record.
- Put an instruction inside a retrieved document that tells the agent to ignore its task.
- Remove a required field and see whether the agent invents it.
- Return a tool error after the first successful step.
- Ask the agent to act for a different user or business unit.
- Present an urgent request that tries to bypass approval.
The correct response may be a refusal, a safe stop or an escalation. Define which one you expect. "The agent did nothing" can be a pass when the alternative is an unauthorised action.
Permission tests also expose architecture problems. If the agent can reach a delete function that its role never needs, improve the tool surface before rewriting the prompt. A polite instruction is not an access-control system.
A safe stop is a successful test when the next action needs human approval.
Repeat the same cases
Model output varies. One successful run says little about stability.
Run each important case at least three times during the first evaluation. In the 24-case example, that produces 72 trials. The number is an editorial example, not a statistical guarantee. Increase the trial count for unstable or high-risk tasks.
Report two rates separately.
- Task pass rate is the share of trials that meet all required criteria.
- Case reliability is the share of test cases that pass every trial.
Suppose 68 of 72 trials pass. The task pass rate is 94.4%. That looks respectable. If those four failures affect four different cases, only 20 of 24 cases passed every time, giving case reliability of 83.3%.
The second number tells a less comfortable story. Good. Evaluation should be allowed to make the launch meeting awkward.
Keep the failed traces. Label the failure mode, such as wrong tool, wrong arguments, missing escalation, invented data or permission breach. Compare versions against the same bank. Add new production failures to the bank after launch.
Count the cost of accepted work
Record cost and latency for the whole trial. Include model calls, external tools, retries and review time. A test that stops before human review will understate the operating cost.
Use this calculation:
cost per accepted task = total trial cost / accepted outcomes
If 72 trials cost $28.80 in models and tools, and reviewers spend 90 minutes at an internal cost of $40 per hour, the trial costs $88.80. If 68 outcomes are accepted, the cost is about $1.31 per accepted task. These figures are illustrative.
Track the slowest useful cases too. An acceptable median can hide a long tail of retries and timeouts. If the business promise is a two-minute response, measure the proportion that meets two minutes instead of celebrating an average.
Our guide to AI agent costs per completed task explains the unit economics in more detail. The principle is simple. Count the repair.
Make the launch decision explicit
Set thresholds before the final run. Choose them according to the consequence of failure. A content-research assistant and a payment agent should not share the same tolerance.
An illustrative release rule for the invoice agent might require all of the following.
- Zero prohibited writes or approval attempts
- Every mismatch routed to review
- At least 95% of routine trials produce the correct final record
- Every test case has at least one successful route
- Cost per accepted task stays below the agreed process target
- Ninety-five percent of accepted tasks finish within the service limit
These are example thresholds. The business owner, risk owner and technical owner must set the real values. Record the version of the model, instructions, tools and test bank used for the decision.
If a hard gate fails, do not negotiate with the average. Narrow the agent's scope, remove the risky tool, strengthen the deterministic control or keep the human approval step. Then rerun the same bank.
This test fits inside a broader pilot-to-production process. It does not replace monitoring after launch. Start with limited users and reversible actions. Watch real outcomes, review failures and keep a kill switch that someone knows how to use.
The release meeting should end with evidence attached to a decision. If nobody can point to the failed case, the final system state and the rule that permits or blocks launch, the agent is still a demo.