How to Evaluate an AI Agent Before You Trust It With Real Work

Why Agents Need a Different Evaluation Than Chatbots

A chatbot produces text you read before anything happens. An agent takes actions: it edits files, sends messages, updates records, or clicks through websites. Two things change.

First, errors compound. A small misreading in step 2 of a 12-step task becomes a wrong file in step 9. Second, the costly failures are not wrong answers but wrong actions: the email sent to the wrong thread, the record overwritten, the dependency upgraded that nobody asked for.

So evaluate the process, not just the final output. An agent that reaches the right result by doing three things you did not authorize has failed the evaluation.

Third, demos mislead more than they do for chatbots. A vendor demo shows the happy path on clean inputs. Your real work has missing fields, contradictory requests, half-finished records, and people who write "same as last time." The evaluation below is built around those messy cases on purpose, because that is where agents spend most of their real runs.

Step 1: Define the Job and Its Blast Radius

Write one paragraph describing the job, its inputs, what done looks like, and what the agent must never do. Then classify every action it might take:

If you cannot list the actions, the agent is not ready for evaluation; the task is too vague. That is a useful finding in itself.

Step 2: Least-Privilege Permissions

Grant the minimum and widen deliberately. Most agent tools now expose this. Claude Code, for example, has allow, ask, and deny permission rules, and a plan mode in which it explores and reads but does not edit your source files. Deny rules take precedence over allow rules, so a broad deny on a dangerous command cannot be accidentally carved open by a narrower allow.

• Use a separate account or API token scoped to the agent, not your personal admin credentials.
• Point it at a copy, branch, sandbox, or test inbox during evaluation.
• Turn off network access or connectors the task does not need.
• Require approval for every tier-three action, even if it slows the agent down.

Connected tools are the main injection risk. An agent reading untrusted web pages or emails can be steered by text inside them. See using AI agents safely with Gmail, Calendar, and Docs for the inbox-specific version.

Step 3: Dry Runs on Cases With Known Answers

A dry run means the agent does the full task but its outputs go somewhere harmless: drafts instead of sends, a branch instead of main, a spreadsheet of proposed changes instead of updated records.

1. Collect 15 to 30 past cases where you know what the right outcome was.
2. Include at least 3 edge cases: missing information, conflicting instructions, and one case where the right action is to do nothing and ask.
3. Run each case in a fresh session with identical setup, and save the full transcript.
4. Score each run with the rubric below before looking at the next.
5. Run the whole set twice. Inconsistency between runs is a finding, not noise.

Read transcripts, not only outputs. The final draft can look fine while the transcript shows the agent opened files it had no reason to open, retried a failing command eight times, or quietly skipped a step and summarized as if it had done it. Those behaviors predict the failures you will see at scale.

Track time and cost per run as well. If a dry run takes 20 minutes and a person does the same task in 5, the agent may still be worth it for volume, but you should know the ratio before you decide.

The Scoring Rubric

Score each dimension 0, 1, or 2 per case:

Set thresholds before you run. A reasonable starting bar: average outcome above 1.7, zero cases with scope discipline of 0, and zero cases where honesty is 0. A single false "done" is disqualifying for unsupervised work, because it means you must check everything anyway.

Worked Example: A Support Triage Agent

Illustrative scenario: a five-person team wants an agent to label incoming support tickets, draft replies for the common ones, and escalate billing disputes. They take 24 resolved tickets from last month, including 4 billing disputes and 2 tickets with missing account details.

In the dry run the agent writes labels and drafts to a spreadsheet instead of the helpdesk. Suppose results look like this: 21 of 24 correctly labeled, drafts usable with light edits on 15, and 1 of the 4 billing disputes drafted as a refund approval instead of escalated.

The labeling is ready for supervised use. The drafting is ready as a suggestion tool. The billing miss is a scope failure on a tier-three action, so billing tickets stay human-only, and the team adds an explicit rule ("any ticket mentioning refund, chargeback, or invoice dispute: escalate, do not draft") and reruns the 4 billing cases before reconsidering.

Failure Logging and the Promotion Ladder

Keep a failure log from the first dry run. Patterns matter more than individual mistakes, and the log is what lets you compare a new model or prompt against the old one.

Date: 2026-11-03
Agent / model / prompt version: triage-v3 / (model name) / prompt r7
Case ID: T-1182
Tier of failed action: 3 (external)
What it did: drafted refund approval
What it should have done: escalate billing dispute
Failure type: scope | wrong-fact | false-done | loop | injection | other
Caught by: human review in dry run
Fix applied: added explicit escalation rule; reran cases B1-B4
Rerun result: 4/4 escalated

Then promote one step at a time:

1. Dry run on historical cases.
2. Shadow mode on live inputs: agent proposes, human acts.
3. Supervised: agent acts on tier-two actions, human reviews daily.
4. Spot-checked: human samples a percentage of runs weekly.

Drop back a level when the model, system prompt, tools, or data source changes, or when a new failure type appears in the log. Do not skip this because a vendor announced a better model; a better model on benchmarks can still behave differently on your cases.

Your saved dry-run set becomes a regression suite. Rerunning the same 24 cases after a change takes an afternoon and tells you more than any amount of live observation, because you already know the right answers.

Where ClaudeKit Fits

Much agent evaluation planning happens in a chat: drafting the job description, writing the rubric, reviewing transcripts. In claude.ai, ClaudeKit helps keep that work organized. Pin the rubric and thresholds so they stay visible in the Workspace panel while you review runs. Fork from a message to test a revised agent instruction against the same case without losing the original reasoning.

Export review conversations as JSON or Markdown to attach to your failure log, and save your rubric prompt in the prompt library so the next agent you evaluate gets the same bar.