Outcome-Based Agent Evaluation vs Transcript Review

The transcript is evidence, but it is not the outcome. A human can write a convincing incident summary while missing the root cause. Agents have the same failure mode at higher speed. They can produce a clean explanation, name the right concepts, and still fail to update the ticket, validate the SQL, or identify the risky infrastructure change.

Situation

A human can write a convincing incident summary while missing the root cause. Agents have the same failure mode at higher speed. They can produce a clean explanation, name the right concepts, and still fail to update the ticket, validate the SQL, or identify the risky infrastructure change.

The pattern matters for database, cloud, and platform teams because agents do not operate in a vacuum. They inherit repository rules, tool permissions, deployment workflows, incident history, and the quality of the evidence available to them.

Operating layer	Default approach	Better alternative
Context	Rely on a long prompt or chat history	Give the agent task-specific evidence and rules
Tooling	Expose broad tools and inspect later	Expose narrow tools with clear approval boundaries
Verification	Read the final answer	Check the artifact, trace, and final state

The Problem

Transcript review rewards the surface area of reasoning. Database and cloud operations need a harder bar: did the final state become safer, more accurate, or more observable?

The practical question is not whether an agent can produce a convincing response. The question is whether the engineering system around that response makes the work observable, reversible, and reviewable.

Failure point	What breaks	Why it matters
Weak boundary	Agent authority is broader than the task	A diagnostic run can become an unsafe change
Missing evidence	The agent cannot cite the state it used	Review becomes opinion instead of verification
No lifecycle	The workflow ends at a message	Ownership, audit, cleanup, and rollback disappear

Outcome-Based Evaluation

For a DB workflow, the outcome might be a passing migration test, a rejected lock-risk DDL statement, a restored backup checksum, or a correctly classified replication delay.

flowchart TD
    A[task request — bounded intent] --> B[outcome-based evaluation — controls]
    B --> C[tool execution — evidence collected]
    C --> D[verification — final state checked]
    D --> E[human handoff — audit retained]

1. Define the operating boundary.
   Write down the task class, allowed tools, environment, data class, and approval mode before the agent runs.

2. Shape the evidence.
   Return compact observations instead of raw dumps. The agent should see enough to reason, but not so much that context is wasted.

3. Require proof of completion.
   Completion should be an artifact or state check: a passing test, a reviewed plan, a valid rollback, a trace, or a linked ticket.

Define outcomes as artifacts: SQL that compiles, a Terraform plan with no unauthorized resources, a PR with rollback attached, or an incident note with cited evidence.

In Practice

Context: Anthropic’s eval guidance separates task execution from grading. The reusable lesson is that the task should be judged by the state that matters, not by whether the model claimed success. Source: Anthropic, Demystifying evals for AI agents.

Action: Define outcomes as artifacts: SQL that compiles, a Terraform plan with no unauthorized resources, a PR with rollback attached, or an incident note with cited evidence.

Result: When the output artifact is machine-checkable, the team can compare agents, prompts, tools, and model versions without debating style.

Learning: For a DB workflow, the outcome might be a passing migration test, a rejected lock-risk DDL statement, a restored backup checksum, or a correctly classified replication delay. This is a documented pattern or a direct consequence of how the named systems behave, not a fabricated production story.

Where It Breaks

Failure mode	Trigger	Fix
Elegant wrong answer	Reasoning reads well but the artifact is invalid	Require executable or inspectable outputs
Missing evidence	Agent states a conclusion without source output	Attach command output, plan diff, or query plan
Unclear success	Task ends with a summary but no final state	Define completion before execution starts
Reviewer fatigue	Humans reread long transcripts	Grade short artifacts and preserve traces for audit

What to Do Next

• Problem: Transcript review rewards the surface area of reasoning. Database and cloud operations need a harder bar: did the final state become safer, more accurate, or more observable?
• Solution: For a DB workflow, the outcome might be a passing migration test, a rejected lock-risk DDL statement, a restored backup checksum, or a correctly classified replication delay.
• Proof: When the output artifact is machine-checkable, the team can compare agents, prompts, tools, and model versions without debating style.
• Action: Replace one transcript review checklist with an outcome checklist: artifact, evidence, final state, and owner approval.

The teams that get value from agents will not be the teams with the longest prompts. They will be the teams that turn agent work into a controlled engineering workflow.