Verify Coding-Agent Completion Against the Change

A coding agent is done only when the requested change is present, the required acceptance checks ran against that exact change, and the protected state stayed within its agreed boundary. A message that says “fixed” or “tests passed” is a claim. It is not proof of the repository state or the test run.

TL;DR — proof for code changes

A reviewable completion decision needs three linked records: the acceptance conditions, the final changed revision, and validation that actually ran against that revision. Keep any missing runtime or customer evidence visible as a gap.

This distinction matters because software work has several separate outcomes. The agent may edit the right file but miss the behavior. It may produce the behavior but fail a required check. It may pass tests while also changing an unrelated file. A trustworthy completion record should preserve those differences instead of compressing them into one green status.

The practical question for engineering teams is not whether an agent can produce a plausible patch. It is whether a reviewer can independently determine what changed, what was checked, and what remains uncertain without relying on the agent’s own summary.

Define completion before the agent starts

Write acceptance conditions as observable states. “Fix the authorization bug” does not identify what the reviewer should inspect. A better contract names the protected operation, the permitted behavior, the denied behavior, and the required evidence.

For example:

• an authorized workspace member can update a record they own;
• a member from another workspace cannot read or change that record;
• the regression test runs against the final diff;
• the production configuration and database schema remain unchanged.

Each condition needs an evidence source. A test result may support the allow and deny behaviors. A diff may show which files changed. A repository check can detect configuration or schema changes. The agent’s final response can point to those artifacts, but it cannot replace them.

Separate the claim from the evidence

A completion review should ask four independent questions:

1. What did the task require? Record the acceptance conditions before implementation.
2. What state changed? Inspect the final diff and relevant system state.
3. What validation ran? Record the command, result, and code revision under test.
4. What remains unproven? List skipped checks, inconclusive results, environment gaps, and protected state that was not observed.

This separates a correct result from a persuasive narrative. If the agent says a test passed but the command log is missing, mark the test as unverified. If the test ran on a prior revision, it does not establish the final diff. If the patch is correct but a browser-only behavior was not exercised, report source correctness and runtime behavior separately.

Use an evidence matrix

A small evidence matrix makes review less dependent on memory. It also shows when one artifact is being asked to prove too much.

Provenance helps connect artifacts to the people, activities, and entities involved in producing them. The W3C PROV family provides a model for describing those relationships. For a coding task, a useful record links the task contract, agent execution, changed revision, validator run, and final decision. The link does not make a weak test strong; it lets reviewers see what the test actually supports.

Tie every check to the exact change

A test result is only useful when it applies to the version being accepted. Record the commit or content hash tested. If the agent edits a file after the check, the prior result is stale. If a command silently tests a different directory, environment, or service instance, the result may not apply.

A minimally useful validation record includes:

OpenTelemetry maintains GenAI-specific semantic conventions for spans, metrics, and events. Those records can help reconstruct execution paths; they do not establish that the resulting code met its acceptance conditions. Observability answers “what was recorded?” Verification asks “does this evidence support the required claim?”

Preserve partial, failed, and inconclusive outcomes

A completion system needs more than pass and fail. A test can time out. A staging service can be unavailable. The requested code may be complete while one required integration check remains blocked. If every non-green result is rewritten as “done with caveats,” the reviewer loses the distinction that matters.

Use explicit outcomes such as:

• Verified: every required condition has applicable evidence.
• Partial: some conditions have evidence; named conditions remain open.
• Failed: evidence shows a required condition was not met.
• Inconclusive: the check did not produce a reliable result.
• Not run: no check was attempted.

These are suggested reporting states, not a standard. Their value is that each state leads to a different decision. Verified work can proceed to the next gate. Partial work needs an owner and a next step. Failed work needs repair. Inconclusive or unrun work cannot be presented as a pass.

Check both the intended change and the untouched boundary

Most reviews focus on whether the intended change exists. Agent work also needs a boundary review: did the agent change something it was not asked to change?

That review should examine the entire final diff, including generated files, lockfiles, migrations, configuration, and scripts. It should compare protected state before and after when the task could affect an external service or durable record. A clean summary is not enough. The agent might omit a change, misunderstand its effect, or make an indirect modification.

For higher-impact work, the system can enforce the boundary before execution. A tool can restrict writable paths. A release service can require separate approval. A test runner can attach the tested revision to its result. These controls reduce dependence on manual review, but they still need explicit failure handling.

Example: a coding agent changes access control

Suppose a task asks an agent to prevent one workspace from reading another workspace’s record. A sound completion contract has four parts:

1. The authorized user can read their own record.
2. The unauthorized user receives a denial.
3. A regression test proves both cases against the final revision.
4. The diff contains no unrelated permission, schema, or production configuration change.

A green unit test might prove only the third item. It does not establish that the test covered a real authorization boundary. A full diff may prove there was no schema edit, but not that the deployed service uses the changed code. A staging readback can provide runtime evidence, but still does not prove every production tenant behaves identically.

The reviewer should report each evidence class separately. Source evidence is not a deployed service. A passing test is not a customer outcome. A real outcome may still lack an independent record. This vocabulary prevents one successful check from carrying claims it cannot support.

Measure proof quality, not just agent output

NIST’s AI Risk Management Framework describes measurement as an ongoing activity and calls for documented methods, uncertainty, repeatable testing, and independent review where appropriate. Teams can apply that discipline to agent completion.

Useful operational measures include:

• percentage of completion claims with artifacts tied to the final revision;
• percentage of required acceptance conditions with independent evidence;
• stale validation results caught before approval;
• out-of-scope changes found during review;
• time needed for a reviewer to reproduce the completion decision;
• frequency of partial or inconclusive outcomes that later become false passes.

Do not optimize only for the number of green runs. A system that labels every run complete may look productive while transferring hidden review work to operators. A better measure asks whether the evidence reduced the amount of independent reconstruction the reviewer had to do.

Set the acceptance threshold

For each mandatory condition, set a threshold before the agent starts. A test condition might require a passing result on the final revision. A protected-state condition might require no diff in a named configuration path. A runtime condition might require a readback that shows the expected state. The threshold needs a result another reviewer can reproduce.

Choose a metric that measures proof linkage, such as the share of mandatory conditions backed by evidence tied to the final revision. A high completion rate alone can hide missing evidence. The tradeoff is review cost: collecting more traces can improve reconstruction while also increasing sensitive-data exposure. Limit captured arguments and results to the minimum needed for review.

The implementation should preserve a receipt for each required validator: revision, command, result, environment, and known limitation. If a mandatory condition lacks evidence, label the task partial or inconclusive. Do not invoke rollback automatically for every failed check; define which changes are safe to reverse and which require an operator to restore state.

A compact acceptance record

For each consequential task, keep a record with:

• task and acceptance conditions;
• actor and authority scope;
• changed revision and complete diff;
• required checks, results, and revision linkage;
• runtime observations when the task needs them;
• protected state checked;
• unresolved gaps and their owner;
• reviewer decision and time.

The record can be brief when the work is low risk. Irreversible changes, production writes, customer communications, and financial actions need stronger evidence and explicit approval. Proportional evidence is useful; silent assumptions are not.

Armalo writes about trust and accountability for agent networks. This article describes an engineering standard for evaluating completion claims. It does not claim that any single trace, score, or product surface proves a coding task is correct.

Make independent verification a property of the evidence

A second model reading the first model’s summary is not automatically an independent verifier. The reviewer needs evidence with a different failure path from the agent’s claim. A repository diff comes from version control. A test result comes from the runner. A live-state readback comes from the service that owns the state. Each source can still be wrong, but the reviewer can assess its limits separately.

Choose the validator from the requirement, not from whichever command is easiest to run. A type check can show that code conforms to a set of static constraints. It cannot establish that a user with the wrong account receives a denial. A unit test can prove the assertion encoded in that test. It cannot prove that the test matches the business rule. A browser check can show the behavior in one environment. It cannot establish that a production deployment has the same configuration.

For important work, map each acceptance condition to a validator and an independent reviewer. Where no suitable validator exists, say so. That is a measurement gap to resolve, not a reason to accept a narrative as proof.

Treat evidence collection as a privacy decision

Execution traces can contain prompts, customer records, credentials, or other sensitive values. More logging is not always better. Record the identifiers and outcomes needed to reconstruct the decision, and restrict or redact content that reviewers do not need.

The evidence design should answer four questions before a workflow runs: who may read the record, how long it remains useful, which values must be excluded, and how a reviewer can verify that the artifact belongs to the tested revision. A content hash can identify a specific artifact without publishing it. It does not by itself prove who produced the artifact or when. Access controls and retention rules should fit the sensitivity of the task.

This creates a real tradeoff. Sparse evidence makes a result hard to challenge. Unbounded logs create privacy and security risk. The right target is not maximum capture; it is sufficient, scoped evidence with a clear owner and expiry.

Respond when verification finds a defect

A failed check is a decision point. Preserve the revision and validation record. Identify whether the problem came from the code, the acceptance contract, the test, or the environment. If an external state changed, assign an operator who can decide whether recovery is safe. After a repair, rerun the required checks against the repaired revision and keep both attempts in the evidence record.

Do not overwrite an earlier failure with the later pass. The sequence explains what was wrong, what changed, and which result applies to the accepted version. A clean final status without the earlier failure can erase information that the next reviewer needs.

FAQ

Does a green test prove the coding task is done?

Only if the test checks the stated acceptance condition and ran against the exact revision being accepted. It does not prove unrelated requirements or production behavior.

Can the agent’s tool trace count as independent proof?

The trace can document execution. A separate validator or system readback must establish the outcome the task requires.

What should happen when a required check cannot run?

Mark the result inconclusive or not run, name the blocker, and assign the next action. Do not convert missing evidence into a pass.

Source material

• National Institute of Standards and Technology, AI Risk Management Framework Core.
• World Wide Web Consortium, PROV Overview.
• OpenTelemetry, GenAI semantic conventions.