what your AI coding agent tested, and what it left for you

The handoff moment after an AI coding agent closes a task is always a little opaque. The agent reports what it did: the feature is implemented, tests pass, the code review found no blockers. What it doesn't report is the full shape of what it left unchecked, and what that means for you as the reviewer.

That's not a knock on the agent. Some verification is outside what an automated process can do on its own. The problem is that without a clear boundary, "tests pass" expands to fill available optimism. You walk into the diff review carrying assumptions that aren't warranted.

what an agent checks on its own

Agents are reliable at verification that produces a deterministic signal. These are the things an agent can own completely:

• Compilation and build - does the code compile? Does the bundle build without errors?
• Type checking - do type annotations hold through the compiler?
• Linting - does the code conform to the project's style rules?
• Existing test suite - do the tests that were passing before still pass?
• New tests for the new code - does the agent write tests for the feature it implemented, and do they pass?

Some agents also do informal behavioral validation - running the changed code path once to confirm it doesn't throw under the happy path. That's useful, but it's not systematic coverage.

what it can't verify

The harder class is correctness under real conditions. A test suite asserts that a function returns the right value for given inputs. It can't assert that the output is what users will find useful, or that the feature integrates cleanly with other features that share the same data.

Specific categories that fall outside agent verification:

Intent fidelity. Does the feature do what the original story asked, vs a literal reading of the spec? Specs are written before the work, with incomplete context. The agent implements the spec; it doesn't audit the spec against reality.

Visual and UI behavior. Automated visual regression tests exist, but most agent workflows don't run them. Layout breaks on mobile, focus management issues, scroll position resets - these don't produce a failing test unless someone explicitly wrote one for them.

Cross-feature integration. The agent changed file A and file B. Did that affect feature C, which also reads from file B but wasn't in scope? Integration coverage is always partial. The agent can only check what it knows is in the task.

Security surface changes. A new API endpoint, a change to authentication logic, a new field in the response payload - these have security implications that don't surface in a unit test. They need a manual read.

Production data edge cases. Test fixtures are clean and controlled. Production data has nulls in unexpected fields, strings with unusual encoding, IDs that no longer correspond to any live entity. The agent tested against fixtures, not a real data snapshot.

Performance under load. The feature works correctly on the first call. Under concurrent load, or with a large dataset, the behavior may be different. Agents don't run load tests by default.

the invisible checklist problem

The practical issue is that none of this is typically written down. After an agent closes a task, you have a diff, a summary, and "tests pass." The remaining verification work is implicitly "everything else" - an open set you reconstruct from the diff and your own memory of what the feature is supposed to do.

This is where overhead accumulates for teams running agents in parallel. Each closed task creates verification debt without a label. You work out what to check from the diff, from scratch, each time. Across ten tasks in a sprint, that reconstruction adds up - and because it's informal, it's easy to skip a category entirely and not notice until a bug surfaces later.

The fix is to make the boundary explicit at the point of handoff. Not just "here's what I did" but "here's what I checked, and here's what I flagged for human review." Not a vague "please review carefully" - a specific list: the behaviors the agent confirmed, the edge cases it tested, and the surface areas that need human eyes because the agent can't get there on its own.

a structured handoff

defract v1.4.0 adds a manual test list to the review stage that separates these two categories explicitly. After the agent's implementation and automated review complete, the task handoff includes two parts: what the agent verified (test runs, lint, build, specific behaviors it confirmed), and what requires human review (UI behaviors, cross-feature interactions, security surface, real data scenarios the agent flagged but couldn't test itself).

The list comes from the agent's own assessment, not from you reconstructing it from the diff. It's part of how the review stage hands off to the human gate - you look at a defined scope, not an open-ended diff.

For parallel agent workflows, this matters more as the task count grows. You can't hold the verification scope for ten concurrent tasks in memory at once. A per-task checklist at handoff is what makes systematic review possible without slowing the whole pipeline down.

The yes-man problem has the same underlying shape: an agent that doesn't push back gives you no signal about where to look harder. A handoff that says "tests pass" without naming what wasn't tested has the same gap. The goal isn't distrust - it's knowing where your attention is actually needed.