Part 8 — Traps, caveats, and the verification gap

This is the section to read before you go broad with a rollout, and to return to when something goes wrong. It’s a consolidated list of the failure modes from the underlying research, the cases where the blueprint in this article doesn’t apply, and guidance on how to weight the sources behind all of this.

8.1 — Before I go broad with this rollout, I want a straight list of how it bites people. What are some of the concrete failure modes in the order teams usually hit them?

Eleven failure modes, in roughly the order teams hit them. The first nine are about configuration. The last two are about the thing the configuration exists to serve — and they’re the ones that quietly cost the most.

Trap 1

Treating the sandbox as a security boundary

It isn’t. allowUnsandboxedCommands defaults to true, meaning a command that fails inside the sandbox can be retried under normal permissions. There are also documented demonstrations of agents reasoning around path-based denylists via indirect paths (e.g., reaching a binary through /proc/self/root/usr/bin/npx). Layer external isolation for high-risk work.

Trap 2

Wiring only the Compliance API for audit

It doesn’t cover Claude Code prompt/tool-use content. If you only wire the Compliance API, you have an audit blind spot over your developers’ actual coding sessions. You need OpenTelemetry for that — they’re two separate pipelines and you need both.

Trap 3

Forgetting to pin model versions

Unpinned, Bedrock and Vertex can silently swap models on update; Foundry can hard-error. Pin every model ID in managed-settings.json before you roll out.

Trap 4

Bloated instruction files

The ETH Zurich study (arXiv:2602.11988) found context files can reduce task success rates and add 20%+ cost. Keep AGENTS.md/CLAUDE.mdlean; push detail into skills. If you’re surprised when the agent ignores your 300-line instruction file, this is why.

Trap 5

Assuming AGENTS.md is read by Claude Code directly

As of Q2 2026 it isn’t — bridge it with @import or a symlink in CLAUDE.md.

Trap 6

Trusting MCP registry enforcement as a hard control

GitHub’s own guidance says it matches on server name, can be bypassed, and doesn’t apply to the cloud agent. Treat it as a governance signal, not a security boundary. Use a real gateway.

Trap 7

Underestimating token cost

Subagent-heavy workflows can run roughly 7x a single-threaded session. Bedrock’s default Opus rate limit (around 25 RPM) needs raising before team rollout. Stand up cost dashboards early — before the first surprised finance conversation, not after.

Trap 8

Depending on preview features

GitHub Agentic Workflows was in technical preview as of mid-2026; the MCP enterprise allowlist remained in preview at the broader AI Controls GA; several Copilot agent-customization features are in public preview. Don’t make any of them load-bearing without a fallback.

Trap 9

Skipping the supply-chain basics

CVE-2025-59536 demonstrated that a malicious MCP config in a cloned repo could execute commands before the trust dialog appeared. The Cline and Shai-Hulud incidents showed what happens with autonomous agents that have write access and no defense against prompt injection. The controls: pin plugins by SHA, allowlist MCP servers, and default autonomous work to read-only.

Trap 10

Shipping verification theater instead of verification

This is the trap that defines the Part 1 spectrum, and the one most invisible in a demo. A working demo proves an agent can succeed once; only an eval suite proves it succeeds reliably. Two mechanisms have to work together, and most harnesses ship only the first:

• Tests verify the deterministic parts — given this input, the function returns that output. Checked by code.
• Evals verify the non-deterministic parts — did the agent take a sensible trajectory, choose the right tools, and produce output that clears the quality bar. Checked by labelled datasets, rubrics, and LM judges.

Without both, you are vibe coding regardless of how sophisticated your prompts and hooks are. Build output evals (did it produce the right artifact) and trajectory evals (did it get there the right way) into the harness, with explicit rubrics, and gate shared-workflow agents on eval coverage the same way you gate a service on test coverage. An eval without a clear rubric measures nothing.

Trap 11

Forgetting the harness exists to change what the human does

The harness is not there to remove the developer. It’s there to relocate them — from typing implementation to exercising judgment. Teams that install all the machinery but never shift the human role get the worst of both worlds: agents producing 80% of a feature fast, and humans still trying to review it line-by-line as if they’d written it, which is exactly where the METR 19% slowdown lives.

The 80% problem is real: agents rapidly generate ~80% of a feature, but the last 20% — edge cases, error handling, integration points, subtle correctness — needs contextual judgment current models often lack. The developers who win reserve their attention for what AI struggles with (ambiguous requirements, architectural trade-offs, correctness verification) and delegate what it’s good at. If your rollout plan upgrades the tooling but not the role, you’ve built a faster way to produce code nobody fully understands. That’s debt, not velocity.

8.2 — I'm not sure this whole harness blueprint fits my situation. How can I figure out when it's overkill and what to drop.

The harness design in this article is built for a specific context: multi-vendor (Claude Code + Copilot), polyrepo, regulated or otherwise compliance-sensitive, hybrid cloud. If your situation is materially different, some of it is overhead.

• Single-cloud, single-IDE shops. If you’re 100% VS Code on Azure, the dual-vendor harness is overhead. Pick one vendor and reinvest the saved effort into deeper skills and MCP coverage.
• Pure prototyping / disposable code. If nothing you’re building outlives the week, you’re on the left end of the Part 1 spectrum by design. A lean AGENTS.md and good instincts beat a governance plane. Don’t pay CapEx you’ll never amortize (Part 5).
• If gh-aw reaches GA with a multi-tenant MCP gateway and solid cross-repo orchestration, shift more autonomous repo work from Claude hooks to Agentic Workflows. The hooks-vs-workflows boundary will move.
• If Microsoft ships a first-party managed-settings equivalent for Copilot agent hooks, simplify the policy layer accordingly.
• If you go fully air-gapped or sovereign, drop the direct Anthropic API, front Bedrock via PrivateLink or a self-hosted Anthropic-compatible proxy through an LLM gateway, and accept losing some launch-day features.

The blueprint is a starting point, not a specification. The right thing to do is understand why each decision was made and adjust it to your actual constraints.

8.3 — I want to weigh the evidence behind all this advice properly rather than taking it on faith. How do I understand how much to trust each type of source.

The backbone is the five DORA software-delivery metrics, which are backed by multi-year research and exceptional in the volume of evidence supporting them. The 2025 DORA report — retitled State of AI-assisted Software Development — matters most here because it added a fifth metric, rework rate, to catch the instability the original four miss. That metric is the lie-detector for an AI harness.

The KPIs that earn your trust

Which assets move which KPIs

The point of the table below is the coupling: a single asset — the lean context file, the eval harness — moves several KPIs at once. That’s the actual evidence. A real harness lifts throughput and stability together rather than trading one for the other.

Why the headline numbers lie on their own

The single most important signal is rework rate. It’s the clearest leading indicator of whether AI speed gains are real or borrowed against future debugging debt — teams that see PR volume surge without a corresponding decline in rework rate are typically converting today’s review shortcuts into tomorrow’s incident load. This matters because throughput read alone is misleading: AI adoption tends to improve throughput by an estimated 2–18% yet often comes with declining stability and significantly higher change failure rates. Watching velocity by itself would tell you the harness works even as quality degrades.

Two cautions to bake into how you read the numbers

1. 1.Acceptance rate is not quality. An acceptance rate above roughly 45% may indicate uncritical acceptance rather than tool quality. High acceptance paired with rising rework is a red flag, not a win — always read acceptance and rework together, never acceptance alone.
2. 2.Attribution requires code-level visibility. Traditional dashboards can’t distinguish AI-written code from human-written code, so to prove the harness (and not something else) is the cause, you need AI-vs-non-AI outcome analytics — comparing AI-generated code’s cycle times, rework rates, incident rates, and test coverage against human-written code. That requires repository-level diff access, not just PR metadata. Establish a baseline before adoption and track the delta over at least a 30-day window.