What makes a feedback loop with an AI coding agent actually effective — surfacing real problems fast and turning fixes into durable, reusable knowledge — rather than just patching symptoms one at a time?

Status: active

Config: journals/quests/config/effective-ai-feedback-loops.yaml

The Answer So Far #

Last updated: 2026-07-26

Working hypothesis, anchored on this project’s own bug escalation ritual (dev/docs/rituals/BUG_RITUAL.md): effective AI feedback loops require (a) root-cause diagnosis before fixing, (b) promoting recurring point-fixes to durable directives/rules, and (c) an explicit escalation path when a fix reveals a gap in the surrounding process itself. Concretely, BUG_RITUAL implements this as: a Step-0 self-check against skipping the ritual; a relief valve for genuinely trivial fixes that still requires visible, explicit user approval to skip; fix-cluster detection when 3+ small fixes accumulate in one area; a “diagnose first” sequence that escalates through L1 (root cause) → L2 (is there a directive/ritual gap) → L3 (does the L2 fix reveal a meta-infrastructure gap); a rule to fix at the highest level the diagnosis reaches rather than patching just the instance; a mandatory test-first protocol; and guard comments at the fix site explaining the non-obvious constraint.

First-cycle evidence is strongly, and somewhat surprisingly, supportive. Three independent sources — including both of this quest’s named watch authors — arrived at close analogues of both halves of the hypothesis without any reference to this project, which is meaningfully stronger corroboration than if the search had simply surfaced people citing similar rituals by name:

• Boris Cherny (Claude Code’s creator) directly instantiates hypothesis (b): “Anytime we see Claude do something incorrectly, we add it to the CLAUDE.md, so Claude knows not to do it next time” — a point-fix promoted to a durable, shared, version-controlled rule, with a GitHub Actions workflow (@.claude PR-review tagging) that automates the capture step.
• Simon Willison, describing Jesse Vincent’s “Superpowers” plugin, directly instantiates hypothesis (a): the “Root Cause Tracing” skill’s explicit instruction is to “trace backward through the call chain until you find the original trigger, then fix at the source” rather than patching where the symptom surfaces — functionally identical to BUG_RITUAL’s L1 diagnose-first step, plus a defense-in-depth recommendation (multiple protective layers, not one correction point) that maps onto BUG_RITUAL’s “fix at the highest level” principle.
• Kiro (AWS’s coding-agent product) independently arrived at a third implementation of hypothesis (a): before writing any code, it formulates a falsifiable root-cause hypothesis and presents it for human review, then differentially tests a “fix property” and a “preservation property” (behavior must stay unchanged where the bug condition doesn’t hold). This adds a genuinely new idea BUG_RITUAL doesn’t yet make explicit: Kiro’s write-up names the failure mode this guards against as the “sledgehammer problem” — AI agents are measured as nearly twice as likely as humans to add unnecessary guard clauses and defensive error handling when fixing a bug, drifting the fix scope beyond the actual defect. BUG_RITUAL’s test-first protocol catches whether the fix works, but doesn’t explicitly test that the fix didn’t also change unrelated behavior — Kiro’s “preservation property” is a concrete, adoptable refinement worth considering.

One genuinely different (not contradictory) mechanism for hypothesis (c) surfaced. Ankit Jain’s “Code Review Is Dead” argues review effort should move from reactive (inspect the finished change) to proactive (inspect the intent/spec before generation, then auto-escalate specific change categories — auth logic, schema changes, new dependencies — to mandatory human review regardless of agent confidence). This is escalation-by-risk-classification-of-the-change, decided upfront; BUG_RITUAL’s escalation is triggered reactively, by what a specific bug’s diagnosis reveals. The two aren’t competitors — a mature system plausibly wants both a proactive risk gate and a reactive diagnostic escalation path — but it’s worth watching whether practitioner convergence favors one over the other, or a hybrid.

A naming caution surfaced from the broader industry. A same-week (2026-07-25) piece on AI code review explicitly uses “ritual approval” to mean the opposite of what this project intends — perfunctory rubber-stamping born of alert fatigue (“the AI cried wolf too many times… the team rubber-stamps approvals”). This project’s BUG_RITUAL is a structured diagnostic escalation process, not a rubber stamp, but the term “ritual” is being used pejoratively elsewhere in the same problem space — worth being deliberate about if this project’s ritual vocabulary is ever discussed externally.

Contextual framing. Andrew Ng’s three-loop model (agentic coding loop: seconds–minutes, agent-internal test/verify iteration; developer feedback loop: minutes–hours, human steering and spec refinement; external feedback loop: days–weeks, real users/production) doesn’t test the hypothesis directly but supplies useful vocabulary: BUG_RITUAL operates inside Ng’s “developer feedback loop,” and its output (a new directive, a new test, a guard comment) is precisely the artifact that lets a single agentic-coding-loop failure compound into durable protection across all future loops, rather than being re-discovered each time.

Net assessment for cycle 1: no contradiction of the working hypothesis found; three independent convergent implementations of both (a) and (b); one concrete refinement candidate (Kiro’s preservation-property / sledgehammer-problem framing); one complementary-not-competing alternative for (c) (Jain’s proactive risk routing); one terminology caution.

Synthesis History #

No prior synthesis — this is the initial gather cycle.

Evidence #

2026-07-25 — Why AI Code Review Overwhelms Developers and How to Fix It #

Type: contextual Names “ritual approval” as the failure mode where high-volume, high-false-positive AI review comments cause developers to rubber-stamp PRs without genuine scrutiny — the opposite connotation of this project’s BUG_RITUAL. Proposes risk-tiered escalation instead: critical changes (auth, APIs, payments) require two human reviewers plus security scanning; standard changes need one reviewer; low-risk changes (docs, tests) auto-approve. A terminology caution plus a concrete example of proactive, risk-classification-based escalation (contrast with BUG_RITUAL’s reactive, diagnosis-triggered escalation).

2026-06-30 — Andrew Ng’s 3 Loops for 0-to-1 AI Products #

Type: contextual Three nested feedback loops at different timescales: agentic coding loop (minutes — agent writes/tests/iterates autonomously), developer feedback loop (hours — human steers product decisions and refines specs), external feedback loop (days–weeks — real users, alpha tests, production data). Explicitly credits Boris Cherny and Peter Steinberger for popularizing “loop engineering” as the underlying agentic-coding-loop concept. Doesn’t test the hypothesis directly but situates BUG_RITUAL’s mechanism inside the “developer feedback loop” layer — the layer where a single agent failure gets converted into a durable artifact.

2026-03-02 — How to Kill the Code Review (Ankit Jain, Latent.Space) #

Type: contextual Argues manual code review can’t scale against AI-generated volume (91% increase in review time despite 98% more PRs merged at high-AI-adoption teams) and proposes shifting human oversight upstream — “from reviewing Code to reviewing Intent” — with automatic escalation for specific change categories (auth logic, schema changes, new dependencies, infra config) regardless of agent confidence. A genuinely different mechanism for hypothesis (c): proactive, change-classification-triggered escalation, versus BUG_RITUAL’s reactive, diagnosis-triggered escalation. Complementary rather than contradictory.

2026-02-19 — The bug fix paradox: why AI agents keep breaking working code (Kiro / AWS) #

Type: supporting Names the “sledgehammer problem”: AI agents are measured as nearly twice as likely as humans to add unnecessary guard clauses and defensive error handling when fixing a bug, drifting scope beyond the actual defect without explicit constraints. Kiro’s countermeasure independently reconstructs hypothesis (a): before writing code, formulate a falsifiable root-cause hypothesis and present it for human review, then define a “fix property” (bug condition → postcondition must hold) and a “preservation property” (behavior must stay unchanged when the bug condition doesn’t hold), testing both differentially. The preservation-property idea is a concrete refinement BUG_RITUAL’s current test-first protocol doesn’t explicitly require.

2026-01-03 (approx.) — 13 Practical Claude Code Tips from Its Creator, Boris Cherny #

Type: supporting Tip 5 directly instantiates hypothesis (b): “Anytime we see Claude do something incorrectly, we add it to the CLAUDE.md, so Claude knows not to do it next time” — with a GitHub Actions workflow tagging @.claude in PR reviews to automate the capture. Tip 4 (shared CLAUDE.md in source control for team-wide learned anti-patterns) and Tip 13 (“verification is the most important tip” — feedback loops via tests/browser validation give a 2-3x quality improvement) reinforce the same mechanism from the tool creator’s own stated practice, independent of this project’s ritual design.

2025-10-10 — Superpowers: How I’m using coding agents in October 2025 (Simon Willison, describing Jesse Vincent’s Superpowers plugin) #

Type: supporting The “Root Cause Tracing” skill directly instantiates hypothesis (a): “Trace backward through the call chain until you find the original trigger, then fix at the source,” explicitly warning against fixing where a symptom appears (e.g., a bug surfacing deep in a call stack from git initialization in the wrong directory). Recommends defense-in-depth — multiple protective layers rather than a single correction point — which maps onto BUG_RITUAL’s “fix at the highest level found” principle. Predates this quest and this cycle’s window but is foundational, directly on-point evidence from a named watch author establishing a real, adopted mechanism rather than a hypothetical one.

How We’re Looking #

Keywords: "feedback loop" AI coding agent, AI code review escalation OR ritual, "root cause" AI generated bug fix, AI pair programming retrospective OR postmortem, CLAUDE.md directive OR ritual pattern, "class of bug" AI agent prevention, AI agent self-correction OR learning workflow, AI coding agent memory OR institutional knowledge

Watch authors: Boris Cherny, Simon Willison

Preferred sources: docs.anthropic.com, simonwillison.net, martinfowler.com, news.ycombinator.com, github.com/anthropics

Negative filters: beginner content, “getting started” tutorials

Strategy Changelog #