Six feedback loops to design into your AI-first engineering pipeline

We were recently implementing a feature that allowed users to delete their accounts. A coding agent had implemented the endpoint, added tests, and produced a pull request. The tests passed, but review showed that active subscriptions remained untouched. Further testing revealed that deleting the owner also broke access to shared objects.

Each discovery gave us information about what the implementation had missed. We wanted to create a proper process to utilise this information: a process to correct the immediate change, then preserve the expectation so it influenced subsequent work.

The work brought us back to six feedback loops across planning, coding, testing, review, release, and incidents. Designing the cycles in which an agent acts, observes the result, and revises its approach now has a name, loop engineering. The loops below extend that idea beyond the coding agent to the whole delivery pipeline, and this is where I feel most engineering leaders should be focusing when designing an AI-first engineering pipeline.

1. Planning: turn unresolved questions into explicit acceptance tests

Our requirement, “Allow users to delete their accounts”, had described an intention but left several product decisions open.

Should deletion happen immediately? What would happen to an active subscription? Could the sole owner of a shared project delete their account?

We agreed that subscriptions had to be cancelled and ownership of shared projects transferred before deletion. Those decisions gave us specific expectations to test:

• An account with an active subscription could not be deleted.
• A sole project owner had to transfer ownership before deletion.
• An eligible user could complete deletion.
• A user could not delete someone else’s account.

AI helped us inspect the existing implementation and surface unanswered questions and edge cases. Resolving the intended behaviour still required product context and engineering judgement.

We closed the planning loop by baking those answers into the specification and then converting them to acceptance tests. Leaving them in a chat would have made them easy for the next implementation to miss.

2. Coding: use execution results to guide the next edit

During implementation, the agent needed a reliable way to run the application and inspect failures. We made build output, relevant tests, and runtime context available so it could examine where its changes broke.

In one correction, the deletion endpoint had called a service with the wrong argument type. We gave the agent the actual error. It inspected the service contract, corrected the call, and reran the failing check.

The rerun mattered because the edit alone did not establish that the problem had been resolved.

Anthropic’s work on long-running coding agents describes failures where agents declared features complete without adequate testing. In its experiments, providing an executable environment and prompting end-to-end verification helped expose problems that code inspection had missed.

We also defined when to stop iterating. If repeated attempts produced the same failure, the agent needed to return the evidence and its attempted fixes for review. Any additional edit needs a proper reason vetted by an engineer.

3. Testing: make discovered failures reproducible

Our initial tests had covered the scenarios we had thought to encode. Further testing exposed another gap: a subscription lookup failure due to a malformed lookup id was being treated as “no active subscription”. The application consequently allowed deletion when it could not establish eligibility.

We agreed that deletion should be blocked in that situation, then added a regression test to reproduce the failure. We checked that it failed against the broken implementation and passed after the correction.

We also reviewed the assertions. Just asserting that the endpoint returned an error would have been incomplete if it had already deleted the account. We needed to inspect the resulting state.

AI helped construct the reproduction and implement the fix. We need to judge whether the test captures the failure accurately and whether its expected outcome matches the business rule.

4. Review: preserve corrections that should apply again

Review surfaced a permission issue that our existing checks had missed. An ownership check trusted a user ID supplied in the request.

Fixing the endpoint addressed the immediate issue. We also considered where the same mistake could recur and how to make the correction available to subsequent work.

We added a test for cross-account access, used an established authorisation helper, and documented when to use it in the repository guidance.

These serve different purposes. The guidance helps the agent choose an approach while the test checks a specific outcome.

We did not turn every review comment into a permanent rule. We focused on corrections that expressed recurring constraints and made them available before the agent’s next implementation. A critical comment buried in a merged pull request becomes a weak dependency for future work until it is formalised.

5. Release: let observed behaviour control rollout

Before release, we defined the evidence we needed to decide whether rollout should continue.

Google’s guidance on canary releases explains how exposing a change to limited traffic and comparing its behaviour with a control set can inform that decision.

For account deletion, request success alone was insufficient. We also needed visibility into background cleanup and unexpected failures in eligibility checks.

We defined pause conditions, assigned different engineers responsibility across them, and mandated recovery before further deployment. Rolling back application code would not restore data already deleted.

AI constantly helped us examine logs and metrics, but the rollout still needed explicit decision rules vetted by engineers. Scaling the canary becomes a direct function of the observed behaviour, where the function rules are defined by us.

6. Incidents: carry production discoveries back into engineering

A later incident exposed a sequence we had missed: an account had become eligible for deletion, a subscription had been created concurrently, and deletion had proceeded using the earlier eligibility result.

We reconstructed the sequence and examined the contributing conditions. AI helped organise the evidence and propose explanations, which we checked against the observed behaviour.

The follow-up included a concurrency test and a change to how eligibility and deletion were coordinated. Improving detection helps but it addresses another part of the problem because an alert alone does not have prevent the sequence.

Google’s postmortem guidance emphasises understanding contributing causes and implementing preventive actions. We assigned owners to the follow-up work and defined the evidence needed to verify that it addressed the failure.

That evidence collected from these production discoveries should be properly routed back to the planning, implementation, and testing phases. It changes the expectation from each phase of the next release. AI helped us brainstorm which phase each evidence can roughly be attributed to, but the final decision of ownership lies with the engineering leader running the whole cycle.

Start with one recurring failure

Across these six stages, we needed to make four things explicit: the feedback signal, who received it, what action it could trigger, and how we would verify the correction.

An agent could act on feedback within a run. Subsequent runs needed the relevant tests, instructions, and evidence made available again. We had to design that continuity into the workflow.

The most low hanging fruit to implement from this entire process is to start with a correction reviewers keep making or a failure that has escaped more than once. Trace where it becomes visible today, then build a check that brings it forward. Use the next relevant change to see whether the loop actually works.