The Feedback Loop Behind Dependable AI Agents

Imagine asking an AI to prepare your company's weekly cash forecast. It finds the data, produces a report, and explains its calculations. Your finance team checks the result and approves it.

Now imagine asking it to prepare and send the forecast every Friday, without the finance team checking each step first.

Someone will make a decision using the number it sends. Before reducing that supervision, the team needs to know that the agent will follow the agreed definitions, flag changes that could invalidate the result, and stay within its permissions. Approving one report has not yet answered those questions.

What does it take to delegate that responsibility?

Agree on the method

An agent can reach a plausible answer through more than one route. For exploratory work, that flexibility is useful: it can try a different source or investigate an unexpected result. For a recurring financial calculation, a changing method makes comparison harder. If this week's number differs, the team needs to know whether the business changed or the calculation did.

Concourse's Procedures announcement addresses that distinction. Its founder describes taking an agent run that a person has validated and converting its steps into code. Subsequent runs follow the fixed path. The company's platform overview also distinguishes calculations performed in code from model-generated analysis.

The useful promise here is a repeatable method. Next week's inputs can still produce a different answer. What becomes easier to inspect is how those inputs became the result.

That leaves an important decision with the finance team. Code can calculate exactly what it was told to calculate while preserving the wrong business definition. In our hypothetical forecast, a rule that treats every unpaid invoice as money arriving next week might execute perfectly and still mislead the person managing cash.

Agreeing on the rule is part of the handover. So is preserving why the rule was chosen, and knowing when it has changed.

Uvi describes its product around this problem of organizational meaning: resolving internal vocabulary, connecting records across systems, and recovering the decisions behind them. Its website illustrates the problem with a ticket marked done while a rollout remains held back after a rollback. The status of one record does not explain the decision still governing the work. An agent following that record needs the relationship between the two.

For the team receiving Friday's forecast, this means deciding which parts of the work should remain open to judgment. For example, investigating an unusual movement may benefit from an agent choosing a new approach, while applying an accepted accounting rule needs a more controlled path. A good handover makes that boundary visible.

Keep the work reliable

An agreed method still has to survive changes around it.

Papaya's founders describe learning this while building agents: model updates and prompt edits shifted behavior, and finding the cause meant reading execution records by hand. Their launch description proposes a maintenance process built around those records, often called traces. A trace shows the sequence of actions an agent took, including its tool calls.

Papaya says it groups production runs by the job being performed, finds possible improvements, and opens a pull request—a proposed code change—for recommendations the team approves. It then monitors for regressions after a fix ships.

The sequence matters. Observing a problem, proposing a change, accepting it, and checking its effect are separate responsibilities. Automating the proposal does not settle whether the change is desirable.

Consider a possible repair to our forecast agent. It makes several attempts to retrieve a missing account balance, and the report arrives late. A change that removes those attempts could improve delivery time. It could also produce a punctual report with an account missing. The person accepting the repair needs to evaluate the completeness of the forecast alongside its speed.

There are also failures that an unchanged calculation cannot catch by itself. Suppose an input retains its familiar name while its meaning changes, or a newly connected business unit follows a different convention. The code can stay fixed while the information it receives stops meaning what its author expected.

The practical question for anyone deploying an agent is who will notice when its output stops matching the job. For our hypothetical forecast, a handover proposal could specify:

Before any report reaches a reviewer or recipient, automated checks verify that all expected accounts are present, inputs meet the agreed freshness cutoff, currencies and reporting periods match, starting balances agree with source records, and the arithmetic is internally consistent. These checks run on every forecast, outside the model producing it. A failed check holds the report and alerts the finance lead and the engineer responsible for the integration.

During initial validation, the finance lead reviews every report that passes the automated checks. Reducing that supervision requires explicit approval and a documented schedule for continued manual sampling. Automated checks remain in place on every run. A material error suspends automatic delivery. Resuming it requires the cause to be addressed, the relevant checks to pass again, and the finance lead's approval. Changes to models, integrations or business rules require renewed validation.

Retain each forecast as issued and compare it with actual cash movements once the relevant period ends. Give report recipients clear charts of cash trends and forecast differences. Give the team producing the reports a view of missing inputs, failed checks, corrections and delivery delays across runs. Those measurements feed back into monitoring: a growing forecast error or recurring missing account raises an alert for investigation. An unusual movement triggers review rather than an automatic correction, because it could reflect a genuine change in the business. Keep periods and definitions comparable, and reconcile actuals against source records so the charts do not merely repeat the reports' own mistakes.

Record every mistake or anomaly in a central issue system, rather than handling it only through private messages or conversation. Each ticket links to the affected report and records who raised it, when it was created and updated, its owner and status, and the full comment thread with authors and timestamps. Its final resolution records what happened, why a correction was chosen or no change was needed, who approved it, and when it was resolved. Give agents read access through an API or MCP connection, respecting the same access permissions as the underlying reports. Agents should retrieve relevant tickets when preparing reports or proposing changes, distinguish open observations from approved resolutions, and test proposed repairs through the existing review process.

The records should also guide improvements to the controls themselves. A recurring error may call for a new automated test; an alert that repeatedly proves unhelpful may need a different threshold. The resolution should explain the change so future agents and reviewers can apply it in the right circumstances.

The loop is simple enough to remember: check → review → detect patterns → record and reuse what was learned → check again. Each correction becomes something the next run can draw on, with the checks still in place to test whether it helped.

Establish whose permission counts

The forecast begins with decisions the team can make for itself. A task involving another business introduces decisions the team cannot make alone.

A customer can ask an assistant to buy a ticket or return a jacket. The assistant then needs access to an account, permission to perform particular actions, and a service willing to receive it. These conditions can fail independently.

Self's Agent Vault announcement makes the customer's side of the problem concrete. Self says an agent can use placeholder credentials while a protected execution environment or proxy supplies the real secrets at the point of payment or sign-in. Users can set spending caps, restrict merchants, and revoke access.

In Self's design, the ability to complete an authorized transaction is separated from possession of a reusable password or card number. A request such as “buy this for me” can have limits enforced outside the model's instructions.

The receiving business has its own decisions to make. In its draft of Personal Agent Protocol, known as Poppy, Sierra proposes a way for companies to publish how agents can interact with them. An agent identifies itself, establishes a session, and uses access approved by the customer through OAuth, the authorization system many services already use for connected applications. The company defines the permitted actions.

Sierra's example is a jacket return. The retailer recognizes the agent acting for a signed-in customer and allows it to initiate a return while preventing changes to stored payment details. Poppy remains a draft, so the example describes an intended interaction rather than an arrangement every retailer already accepts.

There is a reason that acceptance matters. In September, Amazon blocked Meta's Muse agent from shopping on its site. Amazon told GeekWire that third-party services buying for customers should identify themselves and respect a provider's decision about participation. Meta had said Muse kept passwords and payment details in secure storage, out of the agent's view.

The disagreement exposes the limit of solving credential handling alone. A customer can approve a purchase, and an agent can have a way to protect the customer's secrets, while the merchant still refuses that agent access.

Poppy offers a technical structure for willing participants to recognize one another and enforce permissions. Whether a business joins that arrangement is a separate decision. Making the agent better at navigating a page does not resolve it.

The consequences reach the person who handed over the task. An assistant that cannot enter the store has to return an unfinished job, ask for help, or use another route. Its ability to explain the interruption becomes part of whether the customer can rely on it.

Return to next Friday's cash forecast. Before trusting its delivery, the finance team needs an agreed calculation, a way to recognize and review changes, and access that the relevant systems will honor. It also needs a named person who can take responsibility when one of those conditions fails.

The next level of delegation is making the oversight itself open to inspection and improvement. The team should track which errors escaped automated checks, which human judgments were later corrected, which alerts proved unhelpful, and which repairs failed to prevent a problem from recurring.

Those findings should change how the system is supervised: add a missing test, revise an alert, clarify a business rule, or restore human approval where automation proved unreliable. Record the reasoning alongside the change so future agents and reviewers can understand it.

The feedback loop now improves both the work and the way the work is judged. That gives the team a stronger basis for deciding which responsibilities it can delegate next—and which still need closer attention.