The Least Agency Method for Finance Controls: How to Prove an AI Agent Is Ready to Act
Before giving an AI agent more authority, test the proposed configuration against the business's approved error tolerance and the process it will replace. Keep the required human approvals for high-consequence steps. This post explains how we apply the principle of least agency at Celigo: qualify a configuration in shadow mode, monitor it in operation and reduce its authority when results fall short.
A companion to Least Agency in Finance: When Can an AI Agent Act Without Human Review?, written for the teams who run and audit the controls: controllers, finance operations leads and internal audit.
The model behind the numbers
In this model, L3 needs to catch about 84% of the preparer's errors to meet the tolerance of 6 per 1,000, and about 89.3% to match full human review. The modeled 90% catch rate produces about 4.3 errors per 1,000. Qualification must demonstrate both the approved tolerance and the baseline comparison, allowing for uncertainty in the measurements.
The model's assumptions, all illustrative and all to be replaced with your own measurements:
|
Assumption |
Value |
|---|---|
|
Volume |
1,000 items a month |
|
Manual process |
6 minutes per item, 1.5% wrong (15 per 1,000) |
|
Agent error rate |
3% |
|
Human review |
2 minutes per item; people catch 85% of errors reviewing everything, 95% on a short exception queue |
|
L2 |
Agent flags 10% of items; those flags hold 70% of its errors |
|
L3 |
Reviewer flags 6% of items and catches 90% of errors, so a little under half its flags are real |
|
Tolerance |
6 per 1,000, fitting a moderate-consequence step such as invoice coding |
Hours cover preparation for the manual process and review for L1 to L4, including flagged items. They exclude the work to resolve flags and the cost of running and maintaining the automation.
Proving, monitoring and re-proving a level
Each stage needs different evidence, and a small monthly sample cannot prove a low error rate.
Proving a level. Define the qualification sample and decision rules before testing. Run the proposed configuration in shadow mode while current approvals stay in place. Count errors that would escape the proposed configuration's controls, before any extra safeguards used only during shadow testing. Use the same denominator for the tolerance and baseline comparisons.
With zero errors in n checks, the rule of three gives an approximate one-sided 95% upper bound on the error rate of 3/n.³ About 500 error-free checks correspond to an upper bound of about 6 per 1,000; about 3,000 to 1 per 1,000. This assumes representative, sufficiently independent cases, a stable configuration and reliable error adjudication.
That calculation addresses only the tolerance test. Also show that the configuration does at least as well as the current process, and keep its authority within the approved consequence ceiling. If errors occur, calculate a confidence interval that includes them. Do not use the zero-error rule or keep extending the sample until it happens to pass.
Monitoring a level. Once live, a small sample and planted errors watch for drift. They are an alarm, not proof. The sample tapers as trust grows but never reaches zero.
Re-proving after a change. A new model version, data source or type of input needs a fresh shadow run. A minor prompt edit only has to pass a fixed set of past and planted test cases. A change counts as minor only by its effect: anything that alters decision criteria, approval behavior or permitted actions needs a fresh shadow run.
Who decides what counts as an error. Someone other than the agents. A senior reviewer settles disagreements between the reviewer agent and the person, and checks randomly selected agreements, because both can be wrong. Use a representative, independently adjudicated sample for the error-rate estimate; a human answer alone is not ground truth. Reconciliations, disputes and audit adjustments correct the record later.
Moving down. A breach of any threshold drops the step to the last configuration proven to meet the approved requirements, usually L1, or sends unfamiliar inputs to people while the rest continues. New vendor types or contract structures count as unfamiliar input and start lower, even inside a trusted workflow.
The scorecard
Four measures, for each judgment step, every period:
|
Measure |
What it shows |
Why it matters |
|---|---|---|
|
Errors escaping controls per 1,000 items, with a confidence interval |
How many errors get through |
The test against tolerance |
|
Detection rate on planted and known past errors |
Whether the reviewer works |
Run through a test path that cannot create a real transaction |
|
Escalation precision |
Share of flags that are real problems |
If most flags are noise, the human queue grows back |
|
Escapes by severity, weighted by dollar exposure |
Errors found later |
Four miscoded invoices and one wrong payment are not the same result; one severe error can breach a level on its own |
Agents checking agents
One agent reviewing another can cut human review, but the two may share blind spots. Research shows language models often make the same mistakes, even across providers.¹ The test is whether the reviewer catches the errors the preparer makes.
- Check before it posts. For anything consequential, the reviewer is a gate before execution, not an audit afterward. Monitoring catches drift; it does not undo a payment.
- Independent method. The reviewer is prompted to be skeptical, re-performs the check from source data and never sees the first agent's reasoning. A different model adds diversity on top.
- Independent data where possible. If both agents read the same flawed input, they can agree on the wrong answer. Check against a second source: the bank, the contract, the customer's remittance.
- Limits live outside the agents. Permissions, approvals and dollar limits are enforced by the system the agents run in, and neither agent can waive them. This follows the separation described in OWASP's AI Agent Security Cheat Sheet.
- Measured, not assumed. Planted errors, known past errors and random samples show the reviewer works. Without them, a person approving a clean summary is a rubber stamp.
People stay responsible for checking the reviewer through the exceptions, samples and scorecard. In audit terms, the person moves from inspecting outputs to testing the control. That is management's own control process; independent assurance still comes from internal audit, the third line in the IIA's Three Lines Model.
Worked example: invoice coding at L3
Illustrative L3 invoice-coding flow · 6 stages, 1 exception queue
Rules and limits sit in the checks and the gate, judgment sits with the two agents, and people handle what the system cannot settle. Resolved items go back through the posting gate, so a correction never skips an approval.
What it costs to run
Review hours are only part of the business case. At low volumes, a second agent and its controls may cost more than the review time they save.
- Upfront: integration, data cleanup, control design and the shadow runs that qualify a configuration.
- Recurring: platform and model usage for both agents, human time to resolve exceptions, sampling, control review, and maintaining tests and audit records. Model or workflow changes need retesting.
Measure the full cost per completed transaction during the pilot, including exceptions and rework, and compare it with the current process. Then calculate how long net savings take to recover the setup cost, and be explicit about whether the benefit is cash savings, avoided hiring or capacity for other work.
Notes and sources
All figures on this page and in the article are from an illustrative model, not Celigo company data.
Kim et al., Correlated Errors in Large Language Models, ICML 2025.
The Trust Ladder in the companion article is informed by Kevin Feng, David McDonald and Amy Zhang's Levels of Autonomy for AI Agents, Knight First Amendment Institute, 2025. The level definitions and finance controls are our own application.
Hanley JA and Lippman-Hand A. If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators. JAMA. 1983;249(13):1743-1745. doi:10.1001/jama.1983.03330370053031.
