BY NUMBER7 AIAP automation evaluation methodology
Use the same sample. Define what counts as completed.
Agree on the document population, controls and accounting outcome before the test. Include every correction and human intervention in the result.
Read correctly ≠ completed correctly
Does the output agree with the source?
Did the required accounting outcome complete?
How should AP automation be evaluated?
Use a representative fixed sample, define a successful transaction, count all human assistance, and report source extraction separately from accepted accounting outcomes.
Start with real work
Choose the sample before seeing the result.
- 01
Include routine and difficult work
Cover the actual document mix: multi-page invoices, missing evidence, partial receipts, accounting questions and integration failures.
- 02
Keep the source files unchanged
Record document and transaction counts. Do not replace failed inputs or quietly clean the files during the comparison.
- 03
Record every human intervention
Include preparation, corrections, review, clarification, re-entry and failed-handoff recovery.
- 04
Inspect the final outcome
Define the controls, approval state and returned accounting result required for completion.
A worked example
Three readable invoices can have three different outcomes.
Scroll across to compare the columns
| Example transaction | Source reading | Required check | Accounting outcome | Human work |
|---|---|---|---|---|
| A · Routine bill | Correct | Required checks passed | Accepted; returned bill ID | None in this example |
| B · Receipt missing | Correct | Receiving evidence unavailable | Held; no completed posting | Receiving follow-up |
| C · Corrected field | Correct after correction | Reviewer verified the source | Accepted after review | Correction and recheck |
Count B and C in the original sample. A correct extraction is useful evidence; it does not make a held or human-assisted transaction straight-through.
Report both quality and work
Keep the measures separate.
- Extraction correctness
- Do header and line values agree with the source? Report the fields and documents scored.
- Transaction correctness
- Do the values, client context, purchase evidence and required controls support the transaction?
- Straight-through rate
- Which defined eligible transactions completed without human intervention? State eligibility and report exclusions.
- Human effort
- Count active minutes and touches for preparation, review, questions, corrections and reconciliation.
- Accepted accounting outcomes
- Which transactions reached the agreed system and returned an acceptance result?
What a useful evaluation delivers
A result you can inspect.
Retain the unchanged sample, a transaction result table, exception causes, human-intervention log and representative source evidence.
Separate observed results from economic estimates. State the assumptions behind any forecast for your full operation.
- Original population and success criteria
- Completed, held and failed outcomes
- Corrections, reviewer work and waiting time
- Integration acknowledgments and recovery
- Recommended production scope
Next step
Measure the complete workflow.
Use the same source population and success definition for AIdaptIQ and every alternative.