Define success beyond a satisfaction score
Consider a fictional wholesaler using AI to prepare quotation drafts. An employee checks products, quantities, prices and delivery commitments before completing the offer. A readable draft is useful, but the business task succeeds only when the employee can finish an accurate quotation. Use the completed enquiry as your unit of measurement, including corrections and additional investigation.
Before testing, specify which fields must be correct, which information needs support from the source and when the AI should ask for clarification. Identify errors that would prevent expansion, such as inventing an agreed price. OpenAI recommends task-specific evaluations with defined objectives and measurement rules. Your organisation decides what acceptable performance means for its process.
Separate improvement examples from independent checks
Create two collections: examples used to improve instructions and a test set kept separate during that work. The second collection is often called a held-out set. If you repeatedly adjust instructions around the same difficult quotation, you mainly learn how to handle that quotation. The separate set helps show whether improvements also work for other enquiries.
Include common requests, exceptions, unreadable attachments, missing information and conflicting requirements. Keep similar versions of the same enquiry together, so almost identical documents do not slip into both collections. Ask someone who understands the work to record the expected outcome. Correctly asking for missing information can be more valuable than producing a fully completed draft.
Record the source version, instructions, model settings and assessment criteria. Repeat important cases when inconsistent answers could matter. Anthropic distinguishes succeeding once within several attempts from succeeding consistently. A demonstration of the best attempt does not establish how predictable routine use will be.
Assess different kinds of error separately
Awkward wording is usually quick to fix. An incorrect product code could disrupt a delivery. A single average quality score hides that difference. Record separate outcomes and always report how many cases were tested. The following categories are a practical choice for this fictional process, rather than a universal standard.
On a narrow screen, scroll the table sideways. With a keyboard, focus the table and use the arrow keys.
| Area | What to record | How it informs the decision |
|---|---|---|
| Critical accuracy | Wrong price, product or customer | Separate condition for stopping or remediation |
| Completeness | Missing requirement or delivery commitment | Assess recovery time and cause |
| Appropriate clarification | Missing information identified | Do not count a guess as success |
| Actual effort | Review minutes, recovery and usage | Compare with the current process |
Distinguish an error the employee notices immediately from one that could pass through unnoticed. Also record whether reviewers agree. If two experienced employees assess the same answer differently, the working instructions may need clarification first. An automated AI evaluator can help sort results, but it should not decide your business acceptance criteria.
Calculate time savings and financial value separately
Suppose preparation becomes six minutes faster for each of fifty enquiries, before review and recovery. That saves 300 minutes. Two minutes of additional review per enquiry, beyond the checks in the current process, takes 100 minutes. Four additional recovery cases at 25 minutes each take another 100 minutes. The net time saving is therefore 100 minutes.
Only then convert time into money. At an assumed internal hourly value of €60, those 100 minutes represent €100. Subtracting €20 of direct AI usage leaves €80 in this illustration, before fixed licences, implementation and maintenance. These are fictional figures: the €80 represents valued capacity minus additional direct expenditure, not demonstrated profit. Allocate all costs and returns to the same measurement period. Using the time productively can be valuable; demonstrably avoided expenditure or additional revenue is needed to establish a financial result.
Make evaluation a recurring decision
Repeat relevant tests after changes to instructions, models, source material or process steps. Add failures discovered in practice and assign an owner for the assessment rules. NIST recommends evaluation under conditions resembling deployment and monitoring during production. A small test set can miss rare failures, even when every included case passes.
Do: compare the same business task, record review effort and track critical errors separately. Do not: show only successful examples, equate satisfaction with accuracy or average away a weak category. For the design of review points, see AI workflows with human review.
Current development: the method and the product
As of 29 October 2026. OpenAI has announced the retirement of its Evals platform. Existing evaluations are scheduled to become read-only on 31 October, followed by the dashboard and API shutdown on 30 November. Both milestones are still in the future on this reference date. Keep your test cases and assessment rules portable. See the official retirement schedule.
Frequently asked questions
How large should the test set be?
Large enough to examine the variation that matters. No universal number proves reliability. Start practically and expand when important exceptions or small quality differences need investigation.
When should the pilot move forward?
When it demonstrably meets the agreed conditions and review, cost and recovery remain workable. Limited use for one type of enquiry may be the most sensible next step.
Assess your AI pilot with confidence in the evidence
Boermans Digital helps you build a practical evaluation approach around your own work, quality criteria and error costs.
Discuss your AI pilot