Guide
Plan an AI pilot: a practical checklist for SMEs
Plan an AI pilot with a bounded task, baseline and acceptance test. Includes example test cases, decision criteria and a downloadable starting checklist.
AI-assisted editorial article by EVORI. Sources are listed below; worked examples are labelled as assumptions.
A useful AI pilot answers one concrete question: can your team handle a bounded task better with the proposed solution? Agree on the measurement, unacceptable failures and decision owner before starting. After testing, you can explain why you are proceeding, revising or stopping.
This guide is for SME teams testing their first workflow. If you are still choosing between processes, start with the AI use-case assessment framework.
Put the assignment on one page
Complete this sentence: “The solution prepares this task for this team. This person checks the result. We compare this measure with the current workflow.”
For example, sales operations receive a quotation draft for a fixed product group. A member of staff checks price, quantity and timing. The team measures total handling time per enquiry, including rework. The pilot does not send quotations without approval.
Limit document types, systems and the user group. Keep new requests for a later decision. You can evaluate a test more clearly when its scope does not keep changing during implementation.
Measure the current workflow
Observe several real tasks before building. Record handling time, clarification and rework. Separate waiting from active work: a quotation can sit in an inbox for two days while requiring only twenty minutes of work.
Document which tasks you measured. A straightforward standard case and an incomplete enquiry need different handling. Use comparable cases when evaluating the pilot. The time-cost calculator helps organise volumes and time.
Write expected results before testing
Each test case needs an input, expected behaviour and a criterion for judging it. Microsoft's agent-evaluation guidance recommends covering core scenarios with expected responses or behaviour and explicit acceptance criteria. The allocation below is our planning example, not a mandatory rule or measured project result.
| Case group | Example size | Expected behaviour |
|---|---|---|
| Complete standard enquiries | 10 cases | Draft uses correct quantities and approved prices |
| Missing required information | 8 cases | Solution flags the gap and holds the draft |
| Price or product conflicts | 6 cases | Solution identifies the conflict rather than inventing a price |
| Duplicate or repeated inputs | 6 cases | Team can see the relationship; solution prevents uncontrolled duplicate delivery |
The 30 cases help reveal differences. Even a flawless run on this small selection does not establish everyday reliability. Add cases as you discover new risks or variants. Use cases for final testing that you did not already use during configuration.
Define acceptance criteria
| Question | Record this evidence |
|---|---|
| Does the workflow work correctly? | Result for each case, deviation and required correction |
| Does it save work? | Total handling time including review and rework |
| Does oversight work? | Named approver, substitute and blocked delivery without approval |
| Can the team revert? | Tested manual workflow and incident owner |
| Is the next step worthwhile? | Initial costs, running costs and usable released capacity |
Separate critical failures from convenience issues. A wrong price in an external quotation differs from an awkward sort order in a review queue. A critical failure requires investigation and retesting. The requirements depend on the process; the team must know them before testing.
Assign responsibilities and data access
Name a business owner for correct results and a technical owner for access and operations. Agree with IT which data you may use, who may see it and how long you retain test data. Involve the responsible functions when personal data or employee interests are affected.
Reserve time for the people who will use the workflow. Ask them to perform an approval, a clarification request and a failure case themselves. A demonstration by the developers does not replace that trial.
Make a documented decision after the test
Record the decision and unresolved issues:
- Proceed: agreed criteria are met, with owners and a budget for the next scope.
- Revise: a specific cause can be investigated. Define the change and retest.
- Stop: the data, benefit or operational effort does not currently justify continuing.
Give the next stage its own scope and acceptance criterion. The guide to why AI pilots fail to reach production explains the handover problems this can address.
Start with the fill-in checklist
Download the AI pilot checklist and start with task, baseline and acceptance. For a concrete process, use the AI-assisted quotation guide.
If you need help with test cases or integration, explore our AI workflow implementation. An introductory call clarifies a suitable starting point. Scope and implementation price are agreed before commissioning work.
Frequently asked questions
What belongs in an AI pilot plan?
Describe the chosen task, current process, permitted data and systems, accountable people and acceptance tests. Also define which failures stop the pilot and how the team returns to manual handling.
How many test cases does an AI pilot need?
There is no universal number. Cover the important tasks, exceptions and failure modes. Our 30-case example illustrates a starting point; it does not establish everyday reliability or a statistically supported error rate.
Sources
Further reading
Apply this approach to your process?
Whether this is your first AI solution or an existing pilot, we clarify the open questions and the right next step. Any additional analysis is commissioned separately only if needed.