Measure an automation pilot before expanding it

Worked example: the pilot takes more staff time

Suppose 20 synthetic enquiries produce 14 correct completions without intervention, 4 staff handoffs and 2 failed or unresolved cases. The manual baseline for those cases is 80 minutes. Reviewing the pilot, handling exceptions and correcting mistakes takes 95 staff minutes, and entered operating charges total PKR 120.

• Completion rate without intervention: 14 ÷ 20 = 70%.
• Staff handoff rate: 4 ÷ 20 = 20%; failed/unresolved rate: 2 ÷ 20 = 10%.
• Staff time difference: 80 − 95 = −15 minutes. This pilot uses 15 more staff minutes.
• Entered operating cost per tested case: PKR 120 ÷ 20 = PKR 6.
• Entered operating cost per correct completion without intervention: PKR 120 ÷ 14 ≈ PKR 8.57.

The calculation does not support a time-saving claim. Investigate whether failures share a cause, whether staff handoffs were intended and whether the case set reflects real work. A small pilot may help identify a missing permission or unclear rule without establishing a reliable future completion rate. No client outcome is represented by these numbers.

Keep the comparison fair

1. Agree the task, approved information and permitted actions before testing. Include difficult cases, unavailable integrations and a route to a person.
2. Count each case once by its final agreed outcome. An intended staff handoff belongs in the handoff count, even when staff later resolve it correctly.
3. Time all staff work: checking replies, handling exceptions, completing handoffs and correcting mistakes. Compare with the manual process for the same cases and scope.
4. Use actual recorded batch charges where available. Keep setup, training, staff pay and unconfirmed provider fees in the wider proposal review; the calculator does not estimate them.
5. Save the test period, permitted evidence references and unresolved questions. Repeat with a fresh representative batch after a change; do not select only the successful demonstration.