Human-in-the-loop in practice: checkpoints that work

Human-in-the-loop means that a person makes a decision at defined points in a workflow before an action takes effect. The practical question is not whether, but where and how: a poorly placed checkpoint does not catch errors. It just creates more clicking.

The problem with a blanket approval button

When people have to sign off on every single case, the result is predictable: after two weeks, they start waving things through. Approval exists on paper, but no longer checks anything. Effective oversight depends on selection: people see the cases where their attention makes a difference.

Three questions for every checkpoint

1. What are the consequences of an error?

Assign each workflow step to a category based on its consequences:

• Correctable internally (such as a document sorted into the wrong category): spot checks are sufficient.
• Visible externally (such as a customer response): approval before sending, every time.
• Legally or financially binding (such as a posting, purchase order or contractual commitment): approval plus a second-person check or an amount limit.

2. When should a case be escalated instead of prepared?

Define triggers that cause the workflow to send a case straight to a person without creating a draft: low recognition confidence, an unknown counterparty, unusual amounts, emotional or escalating language, or legal issues. This list is operational knowledge. It grows with every edge case and needs to be documented.

3. What does the reviewer see?

An approval is only as good as the information behind it. The review screen needs three things side by side: the result, the sources for each statement, and the reason the case was flagged for review. If people have to search before they can check, they will soon stop checking.

How to tell whether your checkpoints are in the right place

• The correction rate at checkpoints is greater than zero, otherwise you are checking in the wrong place, and well below half, otherwise the workflow is not yet mature enough.
• Escalated cases arrive with context, rather than as a bare forwarded message.
• The team can explain in one sentence which cases never run automatically.
• The list of escalation triggers has been expanded at least once since introduction.

Context: human oversight under the EU AI Act

The EU AI Act explicitly requires effective human oversight for certain systems (Article 14). Whether and how this applies to your use case needs legal review. Whatever the classification, a workflow with documented checkpoints and escalation rules is also the better operational choice.

A checklist to take away

1. Every step has a consequence category.
2. Steps that have an external effect require approval before that effect occurs.
3. Escalation triggers are defined in writing.
4. Reviewers see the result, sources and reason for review side by side.
5. Correction and escalation rates are monitored.

For the wider process of building a workflow, see From an idea to a production workflow. To design checkpoints for a specific process, see AI workflows by EVORI.