Make feedback something a colleague can use
An AI coaching agent can surface a missed clarification, an unsupported promise or an opportunity to explain a policy more clearly. Its usefulness depends on whether a colleague can understand the feedback and act on it. A polished score for empathy means little if nobody can point to the behaviour it describes or agree that the judgment is fair.
For customer support, begin with an observable skill: confirm the issue, ask an essential question, explain the next step or escalate when necessary. Attach feedback to a specific moment and the relevant policy. Offer a short alternative response and explain its purpose. Keep suggestions limited enough to fit the employee’s workflow. Good coaching should reduce ambiguity about the next attempt.
Evaluate the conversation and the outcome
A conversational agent may sound helpful while applying the wrong rule or failing to complete an action. The τ-bench research evaluates tool-using conversations against policy and checks the resulting database state. That distinction is valuable for coaching too: assess what was said, what evidence was used and what actually happened. Repeat important scenarios to expose inconsistent behaviour.
Anthropic’s January 2026 evaluation guidance separates code-based, model-based and human grading. Use deterministic checks where a requirement is explicit, such as including a required disclosure. Use expert rubrics for context-dependent judgments such as whether a clarification was appropriate. A model judge can help scale review, but it needs comparison with human ratings and an examination of disagreements.
Build the rubric with the people doing the work
Have supervisors and experienced staff label a representative sample independently. Discuss disagreements before automating the rubric. Include difficult conversations, transcription errors, different languages and situations where escalation is the right outcome. Evaluate whether the feedback is supported by the transcript, whether the suggested action is permitted and whether it targets a skill the employee can influence.
My work on SENTiVENT Bench uses expert annotations and reproducible scoring to evaluate structured extraction and implicit sentiment. That discipline transfers to coaching design: define the target clearly, preserve the evidence and separate scoring rules from the system being tested. It also discourages treating an inferred feeling as an observable fact. Feedback should describe behaviour and context with appropriate uncertainty.
Close the learning loop
Start with reviewed feedback after a conversation. Move selected suggestions into real time only when their accuracy, latency and usefulness justify interruption. Measure whether colleagues accept the advice, whether supervisors overturn it and whether subsequent attempts improve on the targeted skill. Invite employees to challenge a rating and use those cases to improve the evaluation set.
Keep coaching feedback separate from automatic employment decisions. Give the organisation explicit rules for data access, retention and who can see individual assessments. The practical deliverable is a learning loop: evidence, a useful suggestion, a chance to practise and a check on the next attempt. Reliable evaluation makes that loop credible and gives the team a way to improve both the coach and the workflow.
Sources & further reading
Written by Gilles Jacobs, AI & data engineering consultant at Exverbai.
