A new study tested four specialized AI agents that roleplay therapy patients, score transcripts, and write feedback. Agreement with 16 human raters: 0.96 on MI, 0.93 on CBT — where 1.0 means perfect agreement and 0.0 means none

The bottleneck in psychotherapy training is the expert human rater. A new evaluation in JMIR Medical Education tests a four-agent LLM framework that roleplays patients, scores transcripts, and writes feedback, asking whether the rubric-linked scoring inside a synthetic loop can stand in for some of that rater time.

The framework, designed by a team led by Mohammad Amin Kamaleddin at the University of Toronto, splits a single training encounter into four specialized AI roles: a Patient agent that holds a structured profile (alcohol use, smoking cessation, weight management, and medication adherence for MI; DSM-5-reformatted cases for CBT), a Student agent that conducts the session, an Evaluator agent that scores the transcript against a study-specific rubric, and a Feedback agent that produces criterion-referenced recommendations. The Patient profiles and the simulation loop are entirely synthetic; no real trainees or real patients participated in the test. Full methods and per-criterion results are in the open-access PMC text.

The headline test is whether the Evaluator can tell a novice from an expert. The authors engineered Student-agent profiles at three competence levels (novice, intermediate, expert) and asked the Evaluator to score each transcript. The result was a clean monotonic gradient. On MI, mean overall scores rose from 1.18 (95% CI 1.16-1.20) for novice profiles to 1.75 (1.69-1.81) for intermediate and 3.57 (3.48-3.66) for expert. On CBT, the corresponding scores moved from 0.83 (0.78-0.88) to 2.18 (2.10-2.26) to 4.38 (4.28-4.48). The two scales are not directly comparable: MI used a 1-5 scale derived from the Motivational Interviewing Treatment Integrity (MITI) and Motivational Interviewing Skill Code (MISC) instruments, and CBT used a 0-6 scale derived from the Cognitive Therapy Rating Scale (CTRS). Criterion-level planned contrasts held up after Benjamini-Hochberg false-discovery-rate correction at q<0.05, the authors report.

Two secondary tests checked whether the scoring aligned with anything outside the synthetic loop. The first used 133 publicly available transcripts from the AnnoMI dataset, a public MI counseling corpus; the Evaluator's global scores agreed with the corpus's coarse, metadata-derived high/low session labels at 91.7% classification accuracy. That number is bounded by the labels themselves: AnnoMI carries expert utterance-level annotations, but the study only used corpus-level high/low flags, so this is orientation against a coarse external signal, not validation against item-level fidelity coders.

The second test was a 16-rater human benchmark. For each modality, 16 independent raters (3 practicing physicians plus 13 raters recruited through Prolific) scored 10 selected transcripts. Interrater agreement among the humans was strong for MI (ICC 0.866) and moderate for CBT (ICC 0.769). Agreement between the Evaluator and the human consensus was higher: ICC 0.959 for MI and 0.934 for CBT. The 16-rater panel is pragmatic, not criterion-validated, so the agreement numbers describe alignment with a small reference panel, not alignment with a clinical ground truth.

A secondary prompt-augmentation analysis checked whether the Feedback agent's narrative could push the Evaluator's scoring upward on the same novice transcripts. The shift was real but small: overall MI scores moved from 1.18 to 1.44, and CBT scores from 0.83 to 1.01. The system is responsive to feedback text, but the move stays well below the expert level, which is consistent with rubric-bound scoring rather than with style transfer.

The framework ranks prompt-defined competence levels, agrees with a 16-rater panel, and tracks coarse external labels inside a closed synthetic loop. Whether that scoring transfers to real trainees producing real sessions against item-level clinical fidelity benchmarks is the open test. The authors position the work as preclinical, proof-of-concept evidence pending prospective studies with human learners, standardized or real patients, and formal psychometric testing.

The next move is whether the framework can survive that test. The mechanism that produces the result is the same one that limits it: the Evaluator is grading a synthetic Student against a prompt-defined competence rubric, and the 16-rater panel is a pragmatic reference, not a clinical validation cohort. If real-trainee scores correlate with the synthetic-loop scores, the bottleneck the paper opens with becomes addressable at training-program scale. If they do not, the field still learns where rubric-only AI scoring breaks.