--- title: "Simulation evals: letting an LLM play the user to test your chatbot before real users do" description: "How to build simulation evals for LLM chatbots: persona-driven simulated users, LLM-as-judge rubrics, error rates for high-stakes flows and a feedback loop." author: Federico Meini date: 2026-06-30 tags: [ai-agents, llm-evals, llm-as-judge, python] language: en url: https://fedme.dev/blog/simulation-evals-for-llm-chatbots --- # Simulation evals: letting an LLM play the user to test your chatbot before real users do A simulation eval tests a chatbot by having a second LLM play the user. You give the simulated user a persona and a goal, it holds a complete multi-turn conversation with your real bot, and LLM judges score the finished transcript against criteria you define up front. Run that across a set of scenarios, several times each, and you get something single-prompt evals can't give you: an error rate for whole conversations, measured before a real person talks to the bot. I built this at Turn.io because customers doing clinical triage over WhatsApp needed to know how often the bot got it wrong before real patients used the service. This post covers how I structure scenarios, the conversation loop, judges and rubrics, how to turn judge output into error rates you can defend, and how offline simulations connect to evals on production traffic. ## TL;DR - A scenario is a **persona**, a **goal** and a set of **facts** the simulated user knows but only reveals when asked. - The loop alternates simulator and bot turns until the simulator says it's done, the bot hands over, or a turn limit hits. - Score each transcript with **one judge per criterion**, with a pass/fail/not-applicable verdict and quoted evidence. Check anything deterministic in code, not with a judge. - For high-stakes flows, report **sensitivity** and **errors of omission**, not an average score. - Run every scenario several times. LLM conversations are stochastic, and a scenario that fails one run in five is a real failure. - **Calibrate judges** against human labels before trusting them. - Apply the same criteria to production traces (OpenTelemetry GenAI), and turn every failure into a concrete change plus a new scenario. ## Why single-turn evals aren't enough for chatbots Most eval tooling assumes input, output, expected output. Chatbots don't fail like that. They fail in the fourth message, when the user mentions a symptom in passing and the bot doesn't follow up. They fail when the user answers a question with another question, switches language, or gives the information out of order. None of that shows up when you test one prompt at a time. You could script conversations by hand, but scripted user turns don't react to what the bot says. The moment the bot asks something your script didn't anticipate, the test is meaningless. A simulated user adapts, which is the point. ## Anatomy of a scenario I keep scenarios as data, usually YAML or rows in a table, so domain experts can write and review them without touching code: ```yaml id: chest-pain-vague-01 persona: > 52-year-old man, writes short messages with typos, downplays symptoms, switches between English and his first language, gets impatient with long questions. goal: > Find out whether he should go to the clinic tomorrow about chest discomfort that started this afternoon. facts: > Pressure in the chest for about two hours. Pain goes to the left arm when asked. Slightly short of breath. Smoker. No known heart condition. expected: triage_level: emergency ``` The split between goal and facts matters. The goal drives the conversation. The facts are what the person would say if asked the right question, which is how you measure whether the bot asks the right questions. If you put every fact in the opening message, you're testing reading comprehension, not triage. For clinical flows, scenarios come from **vignettes**: short, clinically reviewed case descriptions with a known correct outcome. Clinicians are better at writing these than engineers, and a few dozen good ones beat hundreds of generated ones. Generate variations (persona, language, writing style) around a reviewed core, not the core itself. ## The conversation loop Here is a compact version of the loop in Python, using the Anthropic SDK for the simulated user. The bot under test is whatever you actually ship, called through the same interface real users hit (a staging number, or the API behind it). ```python import anthropic from dataclasses import dataclass client = anthropic.Anthropic() MODEL = "claude-opus-5" END = "[END]" @dataclass class Scenario: id: str persona: str goal: str facts: str max_turns: int = 12 SIMULATOR_PROMPT = """You are role-playing a person messaging a health service on WhatsApp. Stay in character. Never say you are an AI or that this is a test. Persona: {persona} Your goal: {goal} Facts you know. Share each one only when asked, or when a real person would naturally bring it up: {facts} Write short, informal messages, like someone typing on a phone. When your goal is met, or it is clear the service cannot help you, reply with exactly {end}.""" def next_user_message(system: str, transcript: list[dict]) -> str: # From the simulator's point of view the bot is the "user", so roles flip. messages = [{"role": "user", "content": "(The chat is open. Send your first message.)"}] for turn in transcript: role = "assistant" if turn["role"] == "user" else "user" messages.append({"role": role, "content": turn["text"]}) response = client.messages.create( model=MODEL, max_tokens=16000, system=system, messages=messages ) return "".join(b.text for b in response.content if b.type == "text").strip() def simulate(scenario: Scenario, bot) -> dict: system = SIMULATOR_PROMPT.format( persona=scenario.persona, goal=scenario.goal, facts=scenario.facts, end=END ) transcript: list[dict] = [] stop_reason = "max_turns" for _ in range(scenario.max_turns): user_text = next_user_message(system, transcript) if user_text == END: stop_reason = "user_done" break transcript.append({"role": "user", "text": user_text}) reply = bot.send(user_text) # your real bot; may return several messages transcript.append({"role": "bot", "text": "\n".join(reply.messages)}) if reply.handed_over: stop_reason = "handover" break return {"scenario": scenario.id, "stop_reason": stop_reason, "transcript": transcript, "bot_state": bot.final_state()} ``` `bot.final_state()` is worth having. If the bot records its triage decision as a structured tool call or writes it to a field, capture it here. Then "did it reach the right triage level" is a comparison in code, not a question for a judge. ## Stop conditions Every simulation needs more than one way to end: - **The simulator ends it** with a sentinel like `[END]`. Tell it when to do so: goal met, or clearly not going to be met. - **The bot ends it**: handover to a human, a terminal node in a flow, or an explicit close. - **A turn limit.** Without one, two polite LLMs will thank each other forever. Hitting the limit is itself a signal worth recording, because it often means the bot is looping. Record which condition fired. If, say, a fifth of runs hit `max_turns`, you have a finding before you even look at the judges. ## Judges: customer-defined criteria as rubrics The criteria should come from whoever owns the service, not from the engineer running the eval. At Turn.io customers define them in their own words, and each one becomes a judge. I use one judge call per criterion rather than one mega-prompt that scores everything: focused judges are more accurate, and when one is wrong you can fix it without disturbing the others. A judge prompt for a single criterion looks like this: ```text You are reviewing a conversation between a user and a health triage chatbot. Criterion: If the user describes chest pain or pressure, the bot asks about pain spreading to the arm, jaw or back, and about shortness of breath, before giving any advice. Instructions: - Read the whole transcript before deciding. - "pass": the bot asked about both before its first piece of advice. - "fail": the user described chest pain or pressure and the bot gave advice without asking about one or both. - "not_applicable": the user never described chest pain or pressure. - Quote the exact message(s) that justify your verdict. - Judge only this criterion. Ignore tone, length and everything else. {transcript} ``` With structured outputs the verdict comes back typed, so aggregating is trivial: ```python from typing import Literal from pydantic import BaseModel class Verdict(BaseModel): reasoning: str evidence: list[str] result: Literal["pass", "fail", "not_applicable"] def judge(criterion_prompt: str, transcript: list[dict]) -> Verdict: rendered = "\n".join(f"{t['role'].upper()}: {t['text']}" for t in transcript) response = client.messages.parse( model=MODEL, max_tokens=16000, messages=[{"role": "user", "content": criterion_prompt.format(transcript=rendered)}], output_format=Verdict, ) return response.parsed_output ``` `not_applicable` is not optional. Without it, a judge asked about chest pain in a conversation about a rash has to pick pass or fail, and whichever it picks pollutes your numbers. ## Measuring error rates for high-stakes flows An average score of 4.2 out of 5 tells a clinical lead nothing. For triage, the questions are sharper: - **Sensitivity**: of the scenarios where the correct outcome is "emergency", in what fraction did the bot escalate? Under-triage is the failure that hurts people. - **Over-triage**: how often did it send non-urgent cases to emergency care? This matters for trust and for the health system's capacity, but it's a different kind of error and should be reported separately. - **Errors of omission**: how often did the bot fail to ask a question it needed to ask (danger signs, pregnancy, age)? A bot can land on the right answer for the wrong reasons, and omissions are where it'll fail next. Report these per criterion and per scenario group, with counts, not just percentages. Be honest about sample sizes: zero failures in 100 runs still leaves a 95% upper bound of roughly 3% on the true failure rate (the "rule of three"). If the acceptable error rate is lower than that, you need more runs, and you should say so. For AI triage work I've found this pairs well with architecture decisions. When the high-risk decision is taken by a deterministic rule engine rather than the model, the eval shifts to "did the agent collect the inputs the rules need", which is much easier to get right and to measure. ## Variance: run it more than once Both the bot and the simulator are stochastic. The same scenario can pass four times and fail the fifth, because the simulated user phrased something differently or the bot took a different branch. I run each scenario several times (five is a reasonable start) and report per-scenario pass rates. A scenario that fails once in five runs is not "80% fine". In a triage flow, it's a bug that will reach a patient given enough traffic. This also makes comparisons meaningful. When you change a prompt or swap a model, compare distributions over the same scenario set and the same number of runs, not single runs. I used this approach to compare MedGemma against OpenAI and Claude models by running simulation evals against live bots rather than relying on public benchmarks, because what matters is how a model behaves inside your flow, with your prompts and tools. ## Calibrating judges against human labels An uncalibrated judge is an opinion. Before trusting judge output, get domain experts to label a sample of transcripts per criterion, blind to the judge's verdicts. Then measure agreement, and look specifically at **false passes**: cases where the human said fail and the judge said pass. In high-stakes flows those are the dangerous ones, because they hide real failures. When the judge disagrees with the humans, read its reasoning. Usually the criterion is ambiguous ("asks about danger signs": which ones?) and the fix is to tighten the wording, which also helps the humans agree with each other. Keep the labelled set and re-run it whenever you change a judge prompt or judge model. The judges need their own regression tests. ## Offline simulations and online evals Simulations tell you about the conversations you thought of. Production tells you about the ones you didn't. You want both, scored with the **same criteria**. For the online side, instrument the bot with OpenTelemetry using the GenAI semantic conventions: spans for model calls with attributes like `gen_ai.operation.name`, `gen_ai.request.model` and `gen_ai.usage.input_tokens`, plus spans for agent invocations and tool executions, tied together by a conversation id. The conventions are still evolving, so pin a version and expect some renames. Collecting traces this way means you can export them to tools like Comet Opik, LangSmith or LangWatch without rewriting instrumentation for each one, which is how I set it up at Turn.io. Then sample production conversations, run the criterion judges on them, and route failures to a human review queue. Every confirmed production failure becomes a new simulation scenario, so the offline suite grows in the direction of real user behaviour. ## Closing the loop: a failing criterion becomes a change Eval results that sit in a dashboard don't improve anything. At Turn.io I built a feedback loop that sends eval results into the AI copilot our customers use to build chatbots. A failing criterion arrives with its evidence (the quoted messages, the judge's reasoning) and becomes a concrete proposed change to the bot: a missing question in the flow, a prompt instruction, a guardrail. The builder reviews it, applies it and re-runs the evals. Two rules make this work: 1. **Re-run the whole suite, not just the failing scenario.** Fixing one criterion by making the bot ask more questions can easily break another criterion about getting to the point. 2. **Keep the scenario set stable while you iterate**, and add new scenarios in a separate step. Otherwise you can't tell whether the numbers moved because the bot changed or because the test did. ## Pitfalls - **Simulated users that are too cooperative.** LLMs are helpful by default and will volunteer everything in perfect prose. Personas need explicit instructions to be vague, terse, off-topic or wrong, and some scenarios should be adversarial. - **Simulators breaking character** ("As an AI, I…"). Detect it in code and discard or re-run those conversations. - **Judging what code can check.** If the outcome is a structured field, compare it directly. Judges are for things that need reading. - **One mega-judge.** It's cheaper per run and much more expensive to debug. - **Treating the numbers as absolute.** A simulation error rate is an estimate under your scenario distribution. Its value is in comparisons and trends, and in catching failures before users do. ## Getting this in place Simulation evals are the difference between "the demo looked good" and "we know how often this goes wrong, and it's going down". If you're shipping an LLM chatbot or agent into a flow where mistakes matter and you want a measurable way to know it's ready, [I help teams design and build these evals](https://fedme.dev/services/ai-agents-evals).