Testing a medical triage agent with LangWatch Scenario and pytest

LangWatch Scenario is the easiest way I’ve found to put multi-turn simulation tests for an AI agent into an ordinary pytest suite. You wrap your agent in a small adapter, describe the situation in plain English, list the criteria a judge should check, and Scenario plays the conversation out with a simulated user until the judge reaches a verdict. At Turn.io I used it with OneDay Health, which runs primary-care clinics in Uganda. Their nurses use an AI agent that walks them through the clinical guidelines while they see a patient, and Scenario let us simulate those conversations before the agent reached a real clinic, and I published a stripped-down version of that first setup as a demo: fedme/agent_simulation_tests_example.

In an earlier post I described simulation evals from first principles: personas, hand-rolled conversation loops, one judge per criterion, error rates. This post is the practical companion. It’s about a library that gives you most of that loop for free, and about the code in the demo repo.

TL;DR

• Scenario has three moving parts: an AgentAdapter around your agent, a UserSimulatorAgent that plays the user from a scenario description, and a JudgeAgent that checks a list of criteria in plain English.
• A triage scenario is a short description: who the user is, what they report up front, and which facts they only reveal if asked. That is what tests whether the agent asks the right follow-up questions.
• Give the judge the same clinical guidelines the agent uses. Otherwise it grades medicine from its own training data.
• Tests can run free (the simulator and judge drive everything) or scripted (fixed turns, code assertions, then let it run).
• It’s plain pytest, so it runs in CI with a model API key (plus an optional LangWatch key). The demo requires langwatch-scenario>=0.7.13; the API used here is the same in the current 1.x releases.

What’s in the demo repo

The repo is deliberately small: a pyproject.toml managed with uv, a .env.example with OPENAI_API_KEY and LANGWATCH_API_KEY, the getting-started recipe test from the Scenario docs, and a triage test file with two scenarios. The agent under test is a single LLM call. Its system prompt embeds a primary-care treatment handbook for nurses and clinical officers, and tells the model to ask one or two follow-up questions at a time, follow the handbook’s decision trees, and give the diagnosis and treatment exactly as written there, or recommend a hospital visit if nothing matches.

In the snippets below I’ve trimmed long strings; the names and structure are what’s in the repo.

Wrapping the agent under test

Scenario doesn’t care how your agent is built. You subclass scenario.AgentAdapter and implement one async method, call, which receives a scenario.AgentInput and returns the agent’s reply:

import litellm
import scenario

scenario.configure(default_model="openai/gpt-4.1", max_turns=10, verbose=True)


@scenario.cache()
def generate_triage_response(messages) -> scenario.AgentReturnTypes:
    response = litellm.completion(
        model="openai/gpt-4.1",
        messages=[
            {"role": "system", "content": triage_system_prompt()},
            *messages,
        ],
    )
    return response.choices[0].message


class OneDayAgent(scenario.AgentAdapter):
    async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
        return generate_triage_response(input.messages)

AgentInput carries the full conversation as OpenAI-style messages, the new_messages since the agent last spoke, a thread_id, and the scenario state. You can return a string, an OpenAI-style message, or a list of messages. In a real project, call is where you hit whatever you actually ship: an HTTP endpoint, a LangGraph graph, a staging deployment. Use thread_id as the session id so each simulated conversation gets its own state.

scenario.configure sets the model for the simulator and the judge. Calls go through LiteLLM, so any provider works.

Describing a triage scenario

This is the first scenario from the repo, lightly cleaned up:

description = """
  The user is a nurse currently examining a 4 year old patient with one day of cough and fever.
  Malaria test negative. He's eating well and not vomiting.
  If asked, child's respiratory rate is 65.
  If asked, child has no chest indrawing or stridor.

  As the nurse, you need to describe the patient to the Agent and answer any follow-up
  questions so that the Agent can make a diagnosis.
"""

The description is shared: the user simulator uses it to play the part, and the judge reads it as context. The “If asked” lines are the important bit. They are the facts the simulated nurse knows but won’t volunteer, which is the goal/facts split from my earlier post expressed as one sentence each. The handbook says every child with a cough must have their breathing rate counted, and above 40 breaths per minute at that age counts as fast breathing. So this scenario only goes well if the agent asks for the respiratory rate instead of guessing from cough and fever.

The simulated user is a nurse seeing a patient in the clinic, because that’s who the agent supports. For a patient-facing self-triage flow you change the persona (“a mother messaging about her son, short messages, not sure what counts as fast breathing”) and nothing else. The default simulator already writes short, informal messages, which suits WhatsApp. If you need more control over the persona, UserSimulatorAgent takes its own system_prompt and model.

Judge criteria

Criteria are plain strings, written so that a clinician can read and edit them:

criteria = [
    "Agent provides accurate information closely following the OneDay medical guidelines",
    "Agent doesn't ask for the same information multiple times",
    "Agent asks correct follow-up questions according to the OneDay medical guidelines",
    "Agent provides a diagnosis that strictly follows the OneDay medical guidelines",
]

result = await scenario.run(
    name="4yo with cough and fever",
    description=description,
    agents=[
        OneDayAgent(),
        scenario.UserSimulatorAgent(),
        scenario.JudgeAgent(
            criteria=criteria,
            model="openai/gpt-4.1",
            system_prompt=judge_prompt(description, criteria),
        ),
    ],
)

assert result.success

The second scenario in the repo (an adult with five days of fever, night sweats and joint pain) pins the expected outcome directly in a criterion: the agent must reach a specific diagnosis from the handbook. That’s how you encode a clinical vignette with a known correct answer.

Under the hood, the judge runs after every turn with two tools: continue_test and finish_test. When it has enough information it returns a verdict for each criterion (true, false or inconclusive) plus its reasoning. It stops early if a “should not” criterion is already broken, and on the last turn it must decide. The ScenarioResult you get back has success, reasoning, passed_criteria and failed_criteria, so a failing test tells you which criterion failed and why.

The one change I made to the judge matters a lot in a medical context. By default the judge only sees the scenario and the criteria, so “follows the clinical guidelines” would be graded from the judge model’s general medical knowledge. The repo’s judge_prompt keeps the library’s default judge prompt and adds the handbook:

def judge_prompt(scenario_description: str, criteria: list[str]) -> str:
    return f"""
      <role>
      You are an LLM as a judge watching a simulated conversation as it plays out live
      to determine if the agent under test meets the criteria or not.
      </role>

      The agent under test helps nurses make medical diagnosis strictly following
      the OneDay medical guidelines, which are reported below:

      <guidelines>
      {clinical_guidelines()}
      </guidelines>

      <scenario>
      {scenario_description}
      </scenario>

      <criteria>
      {"\n".join(criteria)}
      </criteria>
    """  # + the default <goal> and <rules> sections, omitted here

The repo keeps the criteria generic because it was a first demo. For a triage agent going to production I’d add sharper, safety-focused ones: “Agent refers the child to hospital immediately if any danger sign is reported”, “Agent asks for the respiratory rate before giving a diagnosis for a child with cough”, “Agent does not recommend medicines or doses that are not in the guidelines”. For a patient-facing self-triage agent you’d flip the diagnosis criterion to “Agent does not give a diagnosis and recommends the right level of care”.

One difference from my earlier post: Scenario’s judge scores all criteria in a single call, where I recommended one judge per criterion. For regression tests that’s fine. When you need calibrated error rates, you can take result.messages and run your own per-criterion judges on the transcripts.

Free-running and scripted simulations

The two triage tests pass no script, so Scenario runs its default: the simulated user opens, the agent replies, the judge decides whether to continue, and so on until a verdict or max_turns. That is the mode to use for “does this case go well end to end”.

Scripts give you control over specific turns. Each step is scenario.user(), scenario.agent(), scenario.judge(), scenario.proceed(), scenario.succeed() or scenario.fail(), or any plain function that receives the scenario state. scenario.user("...") sends a fixed message, scenario.user() lets the simulator write one. This variant isn’t in the repo, but it uses the same agent and judge:

def follows_reply_format(state: scenario.ScenarioState) -> None:
    reply = state.last_message()["content"] or ""
    assert "<EXPLANATION>" in reply  # the system prompt requires it


judge = scenario.JudgeAgent(criteria=criteria, system_prompt=judge_prompt(description, criteria))

result = await scenario.run(
    name="4yo with cough and fever, terse opening",
    description=description,
    agents=[OneDayAgent(), scenario.UserSimulatorAgent(), judge],
    script=[
        scenario.user("4yo boy cough and fever since yesterday"),
        scenario.agent(),
        follows_reply_format,
        scenario.proceed(turns=5),
        scenario.judge(),
    ],
)

The opening is fixed so the case is reproducible, a deterministic rule is checked in code rather than by the judge, and then the simulation runs freely before a forced verdict. If a script ends without a verdict, the run fails and says so, which avoids tests that silently prove nothing.

Running it in pytest and CI

The tests are ordinary async pytest functions marked with @pytest.mark.agent_test and @pytest.mark.asyncio. Scenario ships a pytest plugin that registers the marker and prints a summary at the end with each scenario’s reasoning and passed criteria, and pytest-asyncio comes in as a dependency. Locally:

uv run pytest -s -m agent_test

With -s you watch the conversation turn by turn. LANGWATCH_API_KEY is optional: without it you only get terminal output, with it every simulated conversation shows up in LangWatch, which is much easier to read with a clinician than terminal logs. scenario.configure(debug=True) pauses at each user turn so you can type the message yourself, handy when a test fails and you want to poke at the agent.

The demo repo has no CI workflow, but adding one is a single job. Adapted from the Scenario docs:

- name: Run simulation tests
  run: uv run pytest -m agent_test
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    LANGWATCH_API_KEY: ${{ secrets.LANGWATCH_API_KEY }}
    SCENARIO_BATCH_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}

SCENARIO_BATCH_RUN_ID groups all scenarios from one CI run in LangWatch. For speed and repeatability, set a cache_key in scenario.configure: the agent function (that’s what the @scenario.cache() decorator in the repo is for), the simulator and the judge are then cached on disk, keyed on their arguments. Without a cache_key the decorator does nothing. The key doesn’t include your function’s code, so bump it whenever you change the prompt or model. And don’t measure error rates with caching on: a cached run is one sample replayed, and a triage flow needs many fresh runs.

Where this fits

Simulation testing matters in medicine because triage failures happen inside a conversation: the fourth message where a danger sign is mentioned in passing, a follow-up question that never gets asked. Single-prompt evals don’t see those. And because scenarios and criteria are plain English, the clinicians who own the guidelines can review exactly what’s being tested, which I’ve found is what builds trust in the results.

I think of three layers. Scenario tests are regression tests: a curated set of cases that must pass on every change, in CI. Offline simulation evals measure error rates: many scenarios, several runs each, calibrated judges, sensitivity for the cases that must be escalated. Online tracing covers the conversations you didn’t think of: production traces scored with the same criteria, and every confirmed failure becomes a new Scenario test. Scenario makes the first layer cheap enough that there’s no excuse to skip it.

The library has grown since I wrote the demo (it now covers red teaming and voice agents too), but the core is unchanged. Clone the demo repo, swap in your agent and your guidelines, and you’ll have your first multi-turn test running in an afternoon.

If you’re building an AI agent for a flow where mistakes reach real people and want tests and evals that show it’s ready, I help teams build exactly that.