# AI agent orchestration and LLM evals (simulation, LLM-as-judge, tracing)

> Freelance AI engineer for agent orchestration and evals. Multi-agent handover, sandboxed tools, guardrails, simulation evals, LLM-as-judge, OpenTelemetry tracing and model selection.

URL: https://fedme.dev/services/ai-agents-evals
Provided by: Federico Meini (federico.meini@gmail.com)

## AI agents you can put in front of real users, and the evals that prove it.

Since 2023 I have owned the AI features at Turn.io end to end, including the agent harness, sandboxed tools, simulation and online evals, and the loop that turns failing evals into fixes. Much of it runs in healthcare, where "it seems to work" is not good enough.

## What you get

- An agent architecture that fits the risk. That means deterministic rules where decisions are high-stakes, LLMs where flexibility helps, and a clean handover to humans.
- Tools that agents can call safely, in a sandbox, connected to your real systems.
- Offline simulation evals that measure error rates before launch, and online evals on production traces after.
- A feedback loop where every failing criterion becomes a concrete change and a re-run.
- Model choices backed by your own evals, including open models you host yourself when cost or privacy matter.

## Agents are easy to demo and hard to trust

A demo agent answers the five questions you tried. A production agent meets thousands of people who phrase things differently, leave out key details, change their minds and occasionally try to break it. In healthcare, one missed danger sign is one too many.

At Turn.io I own the AI features end to end:

- **The agent harness** organisations use to run AI agents on WhatsApp, covering tool use, orchestration, guardrails and handover between agents and to humans.
- **Sandboxed tools.** Agents execute tools and code inside a Lua sandbox, which lets customers connect them to medical records and external APIs without being able to break the platform.
- **Offline simulation evals.** An LLM plays the user from a persona and a goal, has the whole conversation with the bot, and LLM judges score the transcript against criteria the customer defines. I built these because teams doing clinical triage needed to know the error rate before real patients used the service.
- **Online evals.** OpenTelemetry GenAI collection and export of all AI traffic, so teams can score live conversations in Comet Opik, LangSmith or LangWatch.
- **The feedback loop.** Eval results flow into the AI copilot that builders use, so a failing criterion becomes a concrete change to the bot, and the builder re-runs the evals to check it worked.

I have also designed triage systems with **two cooperating agents and a deterministic rule engine** for high-risk decisions, measured **sensitivity and errors of omission** with simulation evals on vignette datasets, and compared **MedGemma** with frontier models on live bots rather than benchmarks. For the MedGemma work I quantised the 27B model to FP8, published it on Hugging Face and deployed it on vLLM on Modal with cold starts of about 21 seconds (down from ~100), so it is paid per use. I wrote up [how the deployment works](https://fedme.dev/blog/serving-medgemma-27b-on-modal-vllm).

## What I build

**Agent architecture.** Single agent, orchestrator and specialists, or LLM plus rules engine, chosen according to risk and testability rather than hype.

**Tools and integrations.** Well-typed tools with timeouts, sandboxes and audit logs, connected to your CRM, EMR or internal APIs.

**Eval harnesses.** Personas and goals, scenario datasets built with domain experts, judges with binary criteria, repeated runs to measure variance, and dashboards that show trends across versions.

**Tracing and online evals.** OpenTelemetry GenAI spans across LLM calls, tools and handovers, sampled for scoring in production.

**Model selection and hosting.** Frontier APIs (Anthropic, OpenAI, Gemini) or self-hosted open models (vLLM, quantisation, serverless GPUs), chosen based on your evals.

## How I approach it

1. **Start on a call with the people who will use it.** I run discovery with your users and domain experts, and most of the criteria come from them.
2. **Write the evals before polishing the prompt.** Define what "good" means, then iterate.
3. **Ship a first version fast,** measure it on simulations, then on real traffic.
4. **Close the loop.** Every failure becomes a test case.

## Related

- [WhatsApp Business Platform engineering](https://fedme.dev/services/whatsapp-business-platform): where many of these agents live.
- [Agentic coding for engineering teams](https://fedme.dev/services/agentic-coding): using agents to build software, not just ship them.

## Ways to engage

- **Evals sprint**: Two to three weeks to define criteria with your domain experts, build a simulation and judging harness, and get a baseline for your agent.
- **Agent build**: Design and ship an agent or multi-agent system end to end, from discovery with your users to production.
- **Architecture and model review**: Review an existing agent for failure modes, cost and latency, and compare models on your own data.

## Stack

Anthropic Claude, OpenAI, Gemini, MedGemma, vLLM & Modal, OpenTelemetry GenAI, LangSmith / Opik / LangWatch, RAG & pgvector, Python, Elixir

## FAQ

### What are simulation evals?

An LLM plays a user with a persona and a goal and has a whole multi-turn conversation with your agent. Then LLM judges score the transcript against criteria your domain experts define. Running hundreds of simulations tells you the error rate before real users see the agent.

### Is LLM-as-judge reliable?

It is reliable when the criteria are specific and binary, the judge sees the evidence it needs, and you calibrate it against human labels on a sample. Vague rubrics give noisy scores. I design criteria with the people who know the domain.

### How do you evaluate agents in production?

By collecting traces of real conversations using the OpenTelemetry GenAI conventions, scoring samples with the same judges used offline, and exporting to tools like LangSmith, Opik or LangWatch. That way offline and online results are comparable.

### Which model should we use?

The one that passes your evals at an acceptable cost and latency. I compare frontier models and open models on your own simulations rather than on public benchmarks. Sometimes a quantised open model on serverless GPUs wins, and sometimes it doesn't.

### Do you work on multi-agent systems?

Yes, including triage agents that hand over to specialists, deterministic rule engines for high-risk decisions, and handover to human operators. I only reach for multiple agents when it makes the system easier to test.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact