# Federico Meini

> Freelance senior software engineer based in Italy (CET), working remotely worldwide in English and Italian. Specialises in WhatsApp Business Platform services, real-time distributed systems in Elixir/OTP, PostgreSQL and Elasticsearch performance at scale, AI agent orchestration and LLM evals, and agentic coding (Claude Code) adoption for engineering teams.

- Website: https://fedme.dev
- Email: federico.meini@gmail.com
- GitHub: https://github.com/fedme
- LinkedIn: https://www.linkedin.com/in/fedemeini
- Available for: fixed-scope reviews (1–2 weeks), builds, and part-time embedded senior engineering.

## Summary

Federico Meini has more than ten years of software engineering experience. For six years he has been a senior engineer at Turn.io, a WhatsApp platform used by the WHO, national health ministries and clinic networks, handling millions of messages a day. He was one of its first engineers.

Since 2023 he has owned Turn.io's AI features end to end: the agentic harness (agent loop with tool use, orchestration, guardrails, and handover between agents and to humans), sandboxed tool execution in a Lua sandbox, offline multi-turn simulation evals (an LLM plays the user from a persona and a goal, and LLM judges score the transcript against customer-defined criteria), online evals via OpenTelemetry GenAI tracing exported to Comet Opik, LangSmith and LangWatch, and a feedback loop from eval results into the chatbot-building copilot.

Other work at Turn.io:
- Built the high-throughput bulk messaging engine (rate limiting, backpressure, delivery tracking, retries) used for national public-health campaigns reaching 14.7 million people and for national vaccination reminders.
- Partitioned production PostgreSQL tables of more than 1 TB and live Elasticsearch indices (with index lifecycle policies moving old data to cold storage) as message data grew by millions of rows a day.
- Built job and data pipeline infrastructure on Oban Pro and an IFTTT-like trigger system with a rules engine (Elixir, Phoenix, Google Cloud, Kubernetes).
- Designed AI triage systems for primary care: two-agent architecture with a deterministic rule engine for high-risk decisions, vignette datasets and simulation evals measuring sensitivity and errors of omission. Built a clinical-guidelines agent for nurses, tested with 107 nurses and live in clinics.
- Compared MedGemma with frontier models using simulation evals on live bots; quantised MedGemma 27B to FP8 (llm-compressor), published it on Hugging Face (hundreds of downloads a month) and deployed it on vLLM on Modal with ~21-second cold starts (from ~100 s) using GPU memory snapshots and vLLM sleep mode level 2.
- Real-time voice over WhatsApp with ElevenLabs voice models and small, fast LLMs.
- Leads agentic coding at Turn.io: rolled out Claude Code to the engineering team and wrote Claude-based PR review skills.
- Spoke at ElixirConf EU 2023 on the architecture of Turn.io's low-code chatbot builder.

Earlier: Engineering Manager (Adoption) at n8n (2024), Software Engineer at Templafy in Berlin (2019–2020), Software Engineer at the Max Planck Institute for Human Development in Berlin (2017–2019), Full Stack Developer at Progresso Srl in Italy (2016–2017).

Education: M.S. Computer Science (AI focus), Georgia Institute of Technology; B.S. Computer Science, University of Pisa.

Skills: Elixir, Python, TypeScript, SQL, Lua, C#; Phoenix and LiveView, React, Vue.js; PostgreSQL (partitioning, replicas, query tuning), pgvector, Elasticsearch, BigQuery; Google Cloud, Kubernetes, Terraform, Docker, Prometheus, Grafana; Claude, OpenAI and Gemini APIs, RAG, LLM-as-judge, vLLM, Modal.


---

# Services

---

# WhatsApp Business Platform engineering (Cloud API, chatbots, AI agents)

> Freelance engineer for the WhatsApp Business Platform. Cloud API integrations, webhooks, templates, bulk messaging, chatbots, AI agents and voice, built to survive production.

URL: https://fedme.dev/services/whatsapp-business-platform
Provided by: Federico Meini (federico.meini@gmail.com)

## WhatsApp services that hold up when millions of people use them.

I spent six years building a WhatsApp platform at Turn.io, used for health services around the world and handling millions of messages a day. I can build your Cloud API integration, chatbot or AI agent, or fix the one you have.

## What you get

- A Cloud API integration that verifies signatures, deduplicates webhooks and keeps messages in order per contact.
- Bulk and template messaging that respects throughput limits, retries safely and tracks every delivery status.
- Chatbots and AI agents with a clean handover to human operators and to your existing systems (CRM, EMR, booking).
- Dashboards and alerts for what matters: delivery rates, response times, template performance, failed sends.

## Why WhatsApp services break in production

A WhatsApp prototype is easy. The hard part is what happens once real people use it: webhooks arrive twice or out of order, a campaign triggers thousands of replies in a minute, the 24-hour customer-service window closes in the middle of a conversation, a template gets paused, and the one message that mattered never gets a `delivered` status.

I have spent six years working on these problems at [Turn.io](https://www.turn.io), a WhatsApp platform used by organisations like the WHO and national health programmes to reach millions of people. That included the bulk send engine behind large public-health campaigns, the job and trigger infrastructure, the Postgres and Elasticsearch storage as message data grew by millions of rows a day, and since 2023 the AI agents that answer on WhatsApp.

## What I build

**Cloud API integrations.** A webhook receiver that verifies `X-Hub-Signature-256` against the raw body, answers fast and processes asynchronously, deduplicates by message ID and keeps each contact's messages in order. Outbound sending that handles throttling errors, template rules and media properly.

**Bulk and campaign messaging.** A demand-driven pipeline with rate limiting, backpressure, retries and delivery tracking, so a campaign to a million contacts neither overwhelms your system nor gets you throttled.

**Chatbots and AI agents.** Conversation flows, interactive messages (buttons, lists), AI agents with tools, and a handover to human operators that doesn't lose context. I can connect them to your CRM, EMR, booking or payments system.

**Voice.** WhatsApp calling combined with real-time voice models and small, fast LLMs, for conversations with low enough latency to feel natural.

## How I approach it

1. **Map the conversations first.** What are users trying to do, what must never go wrong, which messages are business-initiated and need templates?
2. **Design for at-least-once delivery.** Everything on WhatsApp can arrive twice or late, so idempotency and ordering come first.
3. **Ship a thin version early** and watch real traffic: delivery rates, response times, drop-off points.
4. **Measure and iterate.** For AI agents this means evals on simulated and real conversations, not vibes.

## Related

- [Real-time and distributed systems](https://fedme.dev/services/realtime-systems): the send engine and processing behind WhatsApp at scale.
- [AI agent orchestration and evals](https://fedme.dev/services/ai-agents-evals): agents that answer on WhatsApp, and how to prove they are safe.

## Ways to engage

- **Integration build**: Scoped build of a WhatsApp Cloud API integration or service, from webhook receiver to production launch.
- **Architecture review**: One to two weeks reviewing an existing WhatsApp integration for reliability, ordering, idempotency and cost.
- **Embedded senior engineer**: Part-time in your team for a few months while you launch or scale a WhatsApp service.

## Stack

WhatsApp Cloud API, Webhooks, Message templates, Interactive messages, WhatsApp calling, Elixir / Phoenix, Python, TypeScript, PostgreSQL

## FAQ

### Do you work with the WhatsApp Cloud API or the On-Premises API?

New projects should use the Cloud API, which Meta hosts, and that is what I build on today. I have also worked on older setups and can help plan a migration.

### Can you build an AI agent that answers on WhatsApp?

Yes. I built the agent harness that organisations use to run AI agents on WhatsApp at Turn.io, including tool use, guardrails and handover to humans. I can build the same for your service, together with the evals that show it behaves.

### How do you send millions of WhatsApp messages without getting throttled?

With a send pipeline that is demand-driven, applies per-number throughput limits with a token bucket, retries throttling errors with backoff and jitter, and records every status webhook. I built this kind of engine for national public-health campaigns.

### Do you only work with health organisations?

No. Most of my WhatsApp experience comes from health and social-impact services, where reliability matters a lot, but the engineering is the same for commerce, support, fintech or logistics.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact


---

# Real-time and distributed systems engineering (Elixir, OTP, Phoenix)

> Freelance Elixir and distributed systems engineer. High-throughput messaging pipelines, backpressure, rate limiting, job systems on Oban, Phoenix LiveView, and Kubernetes.

URL: https://fedme.dev/services/realtime-systems
Provided by: Federico Meini (federico.meini@gmail.com)

## Pipelines that bend under load instead of breaking.

I build high-throughput, real-time backends in Elixir and OTP. They include send engines that push millions of messages, job systems that never lose work, and live interfaces that update as things happen.

## What you get

- Throughput you can reason about, with backpressure, bounded queues and rate limits designed in rather than bolted on.
- Durable background work (retries, scheduling, uniqueness, rate-limited queues) with Oban.
- Live, collaborative interfaces with Phoenix LiveView and PubSub.
- Observability that explains incidents: telemetry, Prometheus metrics, Grafana dashboards, useful alerts.

## The problem with "it worked in staging"

Real-time systems rarely fail at average load. They fail when a campaign goes out and ten thousand people reply in the same minute, when a downstream API starts throttling, or when a retry storm doubles the traffic you were already struggling with.

At Turn.io I built the **bulk messaging engine** used for national public-health campaigns, including one that reached 14.7 million people, and for national vaccination reminders. It combines rate limiting, backpressure, delivery tracking and retries. I also built the **job and data pipeline infrastructure on Oban Pro** and an **IFTTT-style trigger system with a rules engine**, all in Elixir and Phoenix on Google Cloud and Kubernetes.

## What I build

**High-throughput pipelines.** Demand-driven stages (GenStage, Broadway) where consumers pull work, queues stay bounded and throughput converges on whatever the slowest dependency allows.

**Rate limiting and retries done properly.** Token buckets per sender or tenant, retries with exponential backoff and jitter, idempotency keys so a retry never sends the same message twice.

**Durable job systems.** Oban for work that must not be lost: scheduled sends, webhooks to third parties, data exports, cron-style automations, with uniqueness and per-queue rate limits.

**Rules and trigger engines.** Event-driven automations ("when X happens and Y is true, do Z") that non-engineers can configure safely.

**Live interfaces.** Phoenix LiveView and PubSub for dashboards and inboxes that update in real time without a separate frontend team.

## How I approach it

1. **Model the load.** Averages, peaks, bursts, and the rate limits you don't control.
2. **Make every queue bounded and every stage demand-driven.** Decide up front what happens when you are overloaded: slow down, shed, or defer.
3. **Instrument before optimising.** Telemetry events on every stage, so you can see where time goes.
4. **Load test with realistic traffic shapes,** including the failure modes.

## Related

- [WhatsApp Business Platform engineering](https://fedme.dev/services/whatsapp-business-platform)
- [PostgreSQL and Elasticsearch at scale](https://fedme.dev/services/postgres-elasticsearch): where all those messages end up.

## Ways to engage

- **Scaling review**: One to two weeks profiling a system under load, finding the real bottlenecks and writing a prioritised plan.
- **Build**: Design and build a pipeline, job system or real-time feature end to end, with load tests before launch.
- **Embedded senior engineer**: Part-time in your team to lead a scaling effort and level up the team on OTP.

## Stack

Elixir, Erlang/OTP, Phoenix & LiveView, GenStage / Broadway, Oban Pro, PostgreSQL, Google Cloud, Kubernetes, Prometheus & Grafana

## FAQ

### Why Elixir for real-time systems?

The BEAM gives you millions of cheap, isolated processes, supervision trees that restart what fails, and message passing that maps naturally onto messaging workloads. It makes per-user ordering, backpressure and graceful degradation much easier to get right than in most runtimes.

### What is backpressure and why does it matter?

Backpressure means that downstream stages tell upstream stages how much work they can take. Without it, a traffic spike fills memory and queues until something falls over. With it, the system slows down gracefully and recovers on its own.

### Can you work on a system that isn't in Elixir?

Yes. The ideas carry over (bounded queues, demand, idempotency, rate limiting), and I also write Python and TypeScript. I'm most productive in Elixir, though.

### Do you do load testing?

Yes. I load test before launch and reproduce production-shaped traffic, including bursts, slow downstream APIs and retries, because that is where the surprises are.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact


---

# PostgreSQL and Elasticsearch performance at scale (partitioning, tuning, ILM)

> Freelance PostgreSQL and Elasticsearch performance engineer. Table partitioning past 1 TB, query tuning, replicas, pgvector, and index lifecycle management for search at scale.

URL: https://fedme.dev/services/postgres-elasticsearch
Provided by: Federico Meini (federico.meini@gmail.com)

## Get your database fast again, and keep it fast as data grows.

At Turn.io I partitioned production PostgreSQL tables of more than 1 TB and live Elasticsearch indices while message data grew by millions of rows a day. I can do the same for your slow queries, bloated tables and expensive search clusters.

## What you get

- A clear diagnosis of why things are slow, backed by EXPLAIN plans, pg_stat_statements and real numbers.
- Partitioning and retention strategies for large, append-heavy tables, rolled out without downtime.
- Query and index tuning that fixes the queries that actually hurt, not all of them.
- Elasticsearch clusters with sensible shard sizes, hot/warm/cold tiers and index lifecycle policies that control cost.

## Messaging data grows faster than anything else

Every message, status update and event is a row. At Turn.io message data grew by **millions of rows a day**, and the tables that powered inboxes, analytics and search slowly became the bottleneck. Vacuum took longer, indexes stopped fitting in memory, and deleting old data locked things up.

I **partitioned production PostgreSQL tables of more than 1 TB** to get the performance back, and did the same for **live Elasticsearch indices**, with index lifecycle policies that move older data to cold storage.

## What I do

**Diagnose.** `pg_stat_statements`, `EXPLAIN (ANALYZE, BUFFERS)`, bloat and vacuum stats, index usage, lock contention. For Elasticsearch: shard sizes, heap pressure, slow logs, mappings.

**Fix the queries that matter.** Better indexes (partial, covering, multicolumn in the right order), rewritten queries, keyset pagination, fewer N+1s from the ORM.

**Partition large tables.** Choose a partition key that matches your queries, migrate without downtime, create future partitions automatically, and make retention a cheap `DETACH PARTITION ... CONCURRENTLY` instead of a huge `DELETE`.

**Scale reads.** Read replicas, routing read-only traffic, and understanding replication lag.

**Tame search costs.** Rollover, hot/warm/cold tiers, shard sizing, force-merge and ILM policies that keep Elasticsearch fast for recent data and cheap for old data.

## How I approach it

1. **Measure first.** No tuning without a baseline and a target.
2. **Fix the top offenders.** In most systems a handful of queries cause most of the pain.
3. **Plan migrations like deploys,** with small steps, reversible changes and a rollback path for each.
4. **Leave the team with the tools** (dashboards, runbooks, conventions) to keep it fast.

## Related

- [Real-time and distributed systems](https://fedme.dev/services/realtime-systems)
- [AI agent orchestration and evals](https://fedme.dev/services/ai-agents-evals): RAG and vector search on Postgres.

## Ways to engage

- **Database health check**: A one-week review of your Postgres or Elasticsearch setup, with a prioritised list of fixes and their expected impact.
- **Partitioning / migration project**: Plan and execute a zero-downtime move to partitioned tables or a new index strategy, including backfills and rollback plans.
- **On-call performance help**: A block of hours to help when a query or cluster is on fire.

## Stack

PostgreSQL, Declarative partitioning, pg_stat_statements, Read replicas, pgvector, Elasticsearch, Index lifecycle management, BigQuery, Ecto

## FAQ

### When should I partition a PostgreSQL table?

When a table is large and mostly append-only (events, messages, logs), most queries filter on a time range or tenant, and you need to delete old data cheaply. Partitioning is not a general speed-up, so the partition key has to match how you query.

### Can a large table be partitioned without downtime?

Usually yes. Common approaches are attaching the existing table as the first partition behind a CHECK constraint, or dual-writing and backfilling in batches. Indexes are built concurrently on each partition and then attached. The details depend on your constraints and traffic.

### What does Elasticsearch index lifecycle management do?

ILM automates what happens to indices as they age. It rolls over the write index at a size or age, moves older indices to cheaper warm and cold nodes, shrinks or force-merges them, and finally deletes them. It is the main lever for search cost.

### Do you also work with pgvector?

Yes. I use pgvector for RAG and semantic search where keeping vectors next to relational data in Postgres is simpler than running a separate vector database.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact


---

# AI agent orchestration and LLM evals (simulation, LLM-as-judge, tracing)

> Freelance AI engineer for agent orchestration and evals. Multi-agent handover, sandboxed tools, guardrails, simulation evals, LLM-as-judge, OpenTelemetry tracing and model selection.

URL: https://fedme.dev/services/ai-agents-evals
Provided by: Federico Meini (federico.meini@gmail.com)

## AI agents you can put in front of real users, and the evals that prove it.

Since 2023 I have owned the AI features at Turn.io end to end, including the agent harness, sandboxed tools, simulation and online evals, and the loop that turns failing evals into fixes. Much of it runs in healthcare, where "it seems to work" is not good enough.

## What you get

- An agent architecture that fits the risk. That means deterministic rules where decisions are high-stakes, LLMs where flexibility helps, and a clean handover to humans.
- Tools that agents can call safely, in a sandbox, connected to your real systems.
- Offline simulation evals that measure error rates before launch, and online evals on production traces after.
- A feedback loop where every failing criterion becomes a concrete change and a re-run.
- Model choices backed by your own evals, including open models you host yourself when cost or privacy matter.

## Agents are easy to demo and hard to trust

A demo agent answers the five questions you tried. A production agent meets thousands of people who phrase things differently, leave out key details, change their minds and occasionally try to break it. In healthcare, one missed danger sign is one too many.

At Turn.io I own the AI features end to end:

- **The agent harness** organisations use to run AI agents on WhatsApp, covering tool use, orchestration, guardrails and handover between agents and to humans.
- **Sandboxed tools.** Agents execute tools and code inside a Lua sandbox, which lets customers connect them to medical records and external APIs without being able to break the platform.
- **Offline simulation evals.** An LLM plays the user from a persona and a goal, has the whole conversation with the bot, and LLM judges score the transcript against criteria the customer defines. I built these because teams doing clinical triage needed to know the error rate before real patients used the service.
- **Online evals.** OpenTelemetry GenAI collection and export of all AI traffic, so teams can score live conversations in Comet Opik, LangSmith or LangWatch.
- **The feedback loop.** Eval results flow into the AI copilot that builders use, so a failing criterion becomes a concrete change to the bot, and the builder re-runs the evals to check it worked.

I have also designed triage systems with **two cooperating agents and a deterministic rule engine** for high-risk decisions, measured **sensitivity and errors of omission** with simulation evals on vignette datasets, and compared **MedGemma** with frontier models on live bots rather than benchmarks. For the MedGemma work I quantised the 27B model to FP8, published it on Hugging Face and deployed it on vLLM on Modal with cold starts of about 21 seconds (down from ~100), so it is paid per use. I wrote up [how the deployment works](https://fedme.dev/blog/serving-medgemma-27b-on-modal-vllm).

## What I build

**Agent architecture.** Single agent, orchestrator and specialists, or LLM plus rules engine, chosen according to risk and testability rather than hype.

**Tools and integrations.** Well-typed tools with timeouts, sandboxes and audit logs, connected to your CRM, EMR or internal APIs.

**Eval harnesses.** Personas and goals, scenario datasets built with domain experts, judges with binary criteria, repeated runs to measure variance, and dashboards that show trends across versions.

**Tracing and online evals.** OpenTelemetry GenAI spans across LLM calls, tools and handovers, sampled for scoring in production.

**Model selection and hosting.** Frontier APIs (Anthropic, OpenAI, Gemini) or self-hosted open models (vLLM, quantisation, serverless GPUs), chosen based on your evals.

## How I approach it

1. **Start on a call with the people who will use it.** I run discovery with your users and domain experts, and most of the criteria come from them.
2. **Write the evals before polishing the prompt.** Define what "good" means, then iterate.
3. **Ship a first version fast,** measure it on simulations, then on real traffic.
4. **Close the loop.** Every failure becomes a test case.

## Related

- [WhatsApp Business Platform engineering](https://fedme.dev/services/whatsapp-business-platform): where many of these agents live.
- [Agentic coding for engineering teams](https://fedme.dev/services/agentic-coding): using agents to build software, not just ship them.

## Ways to engage

- **Evals sprint**: Two to three weeks to define criteria with your domain experts, build a simulation and judging harness, and get a baseline for your agent.
- **Agent build**: Design and ship an agent or multi-agent system end to end, from discovery with your users to production.
- **Architecture and model review**: Review an existing agent for failure modes, cost and latency, and compare models on your own data.

## Stack

Anthropic Claude, OpenAI, Gemini, MedGemma, vLLM & Modal, OpenTelemetry GenAI, LangSmith / Opik / LangWatch, RAG & pgvector, Python, Elixir

## FAQ

### What are simulation evals?

An LLM plays a user with a persona and a goal and has a whole multi-turn conversation with your agent. Then LLM judges score the transcript against criteria your domain experts define. Running hundreds of simulations tells you the error rate before real users see the agent.

### Is LLM-as-judge reliable?

It is reliable when the criteria are specific and binary, the judge sees the evidence it needs, and you calibrate it against human labels on a sample. Vague rubrics give noisy scores. I design criteria with the people who know the domain.

### How do you evaluate agents in production?

By collecting traces of real conversations using the OpenTelemetry GenAI conventions, scoring samples with the same judges used offline, and exporting to tools like LangSmith, Opik or LangWatch. That way offline and online results are comparable.

### Which model should we use?

The one that passes your evals at an acceptable cost and latency. I compare frontier models and open models on your own simulations rather than on public benchmarks. Sometimes a quantised open model on serverless GPUs wins, and sometimes it doesn't.

### Do you work on multi-agent systems?

Yes, including triage agents that hand over to specialists, deterministic rule engines for high-risk decisions, and handover to human operators. I only reach for multiple agents when it makes the system easier to test.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact


---

# Agentic coding adoption for engineering teams (Claude Code rollout, review skills)

> Help rolling out Claude Code and agentic coding to engineering teams. Setup, conventions, custom skills and PR review automation, so engineers get real work out of it.

URL: https://fedme.dev/services/agentic-coding
Provided by: Federico Meini (federico.meini@gmail.com)

## Get your engineers real work out of coding agents, not just autocomplete.

I lead the agentic coding effort at Turn.io. I rolled out Claude Code to the engineering team, manage the licences, help engineers get real work out of it and wrote the Claude-based PR review skills the team uses every day.

## What you get

- A rollout plan that fits your team, covering licences, permissions, security boundaries and a pilot group.
- Repository conventions (CLAUDE.md, commands, hooks) that make agents effective in your codebase.
- Custom skills for your recurring work, such as PR review, migrations, test coverage and release notes.
- Hands-on pairing sessions so engineers learn workflows that actually save time.

## Licences are the easy part

Buying seats for a coding agent takes an afternoon. Getting a team to use it well takes deliberate work: conventions the agent can follow, permissions that keep it safe, skills for the work you repeat, and engineers who know when to delegate and when not to.

At Turn.io I **rolled out Claude Code to the engineering team**, manage the licences and help engineers get real work out of it. I also **wrote the Claude-based PR review skills** the team uses on its pull requests.

## What I do

**Rollout.** Start with a pilot group, choose the tasks where agents shine, measure, then expand. Licences, permissions and security boundaries are set up from day one.

**Make the codebase agent-friendly.** Project instructions, commands for common workflows, hooks that enforce formatting and tests, and MCP servers for your internal tools.

**Custom skills.** Encode your team's recurring work, including PR review against your standards, database migrations, test generation, incident write-ups and release notes.

**Pairing and workshops.** Hands-on sessions in your own repositories, where engineers learn by doing.

## Related

- [AI agent orchestration and evals](https://fedme.dev/services/ai-agents-evals)

## Ways to engage

- **Team rollout**: Four to six weeks of setup, conventions, custom skills and pairing with a pilot group, then the wider team.
- **Review automation**: Build AI PR review skills tuned to your codebase and standards, wired into your Git workflow.
- **Workshop**: A half-day or full-day hands-on workshop for your engineers, using your own repositories.

## Stack

Claude Code, Skills & slash commands, Hooks, MCP servers, GitHub Actions, Elixir, Python, TypeScript

## FAQ

### Will coding agents replace our engineers?

No. In my experience they change what engineers spend time on. There is less typing boilerplate and more specifying, reviewing and verifying. The teams that benefit most are the ones that invest in conventions and review.

### Is it safe to let an agent work in our repositories?

With the right setup, yes. That means scoped permissions, no production credentials in the agent's environment, sandboxing where appropriate, and human review of every change. I help you set those boundaries explicitly.

### What does an AI PR review skill check?

Whatever your human reviewers keep repeating, for example security pitfalls, migrations that lock tables, missing tests, API conventions or style guide rules. Focused, specialised checks beat one generic "review this" prompt.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact


---

# Blog posts

---

---
title: "Testing a medical triage agent with LangWatch Scenario and pytest"
description: "How I test a medical triage AI agent with LangWatch Scenario: a simulated user, judge criteria, scripted and free-running conversations, in pytest and CI."
author: Federico Meini
date: 2026-09-16
tags: [ai-agents, llm-evals, python, testing]
language: en
url: https://fedme.dev/blog/testing-medical-triage-agents-with-langwatch-scenario
---

# Testing a medical triage agent with LangWatch Scenario and pytest

LangWatch Scenario is the easiest way I've found to put multi-turn simulation tests for an AI agent into an ordinary pytest suite. You wrap your agent in a small adapter, describe the situation in plain English, list the criteria a judge should check, and Scenario plays the conversation out with a simulated user until the judge reaches a verdict. At Turn.io I used it with [OneDay Health](https://onedayhealth.org), which runs primary-care clinics in Uganda. Their nurses use an AI agent that walks them through the clinical guidelines while they see a patient, and Scenario let us simulate those conversations before the agent reached a real clinic, and I published a stripped-down version of that first setup as a demo: **[fedme/agent_simulation_tests_example](https://github.com/fedme/agent_simulation_tests_example)**.

In an earlier post I described [simulation evals](https://fedme.dev/blog/simulation-evals-for-llm-chatbots) from first principles: personas, hand-rolled conversation loops, one judge per criterion, error rates. This post is the practical companion. It's about a library that gives you most of that loop for free, and about the code in the demo repo.

## TL;DR

- Scenario has three moving parts: an **`AgentAdapter`** around your agent, a **`UserSimulatorAgent`** that plays the user from a scenario description, and a **`JudgeAgent`** that checks a list of criteria in plain English.
- A triage scenario is a short description: who the user is, what they report up front, and which facts they only reveal **if asked**. That is what tests whether the agent asks the right follow-up questions.
- Give the judge **the same clinical guidelines the agent uses**. Otherwise it grades medicine from its own training data.
- Tests can run **free** (the simulator and judge drive everything) or **scripted** (fixed turns, code assertions, then let it run).
- It's plain pytest, so it runs in CI with a model API key (plus an optional LangWatch key). The demo requires `langwatch-scenario>=0.7.13`; the API used here is the same in the current 1.x releases.

## What's in the demo repo

The repo is deliberately small: a `pyproject.toml` managed with uv, a `.env.example` with `OPENAI_API_KEY` and `LANGWATCH_API_KEY`, the getting-started recipe test from the Scenario docs, and a triage test file with two scenarios. The agent under test is a single LLM call. Its system prompt embeds a primary-care treatment handbook for nurses and clinical officers, and tells the model to ask one or two follow-up questions at a time, follow the handbook's decision trees, and give the diagnosis and treatment exactly as written there, or recommend a hospital visit if nothing matches.

In the snippets below I've trimmed long strings; the names and structure are what's in the repo.

## Wrapping the agent under test

Scenario doesn't care how your agent is built. You subclass `scenario.AgentAdapter` and implement one async method, `call`, which receives a `scenario.AgentInput` and returns the agent's reply:

```python
import litellm
import scenario

scenario.configure(default_model="openai/gpt-4.1", max_turns=10, verbose=True)


@scenario.cache()
def generate_triage_response(messages) -> scenario.AgentReturnTypes:
    response = litellm.completion(
        model="openai/gpt-4.1",
        messages=[
            {"role": "system", "content": triage_system_prompt()},
            *messages,
        ],
    )
    return response.choices[0].message


class OneDayAgent(scenario.AgentAdapter):
    async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
        return generate_triage_response(input.messages)
```

`AgentInput` carries the full conversation as OpenAI-style `messages`, the `new_messages` since the agent last spoke, a `thread_id`, and the scenario state. You can return a string, an OpenAI-style message, or a list of messages. In a real project, `call` is where you hit whatever you actually ship: an HTTP endpoint, a LangGraph graph, a staging deployment. Use `thread_id` as the session id so each simulated conversation gets its own state.

`scenario.configure` sets the model for the simulator and the judge. Calls go through LiteLLM, so any provider works.

## Describing a triage scenario

This is the first scenario from the repo, lightly cleaned up:

```python
description = """
  The user is a nurse currently examining a 4 year old patient with one day of cough and fever.
  Malaria test negative. He's eating well and not vomiting.
  If asked, child's respiratory rate is 65.
  If asked, child has no chest indrawing or stridor.

  As the nurse, you need to describe the patient to the Agent and answer any follow-up
  questions so that the Agent can make a diagnosis.
"""
```

The description is shared: the user simulator uses it to play the part, and the judge reads it as context. The "If asked" lines are the important bit. They are the facts the simulated nurse knows but won't volunteer, which is the goal/facts split from my earlier post expressed as one sentence each. The handbook says every child with a cough must have their breathing rate counted, and above 40 breaths per minute at that age counts as fast breathing. So this scenario only goes well if the agent asks for the respiratory rate instead of guessing from cough and fever.

The simulated user is a nurse seeing a patient in the clinic, because that's who the agent supports. For a patient-facing self-triage flow you change the persona ("a mother messaging about her son, short messages, not sure what counts as fast breathing") and nothing else. The default simulator already writes short, informal messages, which suits WhatsApp. If you need more control over the persona, `UserSimulatorAgent` takes its own `system_prompt` and `model`.

## Judge criteria

Criteria are plain strings, written so that a clinician can read and edit them:

```python
criteria = [
    "Agent provides accurate information closely following the OneDay medical guidelines",
    "Agent doesn't ask for the same information multiple times",
    "Agent asks correct follow-up questions according to the OneDay medical guidelines",
    "Agent provides a diagnosis that strictly follows the OneDay medical guidelines",
]

result = await scenario.run(
    name="4yo with cough and fever",
    description=description,
    agents=[
        OneDayAgent(),
        scenario.UserSimulatorAgent(),
        scenario.JudgeAgent(
            criteria=criteria,
            model="openai/gpt-4.1",
            system_prompt=judge_prompt(description, criteria),
        ),
    ],
)

assert result.success
```

The second scenario in the repo (an adult with five days of fever, night sweats and joint pain) pins the expected outcome directly in a criterion: the agent must reach a specific diagnosis from the handbook. That's how you encode a clinical vignette with a known correct answer.

Under the hood, the judge runs after every turn with two tools: `continue_test` and `finish_test`. When it has enough information it returns a verdict for each criterion (true, false or inconclusive) plus its reasoning. It stops early if a "should not" criterion is already broken, and on the last turn it must decide. The `ScenarioResult` you get back has `success`, `reasoning`, `passed_criteria` and `failed_criteria`, so a failing test tells you which criterion failed and why.

The one change I made to the judge matters a lot in a medical context. By default the judge only sees the scenario and the criteria, so "follows the clinical guidelines" would be graded from the judge model's general medical knowledge. The repo's `judge_prompt` keeps the library's default judge prompt and adds the handbook:

```python
def judge_prompt(scenario_description: str, criteria: list[str]) -> str:
    return f"""
      <role>
      You are an LLM as a judge watching a simulated conversation as it plays out live
      to determine if the agent under test meets the criteria or not.
      </role>

      The agent under test helps nurses make medical diagnosis strictly following
      the OneDay medical guidelines, which are reported below:

      <guidelines>
      {clinical_guidelines()}
      </guidelines>

      <scenario>
      {scenario_description}
      </scenario>

      <criteria>
      {"\n".join(criteria)}
      </criteria>
    """  # + the default <goal> and <rules> sections, omitted here
```

The repo keeps the criteria generic because it was a first demo. For a triage agent going to production I'd add sharper, safety-focused ones: "Agent refers the child to hospital immediately if any danger sign is reported", "Agent asks for the respiratory rate before giving a diagnosis for a child with cough", "Agent does not recommend medicines or doses that are not in the guidelines". For a patient-facing self-triage agent you'd flip the diagnosis criterion to "Agent does not give a diagnosis and recommends the right level of care".

One difference from my earlier post: Scenario's judge scores all criteria in a single call, where I recommended one judge per criterion. For regression tests that's fine. When you need calibrated error rates, you can take `result.messages` and run your own per-criterion judges on the transcripts.

## Free-running and scripted simulations

The two triage tests pass no `script`, so Scenario runs its default: the simulated user opens, the agent replies, the judge decides whether to continue, and so on until a verdict or `max_turns`. That is the mode to use for "does this case go well end to end".

Scripts give you control over specific turns. Each step is `scenario.user()`, `scenario.agent()`, `scenario.judge()`, `scenario.proceed()`, `scenario.succeed()` or `scenario.fail()`, or any plain function that receives the scenario state. `scenario.user("...")` sends a fixed message, `scenario.user()` lets the simulator write one. This variant isn't in the repo, but it uses the same agent and judge:

```python
def follows_reply_format(state: scenario.ScenarioState) -> None:
    reply = state.last_message()["content"] or ""
    assert "<EXPLANATION>" in reply  # the system prompt requires it


judge = scenario.JudgeAgent(criteria=criteria, system_prompt=judge_prompt(description, criteria))

result = await scenario.run(
    name="4yo with cough and fever, terse opening",
    description=description,
    agents=[OneDayAgent(), scenario.UserSimulatorAgent(), judge],
    script=[
        scenario.user("4yo boy cough and fever since yesterday"),
        scenario.agent(),
        follows_reply_format,
        scenario.proceed(turns=5),
        scenario.judge(),
    ],
)
```

The opening is fixed so the case is reproducible, a deterministic rule is checked in code rather than by the judge, and then the simulation runs freely before a forced verdict. If a script ends without a verdict, the run fails and says so, which avoids tests that silently prove nothing.

## Running it in pytest and CI

The tests are ordinary async pytest functions marked with `@pytest.mark.agent_test` and `@pytest.mark.asyncio`. Scenario ships a pytest plugin that registers the marker and prints a summary at the end with each scenario's reasoning and passed criteria, and `pytest-asyncio` comes in as a dependency. Locally:

```bash
uv run pytest -s -m agent_test
```

With `-s` you watch the conversation turn by turn. `LANGWATCH_API_KEY` is optional: without it you only get terminal output, with it every simulated conversation shows up in LangWatch, which is much easier to read with a clinician than terminal logs. `scenario.configure(debug=True)` pauses at each user turn so you can type the message yourself, handy when a test fails and you want to poke at the agent.

The demo repo has no CI workflow, but adding one is a single job. Adapted from the Scenario docs:

```yaml
- name: Run simulation tests
  run: uv run pytest -m agent_test
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    LANGWATCH_API_KEY: ${{ secrets.LANGWATCH_API_KEY }}
    SCENARIO_BATCH_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
```

`SCENARIO_BATCH_RUN_ID` groups all scenarios from one CI run in LangWatch. For speed and repeatability, set a `cache_key` in `scenario.configure`: the agent function (that's what the `@scenario.cache()` decorator in the repo is for), the simulator and the judge are then cached on disk, keyed on their arguments. Without a `cache_key` the decorator does nothing. The key doesn't include your function's code, so bump it whenever you change the prompt or model. And don't measure error rates with caching on: a cached run is one sample replayed, and a triage flow needs many fresh runs.

## Where this fits

Simulation testing matters in medicine because triage failures happen inside a conversation: the fourth message where a danger sign is mentioned in passing, a follow-up question that never gets asked. Single-prompt evals don't see those. And because scenarios and criteria are plain English, the clinicians who own the guidelines can review exactly what's being tested, which I've found is what builds trust in the results.

I think of three layers. Scenario tests are **regression tests**: a curated set of cases that must pass on every change, in CI. **Offline simulation evals** measure error rates: many scenarios, several runs each, calibrated judges, sensitivity for the cases that must be escalated. **Online tracing** covers the conversations you didn't think of: production traces scored with the same criteria, and every confirmed failure becomes a new Scenario test. Scenario makes the first layer cheap enough that there's no excuse to skip it.

The library has grown since I wrote the demo (it now covers red teaming and voice agents too), but the core is unchanged. Clone the [demo repo](https://github.com/fedme/agent_simulation_tests_example), swap in your agent and your guidelines, and you'll have your first multi-turn test running in an afternoon.

If you're building an AI agent for a flow where mistakes reach real people and want tests and evals that show it's ready, [I help teams build exactly that](https://fedme.dev/services/ai-agents-evals).


---

---
title: "Building a reliable WhatsApp Cloud API webhook receiver at scale"
description: "How to build a WhatsApp Cloud API webhook receiver that holds up: signature checks on the raw body, fast 200s, dedup, per-contact ordering and media handling."
author: Federico Meini
date: 2026-09-03
tags: [whatsapp, webhooks, elixir, phoenix]
language: en
url: https://fedme.dev/blog/whatsapp-cloud-api-webhooks-at-scale
---

# Building a reliable WhatsApp Cloud API webhook receiver at scale

A reliable WhatsApp Cloud API webhook receiver does four things in the request path and nothing else: it verifies the `X-Hub-Signature-256` header against the raw request body, persists the payload durably, responds with a 200 quickly, and hands the work to asynchronous processing. Everything that makes WhatsApp integrations hard (deduplicating at-least-once deliveries, keeping each conversation in order, reconciling out-of-order status updates, respecting the 24-hour window, downloading media) happens after the 200, where a slow step can't cause Meta to retry and multiply your load. I've spent six years at Turn.io working on a WhatsApp platform that handles millions of messages a day, and this is how I'd structure the receiver in Elixir and Phoenix.

## TL;DR

- Keep the **raw body** around and verify `X-Hub-Signature-256` (HMAC-SHA256 with your app secret) over those exact bytes.
- In the request: **verify, persist, return 200**. Return a 5xx only if you couldn't persist, so Meta retries.
- Webhooks are **at-least-once**: deduplicate inbound messages by their message id with a unique index.
- Process each contact's messages **in order, one at a time**: one process per contact, or a queue partitioned by contact.
- Status updates arrive **out of order**: only ever move a message's status forward.
- Track the **24-hour customer service window** per contact and switch to templates outside it.
- Download **media** promptly in a background job: the download URL is short-lived.

## What Meta sends you

Webhook payloads are batched. A single POST contains an `entry` array, each entry has `changes`, and each change has a `value` that can hold `messages` (inbound messages from users), `statuses` (updates about messages you sent), `contacts`, and `metadata` with the `phone_number_id` the event belongs to. Don't assume one message per request, and don't assume a request contains only one kind of event.

Before any of that, Meta verifies your endpoint with a GET request carrying `hub.mode=subscribe`, a `hub.verify_token` you chose, and a `hub.challenge` you must echo back. The verify token is only used for this handshake. The app secret is what signs every POST. They are different values, and mixing them up is a classic first-day bug.

## Verifying X-Hub-Signature-256 over the raw body

The signature is an HMAC-SHA256 of the request body using your app secret, sent as `sha256=<hex digest>`. It's computed over the exact bytes Meta sent. By the time a Phoenix controller runs, `Plug.Parsers` has consumed the body and decoded the JSON, and re-encoding it won't reproduce the same bytes. So keep a copy while parsing, with a custom body reader:

```elixir
defmodule MyAppWeb.CacheRawBody do
  @moduledoc "Keeps the raw request body for webhook signature verification."

  @webhook_path "/webhooks/whatsapp"

  def read_body(conn, opts) do
    case Plug.Conn.read_body(conn, opts) do
      {:ok, chunk, conn} -> {:ok, chunk, store(conn, chunk)}
      {:more, chunk, conn} -> {:more, chunk, store(conn, chunk)}
      {:error, _reason} = error -> error
    end
  end

  # Only keep a copy for the webhook route, so other requests don't pay for it.
  defp store(%Plug.Conn{request_path: @webhook_path} = conn, chunk) do
    Plug.Conn.assign(conn, :raw_body, [conn.assigns[:raw_body] || "", chunk])
  end

  defp store(conn, _chunk), do: conn
end
```

Wire it into the parsers in your endpoint:

```elixir
plug Plug.Parsers,
  parsers: [:urlencoded, :multipart, :json],
  pass: ["*/*"],
  body_reader: {MyAppWeb.CacheRawBody, :read_body, []},
  json_decoder: Phoenix.json_library()
```

Then verify in a plug on the webhook pipeline, using a constant-time comparison:

```elixir
defmodule MyAppWeb.Plugs.VerifyWhatsAppSignature do
  import Plug.Conn

  def init(opts), do: opts

  def call(conn, _opts) do
    secret = Application.fetch_env!(:my_app, :whatsapp_app_secret)
    raw_body = IO.iodata_to_binary(conn.assigns[:raw_body] || "")
    digest = :crypto.mac(:hmac, :sha256, secret, raw_body) |> Base.encode16(case: :lower)

    with [signature] <- get_req_header(conn, "x-hub-signature-256"),
         true <- Plug.Crypto.secure_compare("sha256=" <> digest, String.downcase(signature)) do
      conn
    else
      _ -> conn |> send_resp(401, "invalid signature") |> halt()
    end
  end
end
```

If signatures fail in production but pass locally, look for something between Meta and your app that rewrites the body: a proxy that decompresses or re-encodes JSON, or middleware that parses the body before your reader sees it.

## Respond fast, process later

Meta retries deliveries that fail or time out, with decreasing frequency, for up to several days according to its documentation. That's good for durability and bad for load: if your handler is slow because a downstream dependency is slow, requests time out, Meta retries, and the retries arrive while you're still slow. The fix is to make the request path trivially cheap.

With Oban, the job table is a perfectly good inbox. Insert one job per POST (not one per message) and return:

```elixir
defmodule MyAppWeb.WhatsAppWebhookController do
  use MyAppWeb, :controller

  alias MyApp.Workers.ProcessWhatsAppWebhook

  def verify(conn, %{"hub.mode" => "subscribe", "hub.verify_token" => token, "hub.challenge" => challenge}) do
    if Plug.Crypto.secure_compare(token, Application.fetch_env!(:my_app, :whatsapp_verify_token)) do
      send_resp(conn, 200, challenge)
    else
      send_resp(conn, 403, "")
    end
  end

  def create(conn, _params) do
    case Oban.insert(ProcessWhatsAppWebhook.new(%{"payload" => conn.body_params})) do
      {:ok, _job} -> send_resp(conn, 200, "")
      {:error, _reason} -> send_resp(conn, 500, "")
    end
  end
end
```

The 500 is deliberate: if you couldn't persist the event, you want Meta to send it again. Anything that can fail after persisting is your problem to retry, not Meta's.

The job then fans the payload out:

```elixir
defmodule MyApp.Workers.ProcessWhatsAppWebhook do
  use Oban.Worker, queue: :whatsapp_webhooks, max_attempts: 10

  alias MyApp.WhatsApp.{Inbound, Statuses}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"payload" => payload}}) do
    for %{"changes" => changes} <- Map.get(payload, "entry", []),
        %{"value" => value} <- changes do
      phone_number_id = get_in(value, ["metadata", "phone_number_id"])
      Enum.each(Map.get(value, "messages", []), &Inbound.handle(phone_number_id, &1))
      Statuses.apply_batch(Map.get(value, "statuses", []))
    end

    :ok
  end
end
```

## Deduplicate: webhooks are at-least-once

You will receive the same inbound message more than once: after a retry, after a timeout where you did process it but Meta didn't see the 200, occasionally for no visible reason. Every inbound message has an `id` (the `wamid.…` string). Put a unique index on it and let the database arbitrate:

```elixir
def handle(phone_number_id, %{"id" => wamid, "from" => wa_id, "timestamp" => ts} = message) do
  attrs = %{
    wamid: wamid,
    wa_id: wa_id,
    phone_number_id: phone_number_id,
    payload: message,
    sent_at: DateTime.from_unix!(String.to_integer(ts))
  }

  case Repo.insert(InboundMessage.changeset(attrs), on_conflict: :nothing, conflict_target: :wamid) do
    # With on_conflict: :nothing, a duplicate comes back with no primary key.
    {:ok, %InboundMessage{id: nil}} -> :duplicate
    {:ok, inbound} -> ContactServer.dispatch(inbound)
  end
end
```

One subtlety: if the insert succeeds and processing then crashes, a retried job sees a duplicate and skips it, and the message is never handled. Store a `processed_at` column, set it when the conversation logic has finished, and have a periodic job pick up rows that were stored but never processed. "Seen" and "handled" are different states.

If the messages table is partitioned, a unique index on the message id alone isn't possible (unique indexes must include the partition key). A small dedup table keyed by `wamid` solves it; I cover this in [the Postgres partitioning post](https://fedme.dev/blog/partitioning-a-1tb-postgres-table-without-downtime).

## Keep each conversation in order

A user who sends "hi", "I need help" and a photo in quick succession expects the bot to see them in that order. Two concurrent workers processing two messages from the same user will eventually get it wrong, and a chatbot state machine processing two inputs at once is a race condition with a user interface.

In a single node, the simplest correct approach is one process per contact:

```elixir
defmodule MyApp.Conversations.ContactServer do
  use GenServer, restart: :transient

  @idle_timeout :timer.minutes(5)

  def dispatch(%{wa_id: wa_id} = inbound) do
    pid =
      case DynamicSupervisor.start_child(MyApp.ContactSupervisor, {__MODULE__, wa_id}) do
        {:ok, pid} -> pid
        {:error, {:already_started, pid}} -> pid
      end

    GenServer.cast(pid, {:inbound, inbound})
  end

  def start_link(wa_id), do: GenServer.start_link(__MODULE__, wa_id, name: via(wa_id))

  defp via(wa_id), do: {:via, Registry, {MyApp.ContactRegistry, wa_id}}

  @impl true
  def init(wa_id), do: {:ok, %{wa_id: wa_id}, @idle_timeout}

  @impl true
  def handle_cast({:inbound, inbound}, state) do
    :ok = MyApp.Bot.handle_inbound(state.wa_id, inbound)
    MyApp.WhatsApp.Inbound.mark_processed(inbound)
    {:noreply, state, @idle_timeout}
  end

  @impl true
  def handle_info(:timeout, state), do: {:stop, :normal, state}
end
```

Messages for one contact are handled strictly one after another; different contacts run in parallel; idle processes go away after a few minutes. Across a cluster you need every event for a contact to reach the same place: hash the contact id to a partition owned by one node, use a distributed registry, or use a job queue partitioned by contact with a concurrency of one per partition (Oban Pro supports partitioned limits for this kind of thing). Pick one and make it explicit, because "usually the same node" is not ordering.

Arrival order is not guaranteed either. When messages arrive within a second or two of each other, holding them briefly and sorting by the payload `timestamp` helps, but that timestamp has one-second resolution, so ties fall back to arrival order.

## Status webhooks arrive out of order

For messages you send, you'll receive statuses such as `sent`, `delivered`, `read` and `failed`, and not necessarily in that order: `read` can arrive before `delivered`. Treat status as monotonic. Rank the states and only apply an update if it moves the message forward, and apply them in batches, since statuses arrive at a multiple of your send rate. I go through the SQL for this in [the bulk sending post](https://fedme.dev/blog/backpressure-bulk-whatsapp-sends-elixir).

If you set `biz_opaque_callback_data` when sending, it comes back on the status webhooks for that message, which is handy for correlating statuses with your own records without an extra lookup.

## The 24-hour customer service window

When a user messages you, a 24-hour customer service window opens, during which you can reply with free-form messages. Outside the window you can only send approved template messages, and a free-form send fails (error 131047). When the user replies, a new window opens.

Don't discover this from errors. Store the timestamp of each contact's last inbound message (it's already in your inbound table) and check it before sending. Leave a margin: a reply your bot generates 23 hours and 59 minutes in may land after the window closes. Design flows so that anything that might go out later, such as reminders, follow-ups or a human agent answering the next morning, has a template ready.

## Media: download promptly

Inbound media messages don't contain the file. They contain a media id (plus MIME type and a hash). Getting the bytes is two authenticated requests: fetch the media metadata by id, which returns a short-lived URL (Meta documents it as valid for five minutes), then download from that URL with the same bearer token.

```elixir
def download_media(media_id) do
  version = Application.fetch_env!(:my_app, :graph_api_version)
  auth = [auth: {:bearer, Application.fetch_env!(:my_app, :whatsapp_token)}]

  with {:ok, %{status: 200, body: %{"url" => url, "mime_type" => mime}}} <-
         Req.get("https://graph.facebook.com/#{version}/#{media_id}", auth),
       {:ok, %{status: 200, body: bytes}} <- Req.get(url, auth ++ [decode_body: false]) do
    {:ok, bytes, mime}
  end
end
```

Do this in its own background job, soon after the webhook arrives, and copy the file into your own storage. Don't store the URL, it will have expired by the time anyone clicks it. Don't do it in the request path either: a large video download is exactly the slow step that makes webhooks time out. Size limits and virus scanning belong in the same job.

## What breaks at scale

- **Reply spikes after campaigns.** Send a message to a million people and a slice of them reply within minutes. Your webhook traffic is shaped by your outbound traffic. Load test the inbound path at the peak your biggest campaign will create.
- **Slow handlers turning into retry storms.** Any synchronous work in the request path eventually gets slow, and slow becomes timeouts, retries and duplicates. Keep the request path to verify, persist, respond.
- **Database connection pool exhaustion.** Webhook requests, the job queue and the conversation workers all compete for the same pool. Size it and watch queue time in the pool, not just query time.
- **Hot contacts.** A single number stuck in a loop (often another bot) can flood one contact process. Add per-contact rate limits and loop detection.
- **Deploys.** Rolling restarts kill in-flight processing. That's fine if everything is persisted before the 200 and processing is idempotent; it's data loss if it isn't.
- **Logging whole payloads.** They contain phone numbers and message content. In health services in particular, treat them as sensitive data.

## Building on the WhatsApp Business Platform

The webhook receiver is the front door of any WhatsApp integration, and the most important rule for it is also the simplest: verify, persist, respond, and do everything else afterwards. If you're building on the WhatsApp Cloud API, or your current integration struggles under load, [I help teams build on the WhatsApp Business Platform](https://fedme.dev/services/whatsapp-business-platform).


---

---
title: "Backpressure in Elixir: sending millions of WhatsApp messages without falling over"
description: "Designing a bulk WhatsApp send engine in Elixir: demand-driven Broadway pipelines, per-number token buckets, retries with jitter and batched status tracking."
author: Federico Meini
date: 2026-08-27
tags: [elixir, whatsapp, backpressure, broadway, oban]
language: en
url: https://fedme.dev/blog/backpressure-bulk-whatsapp-sends-elixir
---

# Backpressure in Elixir: sending millions of WhatsApp messages without falling over

To send millions of WhatsApp messages without falling over, the system has to pull work at the rate the slowest part can absorb instead of pushing everything at once. In practice that means a demand-driven pipeline that claims recipients from Postgres in small batches, a token bucket per business phone number that paces calls to the Cloud API, throughput errors that slow the whole number down instead of triggering a retry storm, and delivery statuses written back in batches rather than one row at a time. At Turn.io I built the high-throughput bulk send engine used for national public-health campaigns reaching 14.7 million people, covering rate limiting, backpressure, delivery tracking and retries. This post is the design I'd use for that problem today, with Elixir sketches for each piece.

## TL;DR

- Keep **durable state per recipient in Postgres** and treat in-memory pipeline stages as disposable.
- Use **GenStage/Broadway** so the API's pace drives how fast you read from the database, not the other way round.
- Put a **token bucket per business phone number** in front of the API. Throughput is a per-number limit, so the limiter has to be too.
- Classify errors: **throughput errors pause the number**, per-recipient limits reschedule that recipient, permanent errors fail fast, and ambiguous ones (timeouts) are reconciled, not blindly retried.
- Retry with **exponential backoff and full jitter**.
- Status webhooks arrive at several times your send rate and **out of order**: apply them monotonically and in batches.
- Instrument everything with `:telemetry` and export to Prometheus. You can't tune what you can't see.

## The constraints you're designing against

Three limits shape the whole design.

**Cloud API throughput.** Throughput is per business phone number. By default the Cloud API allows up to 80 messages per second per number, and eligible numbers can be upgraded to more (up to 1,000 at the time of writing; check Meta's current documentation for your numbers). Exceed it and you get throughput errors (error code 130429). There is also a per-recipient pair rate limit (131056) if you send too many messages to the same user in a short window, and messaging limits on how many unique users you can reach with business-initiated messages in a rolling 24-hour period, which depend on your account's standing.

**Your own database.** Every send is at least one read and one write. Every sent message then generates status webhooks (`sent`, `delivered`, `read` or `failed`), so status writes run at a multiple of the send rate. It's easy to size everything around the API limit and forget that the database has to keep up with all of this.

**Shared capacity.** Your business number is probably also handling live conversations. A campaign that uses 100% of a number's throughput makes the service unresponsive to people replying to it. Budget headroom on purpose.

## Architecture

```text
campaign_recipients (Postgres, source of truth)
        │  claim N rows (FOR UPDATE SKIP LOCKED), on demand
        ▼
RecipientProducer ──► processors (acquire token ► call Cloud API)
                                    │
                                    ▼
                         batcher: write results in bulk
```

Nothing is pushed. Processors ask for more work when they're free, the producer only claims as many rows as there is demand for, and the token bucket blocks processors when the number is at capacity. If the API slows down, processors take longer, demand drops, and the producer stops reading from the database. That's backpressure: every stage slows down together instead of queues growing somewhere in the middle.

## Oban jobs or in-memory stages?

A natural first design is one Oban job per message. It's durable, retries are built in, and it works up to a point. At millions of messages per campaign, though, you're inserting millions of job rows, updating each one several times as it moves through states, and pruning them afterwards. The jobs table becomes the hottest, most bloated table in your database, and throughput control is spread across queue configuration instead of living in one place.

What works better is splitting responsibilities:

- **Postgres rows are the durable state**: one row per recipient with a status, an attempt count and a `next_attempt_at`.
- **Oban handles orchestration**: starting a campaign, scheduling it, periodically reclaiming rows stuck in `sending` after a crash, marking the campaign complete. A handful of jobs per campaign, not one per message.
- **GenStage/Broadway does the sending**: fast, in memory, disposable. If a node dies, the pipeline restarts and picks up from the table.

## Claiming work without contention

The claim is a single statement. `FOR UPDATE SKIP LOCKED` lets several producers (or nodes) claim concurrently without blocking each other or taking the same row:

```sql
UPDATE campaign_recipients
SET status = 'sending', claimed_at = now()
WHERE id IN (
  SELECT id FROM campaign_recipients
  WHERE campaign_id = $1
    AND status = 'pending'
    AND next_attempt_at <= now()
  ORDER BY next_attempt_at
  LIMIT $2
  FOR UPDATE SKIP LOCKED
)
RETURNING id, phone, template_params, attempt;
```

Back it with a partial index so the claim stays cheap as the table fills with finished rows:

```sql
CREATE INDEX campaign_recipients_pending_index
  ON campaign_recipients (campaign_id, next_attempt_at)
  WHERE status = 'pending';
```

## A demand-driven producer

Broadway takes any GenStage producer. This one only touches the database when downstream stages have asked for messages, and polls when it runs dry:

```elixir
defmodule Bulk.RecipientProducer do
  use GenStage
  @behaviour Broadway.Producer

  alias Broadway.Message

  @max_batch 500
  @poll_interval 1_000

  @impl true
  def init(opts) do
    state = %{campaign_id: Keyword.fetch!(opts, :campaign_id), demand: 0, poll_scheduled?: false}
    {:producer, state}
  end

  @impl true
  def handle_demand(incoming, state), do: claim(%{state | demand: state.demand + incoming})

  @impl true
  def handle_info(:poll, state), do: claim(%{state | poll_scheduled?: false})

  defp claim(%{demand: 0} = state), do: {:noreply, [], state}

  defp claim(state) do
    limit = min(state.demand, @max_batch)
    recipients = Bulk.Recipients.claim(state.campaign_id, limit)

    messages =
      Enum.map(recipients, fn r ->
        %Message{data: r, acknowledger: Broadway.NoopAcknowledger.init()}
      end)

    state = %{state | demand: state.demand - length(messages)}
    {:noreply, messages, schedule_poll(state, length(messages) == limit)}
  end

  # A full batch means more rows are probably waiting: poll again straight away.
  # Otherwise we've caught up (or what's left is in backoff): check again later.
  defp schedule_poll(%{demand: 0} = state, _full?), do: state
  defp schedule_poll(%{poll_scheduled?: true} = state, _full?), do: state

  defp schedule_poll(state, full?) do
    Process.send_after(self(), :poll, if(full?, do: 0, else: @poll_interval))
    %{state | poll_scheduled?: true}
  end
end
```

Acknowledgement is a no-op because the database row, not the message, is the unit of durability. Results are recorded by the batcher.

## A token bucket per phone number

Broadway has a built-in `rate_limiting` option on the producer, and if one pipeline maps to one phone number it's a fine start. In practice several campaigns and the conversational traffic share a number, so I want one limiter per number that everything sending from it goes through. A small GenServer does the job:

```elixir
defmodule Bulk.TokenBucket do
  use GenServer

  def start_link(opts) do
    id = Keyword.fetch!(opts, :phone_number_id)
    GenServer.start_link(__MODULE__, opts, name: via(id))
  end

  @doc "Blocks the caller until this number has capacity for one more send."
  def acquire(phone_number_id) do
    case GenServer.call(via(phone_number_id), :take) do
      :ok ->
        :ok

      {:wait, ms} ->
        Process.sleep(ms)
        acquire(phone_number_id)
    end
  end

  @doc "Stops all sends from this number for `ms`, e.g. after a throughput error."
  def pause(phone_number_id, ms), do: GenServer.cast(via(phone_number_id), {:pause, ms})

  defp via(id), do: {:via, Registry, {Bulk.Registry, {:bucket, id}}}

  @impl true
  def init(opts) do
    rate = Keyword.fetch!(opts, :rate)
    {:ok, %{rate: rate, tokens: rate * 1.0, updated_at: now()}}
  end

  @impl true
  def handle_call(:take, _from, state) do
    now = now()
    state = refill(state, now)

    cond do
      now < state.updated_at -> {:reply, {:wait, state.updated_at - now}, state}
      state.tokens >= 1 -> {:reply, :ok, %{state | tokens: state.tokens - 1}}
      true -> {:reply, {:wait, ceil((1 - state.tokens) * 1000 / state.rate)}, state}
    end
  end

  @impl true
  def handle_cast({:pause, ms}, state) do
    resume_at = max(state.updated_at, now() + ms)
    {:noreply, %{state | tokens: 0.0, updated_at: resume_at}}
  end

  # While paused, updated_at is in the future and no tokens accrue.
  defp refill(state, now) when now <= state.updated_at, do: state

  defp refill(state, now) do
    tokens = min(state.rate * 1.0, state.tokens + (now - state.updated_at) * state.rate / 1000)
    %{state | tokens: tokens, updated_at: now}
  end

  defp now, do: System.monotonic_time(:millisecond)
end
```

Set `rate` below the number's real limit to leave room for live conversations. Waiting happens in the caller, so the bucket process never blocks. One GenServer call per send is nothing at these rates.

Two things to get right around it. First, the bucket must have a **single owner per number** across the cluster. `Registry` is node-local, so either run each number's sending on one node (a global registry, or routing numbers to nodes) or move the limiter somewhere shared. Two nodes with their own buckets at 80 per second each will cheerfully send 160. Second, **concurrency has to cover latency**. By Little's law, in-flight requests equal rate times latency: to sustain, say, 80 messages per second with 250 ms API latency you need at least 20 concurrent processors, plus margin for slow responses.

## The pipeline

```elixir
defmodule Bulk.Pipeline do
  use Broadway

  alias Broadway.Message

  def start_link(campaign) do
    Broadway.start_link(__MODULE__,
      name: {:via, Registry, {Bulk.Registry, {:pipeline, campaign.id}}},
      producer: [module: {Bulk.RecipientProducer, campaign_id: campaign.id}, concurrency: 1],
      processors: [default: [concurrency: 40, max_demand: 5]],
      batchers: [default: [concurrency: 1, batch_size: 500, batch_timeout: 1_000]],
      context: %{phone_number_id: campaign.phone_number_id}
    )
  end

  @impl true
  def process_name({:via, Registry, {registry, key}}, base_name) do
    {:via, Registry, {registry, {key, base_name}}}
  end

  @impl true
  def handle_message(_processor, %Message{data: recipient} = message, ctx) do
    :ok = Bulk.TokenBucket.acquire(ctx.phone_number_id)

    result =
      :telemetry.span([:bulk, :send], %{phone_number_id: ctx.phone_number_id}, fn ->
        result = WhatsApp.send_template(ctx.phone_number_id, recipient)
        {result, %{phone_number_id: ctx.phone_number_id, ok?: match?({:ok, _}, result)}}
      end)

    Message.put_data(message, {recipient, classify(result, recipient, ctx)})
  end

  @impl true
  def handle_batch(:default, messages, _batch_info, _ctx) do
    messages |> Enum.map(& &1.data) |> Bulk.Recipients.record_results()
    messages
  end
end
```

Small `max_demand` on processors matters. With large demand, each processor buffers many messages it can't send yet because it's waiting on the bucket, and those rows sit in `sending` doing nothing.

## Errors, retries and backoff with jitter

Not all errors mean the same thing, and treating them the same is how retry storms start:

```elixir
# Accepted by the API: record the WhatsApp message id.
defp classify({:ok, wamid}, _recipient, _ctx), do: {:sent, wamid}

# Throughput reached for this number: slow the whole number down, retry later.
defp classify({:error, %{code: 130429}}, r, ctx) do
  Bulk.TokenBucket.pause(ctx.phone_number_id, 1_000)
  {:retry, Bulk.Backoff.delay_ms(r.attempt)}
end

# Too many messages to this one user: reschedule only this recipient.
defp classify({:error, %{code: 131056}}, r, _ctx), do: {:retry, Bulk.Backoff.delay_ms(r.attempt)}

# We don't know whether the message went out: reconcile, don't resend blindly.
defp classify({:error, :timeout}, _r, _ctx), do: :unknown

# Everything else is treated as permanent for this recipient.
defp classify({:error, error}, _r, _ctx), do: {:failed, error}
```

And the backoff, using "full jitter": a random delay between zero and an exponentially growing ceiling. Without jitter, every recipient that failed in the same second retries in the same second, and you hit the limit again in lockstep.

```elixir
defmodule Bulk.Backoff do
  @base_ms 1_000
  @cap_ms 5 * 60_000

  def delay_ms(attempt) do
    :rand.uniform(min(@cap_ms, @base_ms * Integer.pow(2, attempt)))
  end
end
```

Retries go back through the table: set the row to `pending`, bump `attempt`, set `next_attempt_at`. After a maximum number of attempts, mark it failed. Because retries are rows and not sleeping processes, a retry backlog costs nothing in memory and survives deploys.

## Idempotency and the "did it send?" problem

The dangerous case is a request that times out. Meta may have accepted it, or not. Assume the API won't deduplicate a resend for you, so you have to decide which is worse for this campaign: a duplicate message or a missed one. For health reminders, a duplicate is usually the lesser evil. For anything that looks like a payment prompt, it isn't.

Either way, make it reconcilable. Mark the row `sending` before calling the API, and pass your recipient id in `biz_opaque_callback_data` on the send request. The Cloud API echoes that field back in status webhooks, so even if you never stored the WhatsApp message id, a later `sent` or `delivered` status tells you the message did go out. A periodic Oban job then resolves rows stuck in `sending` or `unknown`: anything with a matching status webhook is sent, anything still silent after a generous window gets the campaign's chosen policy.

## Tracking delivery status in batches

Status webhooks arrive at a multiple of your send rate, and not necessarily in order: a `read` can arrive before its `delivered`. Two rules: never move a message backwards, and never write one row per webhook.

Rank the statuses and only apply an update that moves forward:

```sql
CREATE FUNCTION message_status_rank(text) RETURNS int
LANGUAGE sql IMMUTABLE AS $$
  SELECT CASE $1
    WHEN 'sending' THEN 0 WHEN 'sent' THEN 1 WHEN 'failed' THEN 2
    WHEN 'delivered' THEN 3 WHEN 'read' THEN 4
  END
$$;

UPDATE campaign_recipients r
SET status = s.status, status_at = s.at
FROM unnest($1::text[], $2::text[], $3::timestamptz[]) AS s(wamid, status, at)
WHERE r.wamid = s.wamid
  AND message_status_rank(s.status) > message_status_rank(r.status);
```

Buffer incoming statuses for up to a second or a few hundred entries, reduce them in Elixir to the highest-ranked status per message id (if the same id appears twice in one `UPDATE ... FROM`, Postgres applies only one of the matching rows, and you don't get to choose which), then run one statement for the whole batch. The webhook receiver itself should do nothing but verify, persist and acknowledge; I've written up [how I build that part](https://fedme.dev/blog/whatsapp-cloud-api-webhooks-at-scale).

## Observability

Broadway emits `:telemetry` events for its stages, and the `:telemetry.span/3` around the API call gives you start/stop/exception events with durations. Export them with `telemetry_metrics_prometheus` (or PromEx) and build a dashboard with at least:

- sends per second per phone number, against the configured rate
- time spent waiting in `TokenBucket.acquire` (if it's always high, you're bucket-bound; if it's zero and throughput is low, the bottleneck is elsewhere)
- API latency percentiles and error counts by error code
- claim query duration and number of `pending` rows per campaign
- time from `sent` to `delivered`, which tells you about the recipients' side, not yours
- rows stuck in `sending` for longer than a few minutes

Alert on throughput errors, not just on failures. A steady trickle of 130429s means your configured rate is higher than what Meta is actually giving you.

## Mistakes worth avoiding

- **Reading the whole audience into memory** at campaign start. Claim in small batches.
- **Retrying inside the processor with `Process.sleep`.** It holds a processor and a row hostage. Put retries back in the table.
- **Letting several nodes share a number without coordinating.** Per-number limits need a per-number owner.
- **Forgetting the conversational traffic.** When a million people receive a message, some of them reply. Your inbound path needs capacity at exactly the moment the campaign peaks.

## Need a bulk sender that holds up?

A bulk messaging engine is a rate-limited, stateful, distributed system disguised as a loop over a list. If you're building one for WhatsApp, or your current one struggles at campaign peaks, [I help teams design and build real-time systems like this](https://fedme.dev/services/realtime-systems).


---

---
title: "How to partition a 1 TB PostgreSQL table without downtime: the default-partition method"
description: "Partition a live 1 TB Postgres table without downtime: attach it as the DEFAULT partition, copy history out with Oban jobs, then swap in one short transaction."
author: Federico Meini
date: 2026-08-18
tags: [postgresql, partitioning, performance, elixir, oban]
language: en
url: https://fedme.dev/blog/partitioning-a-1tb-postgres-table-without-downtime
---

# How to partition a 1 TB PostgreSQL table without downtime: the default-partition method

To partition a huge, live PostgreSQL table without downtime, make the existing table the DEFAULT partition of a new partitioned parent in one short, catalogue-only transaction, send new writes to real time-range partitions straight away, and move the history out of the default in the background. At Turn.io I partitioned production tables of more than 1 TB this way to get performance back, as message data grew by millions of rows a day: the old table became the default partition, and Oban jobs moved its data into dedicated partitions gradually.

## What I did at Turn.io

The migration I ran at Turn.io was very similar to the approach in this post:

- **The old table became the default partition,** and new data went straight into proper time-range partitions.
- **No pg_partman.** Creating partitions ahead of time and moving the history were plain Oban jobs in our Elixir codebase.
- **Oban jobs copied the history into new tables in the background,** and each finished table was attached as a partition in one atomic operation, so readers never saw a gap or a duplicate.
- **CHECK constraints did most of the heavy lifting.** I added a lot of them `NOT VALID` and validated them afterwards with `VALIDATE CONSTRAINT`, which only takes a `SHARE UPDATE EXCLUSIVE` lock. That way Postgres could trust the ranges on attach instead of scanning terabytes under a blocking lock.
- **Foreign keys were dropped for the duration of the move.** Partitioning forces foreign keys to include the partition key, and they get in the way of attaching and detaching. We already had application-level checks and triggers enforcing the same relationships, so removing the constraints for the migration was a safe trade.

Below is how I'd run it today, rebuilt on PostgreSQL 18.6 with a 4-million-row `messages` table under constant pgbench load. Timings are from that laptop test. For keys, pruning, indexes and retention, see [PostgreSQL partitioning in practice](https://fedme.dev/blog/postgresql-partitioning-in-practice).

## TL;DR

- **Prep online:** unique index on `(id, inserted_at)`, matching indexes, no incoming foreign keys, and a validated `CHECK (inserted_at < boundary)`.
- **Cutover:** one transaction attaches the old table `AS DEFAULT`. Milliseconds.
- **New writes:** the CHECK lets Postgres skip scanning the default when you add partitions.
- **Backfill:** moving one range at a time out of a big default scans the whole default under `ACCESS EXCLUSIVE`. Instead, copy history into unattached partitions with Oban, sync them with a trigger, and swap once (about 50 ms of locking).

## Step 1: prepare the old table while it's live

**The partition key must be in every primary key and unique constraint**, so build the future primary key's index first:

```elixir
defmodule MyApp.Repo.Migrations.AddMessagesIdInsertedAtIndex do
  use Ecto.Migration

  @disable_ddl_transaction true
  @disable_migration_lock true

  def change do
    create unique_index(:messages, [:id, :inserted_at], concurrently: true)
  end
end
```

**Match the indexes.** On attach, Postgres adopts an equivalent index for each parent index and builds missing ones under the lock, so every parent index must already exist on the old table.

**Add the boundary CHECK**, at a month start a few days ahead. Without it, every partition created while the old table is the default scans all of it. `NOT VALID` makes adding it instant; validating takes `SHARE UPDATE EXCLUSIVE`, which doesn't block reads or writes (617 ms on 700 MB):

```elixir
defmodule MyApp.Repo.Migrations.AddMessagesLegacyRangeCheck do
  use Ecto.Migration

  def change do
    create constraint(:messages, :messages_legacy_before_2026_10,
             check: "inserted_at < '2026-10-01 00:00:00+00'",
             validate: false
           )
  end
end
```

Then, in a separate migration: `execute "ALTER TABLE messages VALIDATE CONSTRAINT messages_legacy_before_2026_10"`.

**Incoming foreign keys.** At Turn.io we dropped foreign keys for the duration of the migration, because application-level checks and triggers already enforced the same relationships. If you can't make that trade, plan carefully. Postgres will attach a referenced table, but its foreign keys keep pointing at that one partition and block dropping the old primary key. Drop them and enforce the relationship in code, or add the timestamp to the referencing table for a composite foreign key to the new parent (PostgreSQL 12+).

**Sequences and identity.** `LIKE ... INCLUDING DEFAULTS` copies Ecto's `nextval('messages_id_seq')` default; move the sequence's ownership, or dropping the old table drops it. PostgreSQL 18 won't attach a table with its own identity column: drop it on the old table during the cutover and `RESTART` the parent's above `max(id)`, as [pg_partman's guide](https://github.com/pgpartman/pg_partman/blob/master/doc/pg_partman_howto.md) shows.

## Step 2: the cutover transaction

```sql
BEGIN;
SET LOCAL lock_timeout = '2s';
LOCK TABLE messages IN ACCESS EXCLUSIVE MODE;

ALTER TABLE messages RENAME TO messages_legacy;
ALTER TABLE messages_legacy
  DROP CONSTRAINT messages_pkey,
  ADD CONSTRAINT messages_legacy_pkey PRIMARY KEY USING INDEX messages_id_inserted_at_index;
ALTER INDEX messages_contact_id_inserted_at_index RENAME TO messages_legacy_contact_id_inserted_at_index;
ALTER INDEX messages_inserted_at_index RENAME TO messages_legacy_inserted_at_index;
ALTER TABLE messages_legacy RENAME CONSTRAINT messages_contact_id_fkey TO messages_legacy_contact_id_fkey;

CREATE TABLE messages (LIKE messages_legacy INCLUDING DEFAULTS) PARTITION BY RANGE (inserted_at);
ALTER TABLE messages ADD CONSTRAINT messages_pkey PRIMARY KEY (id, inserted_at);
CREATE INDEX messages_contact_id_inserted_at_index ON messages (contact_id, inserted_at);
CREATE INDEX messages_inserted_at_index ON messages (inserted_at);
ALTER TABLE messages ADD CONSTRAINT messages_contact_id_fkey
  FOREIGN KEY (contact_id) REFERENCES contacts (id);

ALTER TABLE messages ATTACH PARTITION messages_legacy DEFAULT;
ALTER SEQUENCE messages_id_seq OWNED BY messages.id;

CREATE TABLE messages_p2026_10 PARTITION OF messages
  FOR VALUES FROM ('2026-10-01 00:00:00+00') TO ('2026-11-01 00:00:00+00');
COMMIT;
```

Skip `INCLUDING CONSTRAINTS`, or the boundary CHECK lands on the parent. (I use `timestamptz`; with Ecto's default `timestamp` columns, drop the `+00` from bounds.) The primary key swap is required: otherwise the attach fails with "multiple primary keys for table "messages_legacy" are not allowed".

**Why attaching as DEFAULT is cheap.** A default partition holds whatever no other partition takes. With no other partitions there's nothing to validate: 1.1 ms. Creating `messages_p2026_10` would normally scan the default, but the CHECK rules October out; with `client_min_messages = debug1` Postgres logs "updated partition constraint for default partition "messages_legacy" is implied by existing constraints".

**Locks.** Held until `COMMIT`: `ACCESS EXCLUSIVE` on the old table, the new parent and partition, and, to my surprise, on `contacts`: on PostgreSQL 18, attaching a table with its own foreign key locks the referenced table too. The Ecto migration took 70 to 140 ms, and pgbench clients using prepared statements, like Postgrex, saw no failures. Keep `lock_timeout`: behind a long query your `LOCK TABLE` waits, and every new query waits behind you.

## Step 3: route new writes to real partitions

From the boundary on, rows land in real partitions. Keep a few months ahead, never for a range still in the default:

```elixir
defmodule MyApp.Workers.CreateMessagePartitions do
  use Oban.Worker, queue: :maintenance, max_attempts: 5

  alias MyApp.Repo

  @impl Oban.Worker
  def perform(%Oban.Job{}) do
    # From next month on: earlier runs created the current one.
    next_month(Date.utc_today())
    |> Stream.iterate(&next_month/1)
    |> Enum.take(3)
    |> Enum.each(&ensure_partition/1)
  end

  defp ensure_partition(from) do
    name = "messages_p" <> Calendar.strftime(from, "%Y_%m")

    {:ok, _} =
      Repo.transaction(fn ->
        Repo.query!("SET LOCAL lock_timeout = '2s'")

        if Repo.query!("SELECT to_regclass($1)", [name]).rows == [[nil]] do
          Repo.query!("CREATE TABLE #{name} (LIKE messages INCLUDING DEFAULTS)")

          Repo.query!("""
          ALTER TABLE messages ATTACH PARTITION #{name}
            FOR VALUES FROM ('#{from} 00:00:00+00') TO ('#{next_month(from)} 00:00:00+00')
          """)
        end
      end)
  end

  defp next_month(date), do: date |> Date.end_of_month() |> Date.add(1)
end
```

Run it daily with `Oban.Plugins.Cron`, or use pg_partman's `run_maintenance()`. `ATTACH` needs only `SHARE UPDATE EXCLUSIVE` on the parent, unlike `CREATE TABLE ... PARTITION OF` ([docs](https://www.postgresql.org/docs/current/sql-createtable.html)), but still locks a default partition exclusively, scan or not. With the CHECK, a new partition took 2 ms; without it, 700 ms, growing with the default.

## Step 4: move the history out of the default

### What pg_partman does

pg_partman's [online partitioning guide](https://github.com/pgpartman/pg_partman/blob/master/doc/pg_partman_howto.md) uses this same setup, then `partition_data_proc()`. Each batch is one transaction: `DELETE ... RETURNING` a whole child interval from the default into a temporary table, create and `ATTACH` the child, insert the rows back. Batches can't be smaller than the partition interval ("Custom intervals are not allowed when moving data out of the DEFAULT partition"), since the child can't exist while its rows are in the default. It commits per batch, can pause between batches (`p_wait`), and can lock rows first (`p_lock_wait`).

With pg_partman 5.5.0, moving one month (248k rows) took 7 seconds, and readers touching the default were blocked for about 2.5 of them: the attach locks the default, scans it all, and holds the lock through the insert. With concurrent updates to old rows, my first try deadlocked. [Crunchy Data](https://www.crunchydata.com/blog/postgres-partitioning-with-a-default-partition) and pg_partman's docs both say to keep the default small. On a terabyte, that scan repeats every month.

### Why tightening a CHECK doesn't help

The tempting fix: copy a month into a detached table, then delete it from the default and attach the copy in one short transaction, oldest first, with a CHECK on the default ruling the range out. But the attach needs proof: a scan, or a *validated* CHECK (`NOT VALID` doesn't count; I checked), which can only be validated once the rows are gone. In the same transaction that's a full scan under the lock; in a separate one, the rows are briefly visible nowhere.

### Copy out, capture changes, swap once

So never ask Postgres to prove anything about the big heap. Oban jobs copy history into unattached partitions, a trigger keeps them in sync, and one transaction detaches the old default and attaches everything. Each partition's CHECK skips its validation, and with no default there's no default scan. It's the trigger-plus-background-copy idea of [GitLab's partitioning helpers](https://docs.gitlab.com/development/database/partitioning/date_range/), applied only to the history.

```sql
CREATE TABLE messages_backfill (LIKE messages INCLUDING DEFAULTS) PARTITION BY RANGE (inserted_at);
ALTER TABLE messages_backfill ADD PRIMARY KEY (id, inserted_at);

-- One per month; the oldest uses FROM (MINVALUE) and CHECK (inserted_at < ...).
CREATE TABLE messages_p2026_03 (LIKE messages INCLUDING DEFAULTS);
ALTER TABLE messages_p2026_03 ADD CONSTRAINT messages_p2026_03_bounds
  CHECK (inserted_at >= '2026-03-01 00:00:00+00' AND inserted_at < '2026-04-01 00:00:00+00');
ALTER TABLE messages_backfill ATTACH PARTITION messages_p2026_03
  FOR VALUES FROM ('2026-03-01 00:00:00+00') TO ('2026-04-01 00:00:00+00');

CREATE TABLE messages_backfill_changes (
  seq bigserial PRIMARY KEY, id bigint NOT NULL, inserted_at timestamptz NOT NULL
);

CREATE FUNCTION messages_backfill_capture() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
  IF TG_OP IN ('UPDATE', 'DELETE') THEN
    INSERT INTO messages_backfill_changes (id, inserted_at) VALUES (OLD.id, OLD.inserted_at);
  END IF;
  IF TG_OP IN ('INSERT', 'UPDATE') THEN
    INSERT INTO messages_backfill_changes (id, inserted_at) VALUES (NEW.id, NEW.inserted_at);
  END IF;
  RETURN NULL;
END $$;

CREATE TRIGGER messages_backfill_capture
  AFTER INSERT OR UPDATE OR DELETE ON messages_legacy
  FOR EACH ROW EXECUTE FUNCTION messages_backfill_capture();

CREATE TABLE messages_backfill_progress (
  id int PRIMARY KEY DEFAULT 1 CHECK (id = 1),
  last_id bigint NOT NULL DEFAULT 0,
  rows_copied bigint NOT NULL DEFAULT 0,
  changes_applied bigint NOT NULL DEFAULT 0,
  target_max_id bigint
);
INSERT INTO messages_backfill_progress (target_max_id) SELECT max(id) FROM messages_legacy;
```

Create the trigger before the first copy. Two functions do the work:

```sql
CREATE FUNCTION messages_backfill_copy_batch(batch_size int) RETURNS int
LANGUAGE sql AS $$
  WITH batch AS (
    SELECT id, contact_id, direction, status, body, inserted_at, updated_at
    FROM messages_legacy
    WHERE id > (SELECT last_id FROM messages_backfill_progress)
    ORDER BY id
    LIMIT batch_size
  ), copied AS (
    INSERT INTO messages_backfill (id, contact_id, direction, status, body, inserted_at, updated_at)
    SELECT * FROM batch
    ON CONFLICT (id, inserted_at) DO NOTHING
  )
  UPDATE messages_backfill_progress
  SET last_id = coalesce((SELECT max(id) FROM batch), last_id),
      rows_copied = rows_copied + (SELECT count(*) FROM batch)
  RETURNING (SELECT count(*) FROM batch)::int;
$$;

CREATE FUNCTION messages_backfill_apply_changes(batch_size int) RETURNS int
LANGUAGE sql
-- The planner can't estimate CTE sizes and may hash-join against a full scan.
SET enable_hashjoin = off
SET enable_mergejoin = off
AS $$
  WITH claimed AS (
    DELETE FROM messages_backfill_changes
    WHERE seq IN (SELECT seq FROM messages_backfill_changes ORDER BY seq LIMIT batch_size)
    RETURNING id, inserted_at
  ), keys AS (
    SELECT DISTINCT id, inserted_at FROM claimed
  ), removed AS (
    DELETE FROM messages_backfill b USING keys k
    WHERE b.id = k.id AND b.inserted_at = k.inserted_at
      AND NOT EXISTS (SELECT 1 FROM messages_legacy m
                      WHERE m.id = k.id AND m.inserted_at = k.inserted_at)
  ), upserted AS (
    INSERT INTO messages_backfill (id, contact_id, direction, status, body, inserted_at, updated_at)
    SELECT m.id, m.contact_id, m.direction, m.status, m.body, m.inserted_at, m.updated_at
    FROM messages_legacy m JOIN keys k ON m.id = k.id AND m.inserted_at = k.inserted_at
    ON CONFLICT (id, inserted_at) DO UPDATE
    SET contact_id = EXCLUDED.contact_id, direction = EXCLUDED.direction,
        status = EXCLUDED.status, body = EXCLUDED.body, updated_at = EXCLUDED.updated_at
  )
  UPDATE messages_backfill_progress
  SET changes_applied = changes_applied + (SELECT count(*) FROM claimed)
  RETURNING (SELECT count(*) FROM claimed)::int;
$$;
```

Both are idempotent: the copy advances its cursor in the same transaction and skips existing rows; the replay copies each changed row as it is *now*, or removes it. Without the `SET` lines, replaying 10,000 changes took 2 to 4 seconds (the plan scanned every staging partition); with them, 0.3 seconds, whatever the table size.

The worker runs one batch per execution and snoozes until the swap drops its tables:

```elixir
defmodule MyApp.Workers.MessagesBackfill do
  use Oban.Worker,
    queue: :partition_backfill,
    max_attempts: 20,
    unique: [period: :infinity, states: :incomplete]

  require Logger
  alias MyApp.Repo

  # Shared with the swap. Any constant nothing else uses.
  @lock_key 820_417

  @impl Oban.Worker
  def perform(%Oban.Job{}) do
    if staging_exists?() do
      config = Application.get_env(:my_app, __MODULE__, [])
      batch_size = Keyword.get(config, :batch_size, 5_000)

      case Repo.transaction(fn -> run_batch(batch_size) end, timeout: :timer.seconds(90)) do
        {:ok, :busy} -> {:snooze, 10}
        # Copy done, nothing to replay: keep the backlog small until the swap.
        {:ok, {0, 0}} -> {:snooze, 5}
        {:ok, {copied, applied}} -> report_progress(copied, applied)
      end
    else
      :ok
    end
  end

  defp run_batch(batch_size) do
    %{rows: [[locked?]]} = Repo.query!("SELECT pg_try_advisory_xact_lock($1)", [@lock_key])

    if locked? do
      Repo.query!("SET LOCAL lock_timeout = '2s'")
      Repo.query!("SET LOCAL statement_timeout = '60s'")
      %{rows: [[applied]]} = Repo.query!("SELECT messages_backfill_apply_changes($1)", [batch_size])
      %{rows: [[copied]]} = Repo.query!("SELECT messages_backfill_copy_batch($1)", [batch_size])
      {copied, applied}
    else
      :busy
    end
  end

  defp staging_exists? do
    Repo.query!("SELECT to_regclass('messages_backfill_changes') IS NOT NULL").rows == [[true]]
  end

  defp report_progress(copied, applied) do
    %{rows: [[last_id, target]]} =
      Repo.query!("SELECT last_id, target_max_id FROM messages_backfill_progress")

    Logger.info("messages backfill: id #{last_id}/#{target}, +#{copied}, #{applied} replayed")
    {:snooze, Keyword.get(Application.get_env(:my_app, __MODULE__, []), :pause_seconds, 1)}
  end
end
```

`unique` with `states: :incomplete` makes a second insert a no-op. Open-source Oban's queue limits are per node, so the advisory lock is what stops batches overlapping across nodes. `{:snooze, n}` re-runs the same job, and in Oban 2.24 snoozing doesn't consume attempts. Batch size and pause live in config, to slow down when replicas lag.

After the copy, give each staging partition the parent's other indexes with `CREATE INDEX CONCURRENTLY` (in a migration with both flags above, since the worker still writes), plus its foreign key `NOT VALID`, then validate and `ANALYZE`. Anything missing would be built under the swap's lock.

### The swap

```sql
BEGIN;
SET LOCAL lock_timeout = '2s';
DO $$ BEGIN
  IF (SELECT count(*) FROM messages_backfill_changes) > 20000 THEN
    RAISE EXCEPTION 'change backlog too large, let the backfill worker catch up';
  END IF;
END $$;
SELECT pg_advisory_xact_lock(820417);

-- Catch up while everyone can still read and write.
DO $$ BEGIN
  WHILE messages_backfill_apply_changes(10000) > 500 LOOP END LOOP;
END $$;

LOCK TABLE ONLY messages IN ACCESS EXCLUSIVE MODE;
LOCK TABLE messages_legacy IN ACCESS EXCLUSIVE MODE;

-- Replay the last few changes, now that nobody can write.
DO $$ BEGIN
  WHILE messages_backfill_apply_changes(10000) > 0 LOOP END LOOP;
END $$;

ALTER TABLE messages DETACH PARTITION messages_legacy;

DO $$
DECLARE parts record;
BEGIN
  FOR parts IN
    SELECT c.relname AS name, pg_get_expr(c.relpartbound, c.oid) AS bound
    FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid
    WHERE i.inhparent = 'messages_backfill'::regclass
  LOOP
    EXECUTE format('ALTER TABLE messages_backfill DETACH PARTITION %I', parts.name);
    EXECUTE format('ALTER TABLE messages ATTACH PARTITION %I %s', parts.name, parts.bound);
  END LOOP;
END $$;

DROP TRIGGER messages_backfill_capture ON messages_legacy;
DROP TABLE messages_backfill, messages_backfill_changes, messages_backfill_progress;
DROP FUNCTION messages_backfill_copy_batch, messages_backfill_apply_changes, messages_backfill_capture;
COMMIT;
```

What I measured under load:

- `count(*)` through the parent was identical in all 143 samples across copy, index builds and swap.
- After 246,000 concurrent updates, deletes and back-dated inserts during the copy, `EXCEPT` both ways between the detached table and the new partitions returned nothing.
- The locked part of the swap took about 50 ms, with no failed transactions.
- A first attempt blocked everything for 4.6 seconds: 16,000 changes had piled up and were replayed under the lock. Hence the guard and catch-up pass.

### Trade-offs

- **Locks:** only the swap takes strong ones, briefly.
- **WAL and replication:** the whole history is written again, heap and indexes; throttle and watch replica lag. pg_partman also writes every row a second time.
- **Disk:** room for a second copy until the old table is dropped.
- **Bloat and VACUUM:** nothing is deleted from the default, so nothing to vacuum; it goes in one `DROP TABLE`.
- **Rows changing mid-move:** the trigger and replay handle it, at a small cost to writes on old rows.
- **All at once:** history goes live in one swap. Keep the detached table until you've verified the data.

pg_partman's route is fine when the default is small or you can afford the lock in a quiet window.

## Step 5: clean up and keep the default empty

Verify against the detached table, then drop it; the sequence survives because `messages.id` owns it. Drop the `_bounds` CHECKs, as the [docs recommend](https://www.postgresql.org/docs/current/ddl-partitioning.html), and run `ANALYZE messages` yourself: autovacuum never analyzes a partitioned parent.

I prefer no default afterwards: unroutable rows fail loudly and `DETACH PARTITION ... CONCURRENTLY` works (it refuses to while a default exists). To never reject an insert, keep a small `messages_default` and alert when it has rows. Either way, alert when fewer than two future partitions exist:

```sql
SELECT EXISTS (SELECT 1 FROM messages_default) AS rows_in_default;

SELECT count(*) AS future_partitions
FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid
WHERE i.inhparent = 'messages'::regclass
  AND c.relname > 'messages_p' || to_char(now() AT TIME ZONE 'UTC', 'YYYY_MM');
```

## The alternative: attach the old table as one range partition

If history mainly needs keeping until retention removes it, skip the backfill: with the same validated CHECK and `inserted_at NOT NULL`, attach the old table as one range partition (2 ms, no scan):

```sql
ALTER TABLE messages ATTACH PARTITION messages_legacy
  FOR VALUES FROM (MINVALUE) TO ('2026-10-01 00:00:00+00');
```

No copy, WAL or trigger, but one huge partition whose indexes and vacuum work stay as big as today until you drop it. I'd pick it when old data is rarely queried, and the backfill when queries on history must get faster too.

## If you're facing this migration

Rehearse all of it on a copy of production first. If your Postgres (or Elasticsearch) cluster is slowing down as data grows and you'd like someone who has done this in production to plan and run it with your team, [I help teams with exactly this](https://fedme.dev/services/postgres-elasticsearch).


---

---
title: "PostgreSQL partitioning in practice: partition keys, pruning, indexes and retention"
description: "How PostgreSQL partitioning behaves in practice: choosing the key and size, checking pruning with EXPLAIN, indexes, keys, retention and the Ecto gotchas."
author: Federico Meini
date: 2026-08-12
tags: [postgresql, partitioning, performance, elixir]
language: en
url: https://fedme.dev/blog/postgresql-partitioning-in-practice
---

# PostgreSQL partitioning in practice: partition keys, pruning, indexes and retention

PostgreSQL partitioning pays off when almost every query and every maintenance job lines up with one column, usually a timestamp. Queries then touch a few partitions, the indexes for recent data stay small, vacuum works on manageable pieces, and retention becomes dropping a table. It does nothing for a well-indexed lookup, and it makes every query that ignores the key slower. At Turn.io I partitioned production tables of more than 1 TB to get performance back as message data grew; this post collects the fundamentals I check before and after doing that.

Everything below was verified on PostgreSQL 18.6 against a 4-million-row `messages` table split into monthly range partitions, with version notes where behaviour differs. To move an existing large table without downtime, see [How to partition a 1 TB PostgreSQL table without downtime](https://fedme.dev/blog/partitioning-a-1tb-postgres-table-without-downtime).

## TL;DR

- Partition by the column nearly every query filters on, and never update it.
- Planning cost grows with the partitions a query *can't* prune, so keep the count modest unless your queries prune well.
- Check pruning with `EXPLAIN`: pruned partitions are missing from the plan, or counted in `Subplans Removed`.
- Keep the key a bare column compared with a parameter. Functions on it, and Ecto's `ago/2` on `timestamptz`, defeat plan-time pruning.
- Build indexes with `CREATE INDEX ON ONLY`, `CONCURRENTLY` per partition, then `ALTER INDEX ... ATTACH PARTITION`.
- Primary keys, unique constraints and referencing foreign keys must include the key.
- Retire data with `DETACH PARTITION ... CONCURRENTLY` (PostgreSQL 14+), which needs no default partition.
- Autovacuum never analyzes the parent. Run `ANALYZE` yourself.

## When partitioning helps, and when it doesn't

It helps with time-ordered, append-heavy data where most reads hit recent rows: the recent partitions' indexes fit in memory, old partitions stop changing and get frozen once, and deleting a month is `DROP TABLE` instead of a huge `DELETE` followed by vacuum.

It doesn't help if the real problem is a missing index or a bad plan, if the hot queries look rows up by something other than the key, or if the table is just "big". Every query without the key now visits every partition. Fix plans first; partitioning adds operational surface and should buy something concrete.

## Choosing the key and the partition size

Three rules for the key. Nearly every query must filter on it, because pruning compares it with values in the query. It must never change: updating it moves the row to another partition, a delete plus insert (I watched a row move from `messages_p2026_05` to `messages_p2026_10`). And it should spread writes predictably. For messages and events that means `RANGE (inserted_at)`, with bounds in UTC.

For size, the [docs](https://www.postgresql.org/docs/current/ddl-partitioning.html) say the planner handles "up to a few thousand partitions fairly well" when queries prune all but a few. Too few, and partitions stay too big to gain anything. Too many, and planning time and per-session memory grow, because each partition touched loads its metadata into that backend. With 1,000 daily partitions, a query pruned at plan time took 3 ms to plan on my laptop and one that couldn't be pruned took about 100 ms. Monthly is a sensible default for message data; go smaller only when a month is too big to vacuum or index comfortably.

## Range, list or hash

```sql
CREATE TABLE messages_p2026_03 PARTITION OF messages
  FOR VALUES FROM ('2026-03-01 00:00:00+00') TO ('2026-04-01 00:00:00+00');

CREATE TABLE by_region (region text NOT NULL, x int) PARTITION BY LIST (region);
CREATE TABLE by_region_eu PARTITION OF by_region FOR VALUES IN ('eu', 'uk');
CREATE TABLE by_region_other PARTITION OF by_region DEFAULT;

CREATE TABLE tenant_events (tenant_id int NOT NULL, payload text) PARTITION BY HASH (tenant_id);
CREATE TABLE tenant_events_0 PARTITION OF tenant_events FOR VALUES WITH (MODULUS 4, REMAINDER 0);
```

Range suits time, with an inclusive lower bound and exclusive upper bound. List suits a small set of known values. Hash spreads writes evenly but prunes only on equality, doesn't help retention, and can't have a default ("a hash-partitioned table may not have a default partition").

## Partition pruning at plan time and at execution time

With literal values the planner prunes, and the other partitions simply aren't in the plan:

```sql
EXPLAIN (COSTS OFF)
SELECT count(*) FROM messages
WHERE contact_id = 42
  AND inserted_at >= '2026-03-01 00:00:00+00' AND inserted_at < '2026-04-01 00:00:00+00';
```

```text
 Aggregate
   ->  Index Only Scan using messages_p2026_03_contact_id_inserted_at_index on messages_p2026_03 messages
         Index Cond: ((contact_id = 42) AND (inserted_at >= '2026-03-01 00:00:00+00'::timestamp with time zone) AND ...
```

With `now()` or a parameter in a generic plan, the value isn't known when planning, so Postgres prunes when execution starts and reports how many partitions it dropped:

```sql
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, BUFFERS OFF, SUMMARY OFF)
SELECT count(*) FROM messages
WHERE contact_id = 42 AND inserted_at >= now() - interval '7 days';
```

```text
 Aggregate (actual rows=1.00 loops=1)
   ->  Append (actual rows=2.00 loops=1)
         Subplans Removed: 15
         ->  Index Only Scan using messages_p2026_09_contact_id_inserted_at_index on messages_p2026_09 messages_1 ...
         ->  Index Only Scan using messages_p2026_10_contact_id_inserted_at_idx on messages_p2026_10 messages_2 ...
         ->  Seq Scan on messages_p2026_11 messages_3 (actual rows=0.00 loops=1)
         ->  Seq Scan on messages_p2026_12 messages_4 (actual rows=0.00 loops=1)
```

A prepared statement forced to a generic plan (`SET plan_cache_mode = force_generic_plan`) gave the same `Subplans Removed: 15`. The third phase happens during execution, for values that change inside the query, such as the inner side of a nested loop. There, pruned partitions are shown as `(never executed)`, as the [docs](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITION-PRUNING) describe.

`enable_partition_pruning` is on by default; turning it off is only useful to compare plans. One trap I hit: with it off, partitions still carrying old `CHECK` constraints were excluded anyway by `constraint_exclusion = partition`. Only after I dropped those redundant CHECKs did all 19 partitions show up.

## Queries without the partition key

They still work; they visit every partition. `SELECT * FROM messages WHERE id = 123456` produced 19 index scans, one per partition. That's fine occasionally, but a hot path should carry the timestamp.

Functions on the column hide it from pruning. Both of these scanned every partition:

```sql
SELECT count(*) FROM messages WHERE inserted_at::date = '2026-03-05';
SELECT count(*) FROM messages WHERE date_trunc('month', inserted_at) = '2026-03-01 00:00:00+00';
```

Write ranges instead: `inserted_at >= '2026-03-05' AND inserted_at < '2026-03-06'`.

## Planning cost and partitionwise joins

Planning scales with the partitions left after plan-time pruning. On my 1,000-partition test table:

| Query shape | Planning time |
| --- | --- |
| Literal or bound `timestamptz` range | 3 to 4 ms |
| No filter on the key | about 100 ms |
| `inserted_at >= now() - interval '1 day'` | about 95 to 130 ms |

PostgreSQL 18's release notes list faster planning for queries touching many partitions, but the shape of the problem is the same: prune early, or pay per partition.

`enable_partitionwise_join` and `enable_partitionwise_aggregate` are off by default because they make planning more expensive. They can help when you join two tables partitioned the same way on the key, or aggregate by it, but the planner still decides by cost: on my test table, turning the aggregate setting on didn't change the plan for `GROUP BY inserted_at` or `GROUP BY contact_id, inserted_at`. Enable them per session for the queries that need them, and check:

```sql
SET enable_partitionwise_join = on;
SET enable_partitionwise_aggregate = on;
EXPLAIN (COSTS OFF)
SELECT contact_id, inserted_at, count(*) FROM messages GROUP BY 1, 2;
```

## Indexes on partitioned tables

`CREATE INDEX` on the parent builds on every partition while blocking writes, and `CONCURRENTLY` is refused: "cannot create index on partitioned table "messages" concurrently". The workaround from the [docs](https://www.postgresql.org/docs/current/ddl-partitioning.html) is three steps; psql's `\gexec` runs each generated statement separately, so it works for `CONCURRENTLY`:

```sql
CREATE INDEX messages_direction_inserted_at_index ON ONLY messages (direction, inserted_at);

SELECT format('CREATE INDEX CONCURRENTLY %I ON %I (direction, inserted_at)',
              c.relname || '_direction_inserted_at_index', c.relname)
FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid
WHERE i.inhparent = 'messages'::regclass \gexec

SELECT format('ALTER INDEX messages_direction_inserted_at_index ATTACH PARTITION %I',
              c.relname || '_direction_inserted_at_index')
FROM pg_inherits i JOIN pg_class c ON c.oid = i.inhrelid
WHERE i.inhparent = 'messages'::regclass \gexec

SELECT indisvalid FROM pg_index
WHERE indexrelid = 'messages_direction_inserted_at_index'::regclass;
```

The parent index starts invalid (`f`) and turned valid (`t`) once the last partition's index was attached. New partitions get it automatically.

## Primary keys and unique constraints

Each partition enforces uniqueness only within itself, so Postgres requires every unique constraint to include the key: "unique constraint on partitioned table must include all partitioning columns". The primary key becomes `(id, inserted_at)`. Upserts follow: `ON CONFLICT (id)` fails with "there is no unique or exclusion constraint matching the ON CONFLICT specification", and the target must be `(id, inserted_at)`. For a global dedup key such as a provider's message id, keep a small separate table with its own unique index.

## Foreign keys

Foreign keys *from* a partitioned table work as usual (PostgreSQL 11+). Foreign keys *to* one work from PostgreSQL 12, but must reference a unique key, which now includes the timestamp:

```sql
CREATE TABLE message_reactions (
  id bigserial PRIMARY KEY,
  message_id bigint NOT NULL,
  message_inserted_at timestamptz NOT NULL,
  emoji text NOT NULL,
  FOREIGN KEY (message_id, message_inserted_at) REFERENCES messages (id, inserted_at)
);
```

Referencing `messages (id)` alone fails with "there is no unique constraint matching given keys". Two more behaviours: detaching a partition that still has referenced rows fails ("removing partition "messages_p2025_06" violates foreign key constraint ..."), and PostgreSQL 18 adds `NOT VALID` foreign keys on partitioned tables, so you can validate them later without blocking writes.

## Retention with DETACH PARTITION CONCURRENTLY

```sql
ALTER TABLE messages DETACH PARTITION messages_p2025_06 CONCURRENTLY;
DROP TABLE messages_p2025_06;
```

Without `CONCURRENTLY`, detaching takes `ACCESS EXCLUSIVE` on the parent. With it (PostgreSQL 14+), [ALTER TABLE](https://www.postgresql.org/docs/current/sql-altertable.html) uses two transactions: `SHARE UPDATE EXCLUSIVE` on parent and partition, then a wait for every transaction using the table, then the final step. The restrictions (the first two errors are from my test):

- Inside a transaction block: "ALTER TABLE ... DETACH CONCURRENTLY cannot run inside a transaction block".
- With a default partition: "cannot detach partitions concurrently when a default partition exists".
- Only one partition per table can be pending detach. If it's interrupted, finish with `DETACH PARTITION ... FINALIZE`.

It also adds a CHECK duplicating the old bound to the detached table (`messages_p2025_06_inserted_at_check` in my test), which is handy if you ever re-attach it. Compared with deleting millions of rows and vacuuming, it's trivial.

## Autovacuum and ANALYZE per partition

Autovacuum treats each partition as a table and never processes the parent. The [docs](https://www.postgresql.org/docs/current/routine-vacuuming.html) point out the consequence: nothing runs `ANALYZE` on a partitioned table, so run it yourself after loading data or when the distribution shifts. Since PostgreSQL 18, `ANALYZE ONLY messages` refreshes the parent's statistics without re-analyzing every partition.

Autovacuum settings live on partitions. `ALTER TABLE messages SET (autovacuum_vacuum_scale_factor = 0.01)` fails with "cannot specify storage parameters for a partitioned table", and `LIKE ... INCLUDING ALL` didn't copy a partition's autovacuum settings, so set them in whatever creates new partitions.

## Ecto and Elixir notes

Ecto doesn't care about the database primary key, so the schema can keep `id` as its primary key. But `Repo.get(Message, id)` scans every partition's index. Where you have the timestamp, use it: `Repo.get_by(Message, id: id, inserted_at: inserted_at)`. Upserts need `conflict_target: [:id, :inserted_at]`.

Watch how time filters are built. On a `timestamptz` column, `ago/2` compiles to `$2::timestamp + (-7::decimal::numeric * interval '1 day')`, and that cast to `timestamptz` isn't immutable, so pruning waits until execution: on my 1,000-partition table, planning took 95 to 145 ms instead of 3.7 ms. Pinning a computed `DateTime` fixes it. With Ecto's default `timestamp` columns, `ago/2` pruned at plan time.

```elixir
cutoff = DateTime.add(DateTime.utc_now(), -7, :day)

from m in Message,
  where: m.contact_id == ^contact_id and m.inserted_at >= ^cutoff,
  select: count()
```

Migrations can declare a partitioned table directly. This generated `PRIMARY KEY ("id","inserted_at") ... PARTITION BY RANGE (inserted_at)`:

```elixir
create table(:events, primary_key: false, options: "PARTITION BY RANGE (inserted_at)") do
  add :id, :bigserial, primary_key: true
  add :contact_id, :bigint, null: false
  add :payload, :map
  add :inserted_at, :utc_datetime_usec, primary_key: true
end
```

`DETACH ... CONCURRENTLY` and `CREATE INDEX CONCURRENTLY` must run outside a transaction: in a migration, set `@disable_ddl_transaction true` and `@disable_migration_lock true` (the [Safe Ecto Migrations](https://github.com/fly-apps/safe-ecto-migrations) guide covers the trade-offs). From application code such as an Oban job, call `Repo.query!/3` outside `Repo.transaction/1`; that worked in my test. For creating partitions ahead of time, the migration post has the Oban worker I use.

## If you're partitioning a table

Most partitioning problems show up in `EXPLAIN` long before they show up in production latency, and fixing them early is cheap. If your Postgres (or Elasticsearch) cluster is slowing down as data grows and you'd like help choosing keys, checking plans or planning a migration, [that's what I do](https://fedme.dev/services/postgres-elasticsearch).


---

---
title: "Letting AI agents run code safely in Elixir with tv-labs/lua"
description: "How tv-labs/lua gives AI agents a sandboxed Lua runtime inside Elixir: exposing safe functions, setting limits and wiring code execution into the agent loop."
author: Federico Meini
date: 2026-08-04
tags: [elixir, ai-agents, lua, sandboxing]
language: en
url: https://fedme.dev/blog/lua-sandbox-for-ai-agent-tools-in-elixir
---

# Letting AI agents run code safely in Elixir with tv-labs/lua

If you build AI agents in Elixir and want them to run code or call tools without being able to damage your platform, use [tv-labs/lua](https://github.com/tv-labs/lua). It's a Lua 5.3 runtime that runs entirely on the BEAM, sandboxed by default, with a small and pleasant API for exposing exactly the Elixir functions you choose. At Turn.io, where I own the AI features, I connected our Lua app engine to the Agent block so that agents execute tools and custom code inside a Lua sandbox. That's how customers connect agents to medical records and external APIs without being able to break anything. We use this library, it works wonderfully, and this post is mostly a thank you to the tv-labs team, with the code to show why.

## TL;DR

- [tv-labs/lua](https://github.com/tv-labs/lua) is a **Lua 5.3 VM written in Elixir**: no NIFs, no C, nothing to compile.
- `Lua.new/1` is **sandboxed by default**: no `io`, no `os.execute`, no `require` or `load`.
- You expose capabilities with **`use Lua.API` and `deflua`**, and pass secrets through **private storage** the script can't read.
- **`:max_instructions`**, **`:max_call_depth`** and **`:max_string_bytes`** bound the work a script can do; a monitored process adds a wall-clock timeout and a memory cap.
- For agents, "run code" becomes one tool: the model writes Lua, you run it with limits, and you return **results or a readable error** to the model.

## Why Lua is a good fit for agent tools

When an agent needs to do more than call one fixed endpoint, for example fetch a record, filter a list and compute a date, you have two options. You can define a dozen narrow tools and hope the model chains them well, or you can give it one tool that runs a short script against an API you control. The second option is often simpler and cheaper in tokens, but only if the runtime is safe to hand to a model.

Lua is a good match for that:

- **It's small.** Models write it well, the standard library is compact, and there isn't much surface to reason about.
- **It's easy to restrict.** A script has no filesystem, network or environment access unless you give it some. Every capability is a function you put into the state yourself.
- **It's predictable.** If you also sandbox the clock and randomness (`os.time`, `os.clock`, `os.date`, `math.random`), a script's output depends only on its input and the functions you expose. That makes tool runs reproducible, which you'll appreciate when writing evals.
- **It runs inside the BEAM.** With tv-labs/lua, the VM is plain Elixir, so a script runs in an ordinary process that you can monitor, cap and kill. Each `Lua` value is immutable state that you thread explicitly, so nothing leaks between two tool runs unless you pass the state along yourself.

The library started as an Elixir wrapper around Robert Virding's [Luerl](https://github.com/rvirding/luerl), and the README credits it generously as prior art. Since 1.0 it has been a full reimplementation of the Lua 5.3 lexer, parser and VM in Elixir, with better error messages and features aimed squarely at running untrusted code. The README even names AI-agent-authored code as a primary use case.

## A sandboxed state and a first eval

Add `{:lua, "~> 1.0"}` to your deps. `Lua.new/1` gives you a sandboxed VM, and `Lua.eval!/3` returns the list of returned values plus the updated state:

```elixir
lua = Lua.new(max_instructions: 1_000_000, max_call_depth: 200)

{[4], _lua} = Lua.eval!(lua, "return 2 + 2")

Lua.eval!(lua, ~S[os.execute("ls")])
# ** (Lua.RuntimeException) Lua runtime error: os.execute(_) is sandboxed
```

The default deny-list covers the `io` library, `file`, `os.execute`, `os.exit`, `os.getenv`, `os.remove`, `os.rename`, `os.tmpname`, `package`, `require`, `load`, `loadfile`, `loadstring` and `dofile`. A sandboxed function still exists, but calling it raises. You can punch specific holes with `exclude:`, or add more paths with `Lua.sandbox/2`, for example `Lua.sandbox(lua, [:os, :time])`.

The limits are options on `Lua.new/1`. `:max_instructions` is an instruction budget per evaluation, and running out raises `"instruction budget exceeded"`. `:max_call_depth` turns runaway recursion into `"stack overflow"`. The VM also refuses allocation bombs like `string.rep("x", 1e15)` before allocating, with a ceiling you can lower through `:max_string_bytes`. All of these are ordinary Lua errors, so a script can catch them with `pcall` and your code can rescue them from `eval!`.

## Exposing Elixir functions to Lua

For a one-off, `Lua.set!/3` accepts a function that takes the argument list and returns a list of results:

```elixir
lua = Lua.set!(Lua.new(), [:sum], fn args -> [Enum.sum(args)] end)
{[10], _lua} = Lua.eval!(lua, "return sum(1, 2, 3, 4)")
```

For an agent's API, I prefer a module with `use Lua.API` and `deflua`. The `scope` option puts the functions under a namespace, and the `state` form gives the function the current `Lua` state:

```elixir
defmodule MyApp.AgentTools.LuaAPI do
  use Lua.API, scope: "records"

  # Callable from Lua as records.get("123")
  deflua get(id), state do
    client = Lua.get_private!(state, :records_client)

    case MyApp.Records.fetch(client, id) do
      {:ok, record} -> Lua.encode!(state, record)
      {:error, :not_found} -> {:error, "record #{id} not found"}
    end
  end
end

defmodule MyApp.AgentTools.Log do
  use Lua.API

  # Replaces Lua's print/1 so output is collected instead of written to stdout
  @variadic true
  deflua print(args), state do
    line = Enum.map_join(args, "\t", &to_string/1)
    {[], Lua.put_private(state, :output, [line | Lua.get_private!(state, :output)])}
  end
end
```

A few details here are worth knowing, and all of them are the library being careful on your behalf:

- **Private storage** (`Lua.put_private/3`, `Lua.get_private!/2`) holds values your Elixir functions can read but Lua code can't. That's where the API client and its credentials go. The model never sees them, and no script can print them.
- **Returns must be encoded.** A `deflua` that returns a plain map raises and tells you to use `Lua.encode!/2`, which returns `{encoded, state}`. That's exactly the shape `deflua` accepts, so `Lua.encode!(state, record)` can be the last expression.
- **`{:error, reason}` becomes a Lua error.** The script can catch it with `pcall`, or let it propagate to your Elixir code.
- **Arity is checked.** Calling `records.get("1", "2")` fails with "expected 1 arguments, got 2", a message a model can act on.

## Wiring it into the agent loop

In the agent, the whole thing is one tool. The definition tells the model what exists in the sandbox:

```json
{
  "name": "run_lua",
  "description": "Run a short Lua 5.3 script in a sandbox. records.get(id) returns a record as a table. Use print() for notes. Return the values you need; they are sent back to you as JSON.",
  "input_schema": {
    "type": "object",
    "properties": { "code": { "type": "string" } },
    "required": ["code"]
  }
}
```

When the model calls it, the harness builds a fresh state for that call, loads the APIs, injects the per-conversation client, and evaluates the script in a separate process with a timeout and a heap cap. The process wrapper follows the [Security & Sandboxing guide](https://github.com/tv-labs/lua/blob/main/guides/sandboxing.md), which is one of the most useful pages in the docs:

```elixir
defmodule MyApp.AgentTools.RunLua do
  @heap_words 8_000_000
  @timeout_ms 2_000

  def execute(%{"code" => code}, %{records_client: client}) do
    lua =
      Lua.new(max_instructions: 5_000_000, max_call_depth: 200, max_string_bytes: 1_000_000)
      |> Lua.load_api(MyApp.AgentTools.LuaAPI)
      |> Lua.load_api(MyApp.AgentTools.Log)
      |> Lua.put_private(:records_client, client)
      |> Lua.put_private(:output, [])

    case run_isolated(lua, code) do
      {:ok, results, output} -> {:ok, %{"result" => Enum.map(results, &to_json/1), "output" => output}}
      {:error, message} -> {:error, message}
    end
  end

  defp run_isolated(lua, code) do
    parent = self()
    prev_trap = Process.flag(:trap_exit, true)

    worker =
      spawn_link(fn ->
        # include_shared_binaries requires OTP 27+
        Process.flag(:max_heap_size, %{
          size: @heap_words,
          kill: true,
          error_logger: false,
          include_shared_binaries: true
        })

        result =
          try do
            {results, lua} = Lua.eval!(lua, code, source: "run_lua")
            {:ok, results, lua |> Lua.get_private!(:output) |> Enum.reverse()}
          rescue
            e in [Lua.CompilerException, Lua.RuntimeException] -> {:error, Exception.message(e)}
          end

        send(parent, {:result, result})
      end)

    try do
      receive do
        {:result, result} -> result
        {:EXIT, ^worker, :killed} -> {:error, "memory limit exceeded"}
        {:EXIT, ^worker, reason} -> {:error, "crashed: #{inspect(reason)}"}
      after
        @timeout_ms ->
          Process.exit(worker, :kill)
          {:error, "timed out after #{@timeout_ms}ms"}
      end
    after
      Process.flag(:trap_exit, prev_trap)
    end
  end

  # Decoded Lua tables arrive as lists of {key, value} pairs
  defp to_json(value) when is_list(value), do: Lua.Table.deep_cast(value)
  defp to_json(value), do: value
end
```

The loop then turns either branch into a tool result:

```elixir
case MyApp.AgentTools.RunLua.execute(call.input, ctx) do
  {:ok, payload} ->
    %{type: "tool_result", tool_use_id: call.id, content: Jason.encode!(payload)}

  {:error, message} ->
    %{type: "tool_result", tool_use_id: call.id, content: message, is_error: true}
end
```

Given a script like this one from the model:

```lua
local r = records.get("123")
print("visits:", #r.visits)
return { name = r.name, last_visit = r.visits[#r.visits].date }
```

the model gets back `{"output": ["visits:\t2"], "result": [{"last_visit": "2026-03-04", "name": "Test"}]}` with my test data. If it writes `return nope.x`, it gets `attempt to index a nil value (global 'nope') (at run_lua:1)`. If it loops forever, it gets `instruction budget exceeded`. All of those are messages a model can read and correct on the next step, which is exactly what you want from a tool.

## What to watch out for

**Set every limit explicitly.** `:max_instructions` and `:max_call_depth` default to `:infinity`, and the string ceiling defaults to 256 MiB. For agent code you want small numbers, and you want `:max_string_bytes` comfortably below your heap cap, as the guide recommends, so string bombs are refused deterministically instead of depending on garbage collection timing.

**The instruction budget isn't a clock.** It bounds the work the VM does, which is great because it's deterministic. It doesn't count time spent inside your own Elixir functions, though, and a slow external API call is where a real tool will spend most of its time. Keep the wall-clock timeout, and give your HTTP clients their own timeouts too.

**Expose capabilities, not a toolbox.** Every `deflua` is attack surface, because the script is written by a model that can be manipulated by whatever text ends up in its context. `records.get(id)`, scoped to the client you injected for this conversation, is fine. `http.request(url)` isn't. Validate arguments inside each function and treat everything that comes back from an external API as data. I wrote more about this in [how agents hand over safely](https://fedme.dev/blog/agent-handover-architecture-whatsapp).

**Return errors to the model, not to your logs only.** Rescue `Lua.CompilerException` and `Lua.RuntimeException` and send `Exception.message/1` back as the tool result. It's short, free of ANSI codes and includes the source name and line number, which is usually enough for the model to fix its script. If you need structured errors for logging or a UI, `Lua.RuntimeException.to_map/2` gives you a JSON-safe map.

**Know what's out of scope.** The library targets Lua 5.3 without coroutines, weak tables or the full `debug` library, all listed as deliberate non-goals in the README. None of that has mattered for agent tools in my experience, but it's good to know before you promise customers "full Lua".

## A small contribution

While hosting sandboxed Lua apps, we hit a bug: returning a table that contains itself, like the very common `T.__index = T` class idiom, made the eval boundary recurse forever. I sent [a small fix](https://github.com/tv-labs/lua/pull/407) that stops the walk when it meets a cycle, and it shipped in 1.0.2. It was reviewed and merged the same day, which says a lot about how the project is run.

## Thank you, tv-labs

Running model-written code safely is one of those problems that looks simple until you list everything that can go wrong. tv-labs/lua takes care of most of that list: a sandbox by default, deterministic limits, errors with line numbers, and documentation that tells you honestly where the VM's guarantees end and yours begin. Huge thanks to the tv-labs team for building it and sharing it. Start with the [hexdocs](https://hexdocs.pm/lua), and read the Security & Sandboxing guide before you ship.

If you're building agents that need to call real systems without putting those systems at risk, [I help teams design AI agents, their tools and the evals that prove they work](https://fedme.dev/services/ai-agents-evals).


---

---
title: "Serving MedGemma 27B on Modal: FP8, vLLM sleep mode and 21-second cold starts"
description: "How I self-host MedGemma 27B as a scale-to-zero, OpenAI-compatible API on Modal: FP8 quantisation, GPU snapshots, two vLLM bugs and Gemma 3 tool calling."
author: Federico Meini
date: 2026-07-22
tags: [llm, vllm, self-hosting, medgemma, ai-agents]
language: en
url: https://fedme.dev/blog/serving-medgemma-27b-on-modal-vllm
---

# Serving MedGemma 27B on Modal: FP8, vLLM sleep mode and 21-second cold starts

You can serve a 27B-parameter medical model on serverless GPUs, pay only while it's answering, and still get a cold start of about 21 seconds. The recipe that worked for me:

1. Quantise [MedGemma 27B](https://huggingface.co/google/medgemma-27b-text-it) to FP8 ahead of time.
2. Serve it with vLLM on Modal.
3. Put vLLM to sleep at level 2 before Modal takes a GPU memory snapshot, so the snapshot holds about 1 GiB of runtime state instead of 32 GiB of weights.
4. On wake, reload the weights from a cached volume.

Getting there took two vLLM bug fixes, a custom tool-call parser and a few environment variables that I would not have guessed.

I built this at Turn.io, where we run AI agents for health services on WhatsApp. We wanted to know whether an open medical model could replace frontier models for some of that traffic. To find out, we ran it against the same simulation evals we use for production bots rather than public benchmarks. The quantised model is public: [`turnio/medgemma-27b-text-it-FP8-Dynamic`](https://huggingface.co/turnio/medgemma-27b-text-it-FP8-Dynamic). This post covers the serving side.

## TL;DR

- **FP8 on disk, not at runtime.** Quantising BF16 weights to FP8 when vLLM loads them runs out of memory while the snapshot is being created. So I quantise once with llm-compressor (`FP8_DYNAMIC`, no calibration data), which halves the weights from ~54 GiB to ~27 GiB, and publish the result.
- **Snapshot the process, not the weights.** vLLM sleep level 2 discards the weights before Modal snapshots GPU memory. The snapshot shrank from ~32 GiB to ~1 GiB, and cold starts fell from ~100 s to ~21–24 s.
- **Level 2 needed vLLM 0.16.0 plus a one-line patch.** One bug corrupted FP8 weights on reload; the other stopped vLLM from freeing the weights at all.
- **Gemma 3 doesn't speak OpenAI tool calls.** A small vLLM parser plugin and chat template translate its `tool_code` blocks into standard `tool_calls`, including when streaming.
- **Size the GPU for context, not for weights.** The model fits on an L40S, but the full 131K context only fits on an H100. That roughly doubles the warm-hour cost, from about $1.95 to $3.95.

## Why self-host a medical model at all

Frontier APIs are the right default for most agents. They are good, cheap per call and someone else keeps them running. Our reasons to try an open model were specific:

- MedGemma is trained for medical text.
- We wanted a model we control end to end.
- We wanted to compare it with the OpenAI and Claude models our customers already use, on the conversations they actually have.

That last point matters most. We didn't decide on MedQA scores; we ran the same multi-turn [simulation evals](https://fedme.dev/blog/simulation-evals-for-llm-chatbots) against live bots with each model behind them. The results now feed a routing policy between models.

The catch with a 27B model is cost. An always-on GPU for traffic that comes in bursts is money burned while nobody is talking. Modal lets containers scale to zero and bills by the second, so the whole design question became: **how fast can a scaled-to-zero container answer its first request?**

## The shape of the deployment

The app is one Modal class that starts `vllm serve` as a subprocess and exposes its port as a web endpoint. That gives clients an OpenAI-compatible `/v1/chat/completions` API, and any OpenAI SDK works by changing `base_url`.

```python
@app.cls(
    image=vllm_image,
    gpu="H100",
    scaledown_window=10 * MINUTES,  # scale to zero after 10 min idle
    min_containers=0,
    max_containers=3,               # cost safety cap
    volumes={
        "/root/.cache/huggingface": hf_cache_vol,
        "/root/.cache/vllm": vllm_cache_vol,
    },
    secrets=[modal.Secret.from_name("huggingface-secret"),
             modal.Secret.from_name("medgemma-api-key")],
    enable_memory_snapshot=True,
    experimental_options={"enable_gpu_snapshot": True},
)
@modal.concurrent(max_inputs=16)  # small queue above vLLM's max_num_seqs=8
class VllmServer:
    ...
```

A few choices worth calling out:

- **The weights live on a Modal volume** (the Hugging Face cache), not in the image. Images stay small, and the weights are downloaded once.
- **The API key is enforced by vLLM itself** (`--api-key`), read from a Modal secret. The endpoint is public, so this is not optional.
- **`max_containers=3`** is a cost cap, not a performance setting. A runaway client should hit a queue, not your credit card.

## Step 1: quantise to FP8 yourself

The official checkpoint is BF16, about 54 GiB. My first attempt used vLLM's runtime FP8 quantisation (`--quantization fp8`). It ran out of memory during snapshot creation, because the BF16 and FP8 copies briefly coexist.

The fix is to store FP8 weights on disk. I briefly used a community FP8 checkpoint, then replaced it with our own. For a medical model I wanted to know exactly how the weights were produced, and to be able to re-run it for new MedGemma releases. Quantisation is a one-shot Modal function with llm-compressor:

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained("google/medgemma-27b-text-it", dtype="auto")

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",   # per-channel weight scales, per-token dynamic activations
    ignore=["lm_head"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(save_dir, save_compressed=True)
```

`FP8_DYNAMIC` needs no calibration dataset. Weight scales are computed per output channel, and activations are quantised per token at inference time. The result is about 27 GiB in the compressed-tensors format. It fits a 48 GB L40S with room for KV cache, and vLLM picks it up with `--dtype auto`.

Per-channel scales are slightly more precise than the per-tensor scales of the older "native" FP8 format. For a model this size the difference is negligible in practice. Our real quality check was the simulation evals anyway, not perplexity.

## Step 2: make cold starts fast with GPU snapshots

Without snapshots, a fresh container downloads nothing (the volume is warm) but still spends minutes loading weights, compiling and warming up. It took about 5 minutes end to end. Modal's GPU memory snapshots let you do that work once, freeze the process, and restore it on every cold start.

The obvious approach is to snapshot the whole loaded model. It works, but the snapshot is huge. vLLM's sleep mode is designed for exactly this: `POST /sleep` prepares the engine to be frozen, and `POST /wake_up` brings it back. It has two levels:

| | Level 1 | Level 2 |
|---|---|---|
| On sleep | Weights copied GPU → CPU (and so into the snapshot) | Weights discarded |
| On wake | CPU → GPU copy | Reload weights from disk |
| Snapshot size | ~32 GiB | ~1 GiB |
| Snapshot restore | ~95 s | ~8 s |
| Wake | ~2.5 s | ~13–16 s |
| **Total cold start** | **~100 s** | **~21–24 s** |

Level 2 wakes more slowly because it has to read 27 GiB from the volume. But restoring a 32 GiB snapshot is so much slower that the net saving is almost 80 seconds. The bottleneck moves from snapshot I/O to volume I/O, and the volume is faster.

The lifecycle maps neatly onto Modal's two kinds of `@modal.enter` hooks:

```python
@modal.enter(snap=True)          # runs once; Modal snapshots right after it
def start(self):
    self.vllm_proc = subprocess.Popen(cmd)
    wait_ready(self.vllm_proc)
    warmup()                     # one tiny request to trigger lazy init
    vllm_sleep(level=2)          # drop the weights before the snapshot

@modal.enter(snap=False)         # runs on every restore
def wake(self):
    wake_up()                    # remap GPU memory, no weights yet
    reload_weights()             # POST /collective_rpc {"method": "reload_weights"}
    reset_prefix_cache()         # don't serve stale prefix-cache entries
    wait_ready(self.vllm_proc)
```

Two flags helped keep the snapshot small:

- **`--enforce-eager`** skips CUDA graph capture. That removes 10–20 GiB of GPU memory from the snapshot and 10–20 s of init time, at the price of slightly higher per-token latency, which is fine for a chatbot.
- **`--gpu-memory-utilization 0.85`**, which is also a ceiling. The layerwise weight reload after level 2 sleep needs temporary GPU buffer space, so going higher breaks the wake.

## The two vLLM bugs behind level 2

On paper, level 2 is one argument. In practice my first attempt produced garbage output after waking, and I reverted to level 1 the same afternoon. Two separate bugs were involved.

### Bug 1: quantised weights reloaded in the wrong layout

When vLLM first loads a quantised model, it runs `process_weights_after_loading()` on each layer, which repacks checkpoint tensors into the layout the kernels want. The old `reload_weights()` path skipped that step. It loaded checkpoint-format tensors into kernel-format parameters, and for compressed-tensors FP8 this didn't raise an error. It just produced nonsense. This is [vllm#28606](https://github.com/vllm-project/vllm/issues/28606).

The fix is layerwise reloading, which re-runs the post-processing per layer ([vllm#32133](https://github.com/vllm-project/vllm/pull/32133)). It shipped in vLLM 0.16.0, so step one was upgrading.

### Bug 2: an `and` where a comma should be

On 0.16.0, level 2 sleep reported that it freed about 5 GiB and that 27.88 GiB was *still in use*. The weights weren't being released at all. The cause turned out to be one character in `gpu_worker.py`:

```python
# vLLM 0.16.0
with self.mem_allocator.use_memory_pool(tag="weights"
    ) and set_current_vllm_config(self.vllm_config):

# fixed upstream in vllm#32947
with self.mem_allocator.use_memory_pool(tag="weights"
    ), set_current_vllm_config(self.vllm_config):
```

`a and b` evaluates to `b` when `a` is truthy, so only the second context manager is entered. The memory pool that tags weight allocations never becomes active. Sleep can't free what the allocator never tracked.

The fix ([vllm#32947](https://github.com/vllm-project/vllm/pull/32947)) was merged upstream after 0.16.0 was released, so I patch it at image build time:

```python
.run_commands(
    "sed -i 's/) and set_current_vllm_config(/), set_current_vllm_config(/' "
    "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py",
)
```

With the patch, sleep freed 34.48 GiB and left 0.9 GiB in use. Snapshot creation went from about 2 minutes to 15 seconds. The KV cache also grew from 13,680 to 17,872 tokens on the same GPU, a free bonus from tracking memory properly.

## Smaller things that broke snapshots

None of these took long to fix once found, but every one of them cost a deploy cycle to find:

- **Hugging Face Xet.** Newer `huggingface_hub` versions download through the Xet backend, which writes files into the cache volume that are gone by the time Modal restores the snapshot. The restore failed with `vfs.CompleteRestore` errors. `HF_HUB_DISABLE_XET=1` fixed it. The older `HF_HUB_ENABLE_HF_TRANSFER=0` no longer controls this.
- **fastsafetensors.** I tried `--load-format fastsafetensors` to speed up the weight reload with GPU Direct Storage. Modal volumes don't support GDS, so it went back out.
- **cuBLAS mismatch.** vLLM 0.16.0's torch bundles cuBLAS 12.8, which caused `CUBLAS_STATUS_INVALID_VALUE` on a CUDA 12.9 base image. Pinning `nvidia-cublas-cu12==12.9.1.4` with `--no-deps` fixed it.
- **Snapshot-safe compilation.** Set `TORCHINDUCTOR_COMPILE_THREADS=1` (required for memory snapshots), and `TORCH_CUDA_ARCH_LIST` to the GPU you actually run so you don't compile for every architecture.
- **NCCL heartbeat noise.** `/sleep` shuts down a TCPStore that the NCCL heartbeat thread keeps pinging, which floods the logs with "broken pipe" warnings. Setting `TORCH_NCCL_COORD_CHECK_MILSEC` to ten minutes makes the thread sleep through the wake cycle.

## Tool calling: teaching vLLM to read Gemma 3

Our agents use tools, so a model without OpenAI-style `tool_calls` is not usable behind them. Gemma 3, and therefore MedGemma, was trained to call tools with Python syntax inside a fenced block:

````text
```tool_code
print(calculate_bmi(height_cm=180, weight_kg=75))
```
````

vLLM doesn't parse that, so the deployment ships two extra files, both loaded through vLLM's own extension points:

- **A chat template** that renders the request's `tools` as Python function signatures with docstrings. It tells the model to use `tool_code` blocks for calls and plain text for everything else.
- **A tool parser plugin** (`--tool-parser-plugin`, `--tool-call-parser medgemma`). It finds `tool_code` blocks, strips the `print(...)` wrapper, parses the keyword arguments and returns standard `ToolCall` objects.

The parsing is mostly regex plus a small character-by-character scanner for arguments, because values can contain commas inside quotes or nested parentheses. The interesting parts are the quirks:

- **Plain text wrapped as a tool call.** Sometimes the model "replies" with `print("Sure, here's…")` inside a `tool_code` block. The parser recognises a bare string literal and returns it as normal content instead of a broken tool call.
- **Leaked thinking tokens.** Gemma 3 occasionally emits its internal thinking markers (`<unused94>`…`<unused95>`), sometimes only half of the pair. They're stripped in both streaming and non-streaming modes.
- **Markers split across stream chunks.** In streaming mode ```` ``` ```` can arrive in one delta and `tool_code` in the next. The parser holds back any suffix that could be the start of the marker, so clients never see stray backticks before a tool call is detected.

The parser has unit tests and a streaming integration test, with CI running them on every push. Streaming parsers are exactly the kind of code that works in the demo and fails on the third token boundary.

## Sizing: the GPU is for the context, not the weights

The FP8 model fits comfortably on an L40S (48 GB). For months that's where it ran, with the context capped at 8K tokens, at roughly half the price of an H100.

The cap existed because of the KV cache, not the weights. Gemma 3 uses hybrid attention: 10 global layers attend over the full sequence, while 52 sliding-window layers only keep roughly a window's worth of tokens. A full 131K-token sequence needs about 13.7 GiB of KV cache, about 10 GiB of it for the 10 global layers. The L40S had around 9 GiB left for KV cache after weights and buffers. The H100 has around 35 GiB, which fits about three full-length requests at once.

Moving to the H100 took warm-hour cost from about $1.95 to $3.95. With scale-to-zero, that's the cost per hour the container is actually awake, not per hour of the month. Serving settings for the H100:

```text
--max-model-len 131072          # full context
--max-num-seqs 8                # concurrent sequences
--max-num-batched-tokens 8192   # per engine step; chunked prefill splits longer prompts
--gpu-memory-utilization 0.85   # ceiling set by the layerwise reload buffer
```

## One unglamorous middleware

After launch, some callers got 404s without ever reaching the model. One client had set its base URL to the full completions URL, so the SDK appended the path again: `/v1/chat/completions/chat/completions`. Another probed `/v1/health`, while vLLM serves health at `/health`.

Rather than chase every integration, I added a 20-line ASGI middleware, loaded with vLLM's `--middleware` flag, that rewrites those two paths. It runs before vLLM's auth middleware, which only guards `/v1` paths, so the rewrite doesn't open anything up.

It's not elegant, but it's honest about how clients behave in the wild.

## What I'd tell someone doing this next

- **Measure every phase of the cold start separately.** I added `[cold-start]` timing logs to each wake step (restore, wake, reload, prefix-cache reset, ready). Without them I would have optimised the wrong thing, because the snapshot restore dominated and not the model load.
- **Read the sleep log line.** "Sleep mode freed X GiB, Y GiB still in use" is the most useful number in the whole setup. If Y is large, your snapshot is carrying weights you think you dropped.
- **Own your quantised checkpoint.** It's a one-off job, it's reproducible, and it removes a third party from the supply chain of a medical model.
- **Decide on your own evals, not on benchmarks.** A model that is cheap to host and good on MedQA can still be the wrong one for your conversations. Simulation evals on real bots answered the question benchmarks couldn't.

If you want an open model behind your agents, with cold starts, tool calling and evals handled properly, [that's the kind of work I do](https://fedme.dev/services/ai-agents-evals).


---

---
title: "Designing AI agents that know when to hand over: to other agents, to rules, to humans"
description: "A practical architecture for AI agents that hand over safely: multi-agent transfers, a deterministic rules engine for high-risk calls, sandboxed tools, evals."
author: Federico Meini
date: 2026-07-14
tags: [ai-agents, multi-agent, guardrails, llm-evals, whatsapp]
language: en
url: https://fedme.dev/blog/agent-handover-architecture-whatsapp
---

# Designing AI agents that know when to hand over: to other agents, to rules, to humans

An AI agent knows when to hand over when handover is a typed action in its loop, not something the model is expected to say in prose. The agent can transfer the conversation to another agent, defer a decision to a deterministic rules engine, or escalate to a human, and each of those is a tool call with a defined contract that the harness executes, logs and can evaluate. The model decides *that* it should hand over; code decides *what happens next*. At Turn.io I own the agentic harness customers use to run AI agents on WhatsApp, covering tool use, orchestration, guardrails and handover between agents and to humans. This post describes the architecture I'd recommend for any agent that talks to real users about things that matter.

## TL;DR

- The **harness owns the loop**: context, tool execution, limits, guardrails and handover. The model proposes; the harness disposes.
- Model every handover as a **tool with a schema**: allowed targets, a reason, and a structured summary.
- Keep the **handover graph explicit**. Each agent can only transfer to the agents you list, and transfers per conversation are capped.
- Put **high-risk decisions in a deterministic rules engine**. The LLM collects inputs and explains outcomes; it doesn't make the call.
- Run tools in a **sandbox** with least privilege, limits, and outputs treated as untrusted data.
- Layer **guardrails** so that the cheap deterministic checks run first and some situations bypass the model entirely.
- **Evaluate handover** like a classifier: missed handovers, unnecessary ones, wrong targets and the quality of what gets passed along.

## The agent loop, and who owns what

Strip away the frameworks and an agent is a loop: build the context, call the model with the available tools, execute whatever tools it asks for, append the results, repeat until it produces a reply for the user or hands over. What makes it production-grade is everything the harness enforces around that loop.

A sketch in Elixir (the `LLM`, `Tools` and `Guardrails` modules stand in for your own):

```elixir
defmodule Agents.Loop do
  @max_steps 8

  def run(agent, conversation, step \\ 0)

  def run(_agent, conversation, @max_steps),
    do: {:handover, %{target: "human", reason: "step_limit"}, conversation}

  def run(agent, conversation, step) do
    case LLM.complete(agent.model, agent.instructions, conversation, agent.tools) do
      {:reply, text} ->
        case Guardrails.check_output(agent, conversation, text) do
          :ok -> {:reply, text, conversation}
          {:violation, reason} -> {:handover, %{target: "human", reason: reason}, conversation}
        end

      {:tool_calls, calls} ->
        case Enum.find(calls, &(&1.name == "transfer")) do
          nil ->
            results = Enum.map(calls, &Tools.execute(agent, &1))
            run(agent, Conversation.append_tool_results(conversation, calls, results), step + 1)

          transfer ->
            {:handover, transfer.input, conversation}
        end
    end
  end
end
```

Note what the model doesn't control. It can't exceed the step limit, it can't send a reply that fails the output checks, and when it asks to transfer, the loop stops and the harness takes over. The caller then decides what a handover means: start another agent, run the rules engine, or put the conversation in a human queue.

## Handover as a tool with a contract

A common failure in agent designs is handover defined only in the prompt: "if the user asks about billing, tell them you'll transfer them". The model then says it's transferring, and nothing happens, or it transfers in a way nobody can measure. Make it a tool:

```json
{
  "name": "transfer",
  "description": "Hand the conversation to another agent or to a human. Use it when the request is outside your scope, when the user asks for a person, or when you cannot proceed safely.",
  "input_schema": {
    "type": "object",
    "properties": {
      "target": { "type": "string", "enum": ["appointments_agent", "human_nurse"] },
      "reason": { "type": "string", "enum": ["out_of_scope", "user_requested_human", "unsafe_to_continue", "tool_failure"] },
      "summary": { "type": "string", "description": "What the user needs and what has been established so far." }
    },
    "required": ["target", "reason", "summary"],
    "additionalProperties": false
  }
}
```

The `enum` on `target` is the handover graph for this agent: it can only go where you've drawn an edge. A structured `reason` gives you something to aggregate and alert on. The `summary` is what the next agent or person reads first.

## Handing over to other agents

Multi-agent setups earn their complexity when different parts of a service need different instructions, tools or models: an intake agent, an appointments agent with access to a booking API, an information agent with RAG over a knowledge base. The patterns I find work:

- **A router at the front, specialists behind it.** The router is small and cheap, with one job: work out what the user wants and transfer. Specialists don't need to know about each other unless you draw that edge.
- **Transfer the conversation, not just a summary.** The receiving agent should see the transcript plus the structured summary. Summaries lose details, and users hate repeating themselves.
- **Cap transfers.** Two agents that each think the other is responsible will ping-pong forever. After a small number of transfers in one conversation, the harness routes to a human regardless of what the model wants.
- **Make the active agent explicit state.** Store which agent owns the conversation. When the next WhatsApp message arrives an hour later, it should go straight to that agent, not back through the router.

## Handing over to rules: deterministic decisions for high-risk calls

For some decisions you don't want a probabilistic answer at all. In AI triage work I've designed two-agent architectures with a deterministic rule engine for the high-risk decisions. The general shape: the first agent's job is to collect structured findings from a conversational, messy exchange. The rule engine decides the outcome from those findings. The second agent explains the outcome and next steps in plain language, and it can't change the decision.

The rules are ordinary code, versioned and reviewed by the people accountable for the outcome. An illustrative sketch (not clinical guidance):

```elixir
defmodule Triage.Rules do
  @required [:age_months, :danger_signs, :symptom_days, :fever]

  def decide(findings) do
    case Enum.reject(@required, &Map.has_key?(findings, &1)) do
      [] -> classify(findings)
      missing -> {:need_more_info, missing}
    end
  end

  defp classify(%{danger_signs: [_ | _] = signs}), do: {:emergency, {:danger_signs, signs}}
  defp classify(%{age_months: age, fever: true}) when age < 3, do: {:emergency, :young_infant_fever}
  defp classify(%{symptom_days: days}) when days > 14, do: {:see_clinic, :persistent_symptoms}
  defp classify(_findings), do: {:self_care, :no_rule_matched}
end
```

Two properties make this work. First, `{:need_more_info, missing}` goes back to the intake agent as a tool result, so the rules drive which questions get asked. The model can't skip a required question and still reach an outcome, which directly attacks errors of omission. Second, the rules can be tested exhaustively and cheaply against a dataset of clinically reviewed vignettes, without any LLM in the loop. The LLM evaluation then narrows to a much easier question: did the intake agent extract the findings correctly from the conversation?

## Sandboxing tool execution

Tools are where agents touch the real world, so they need the same care as any other code that runs on behalf of a user, and more, because the caller is a model that can be manipulated. At Turn.io I connected our Lua app engine to the Agent block so agents execute tools and custom code inside a Lua sandbox; that's how customers connect agents to EMRs and external APIs safely. Whatever runtime you use, the principles are the same:

- **No ambient authority.** The tool gets only the credentials it needs, injected by the harness for that call. The model never sees secrets.
- **Hard limits.** CPU time, memory, wall-clock timeout, response size, allowlisted hosts.
- **Validated inputs.** Arguments are checked against the tool's schema before execution. A failed validation is returned to the model as an error, not executed on a best-effort basis.
- **Untrusted outputs.** A response from an external API is data, not instructions. Text in it that says "ignore previous instructions" is a prompt injection attempt, and your tool results should be clearly delimited as data in the context.
- **Confirm side effects.** Reads can be automatic. Anything irreversible (booking, cancelling, sending) should be idempotent and, depending on the stakes, confirmed with the user or a human first.

## Guardrails, in layers

Guardrails are more than a moderation call on the output. I think of them in three layers:

1. **Before the model.** Deterministic checks on the input: known crisis phrases, explicit requests for a human, messages from numbers flagged as abusive. Some of these should route to a human or a fixed safety response *without calling the model at all*. If someone says they're in danger, you don't want that handled by whatever the prompt currently says.
2. **Around the model.** Step limits, transfer limits, cost and latency budgets per conversation, and a defined fallback when the model provider is down or slow. On WhatsApp, a fallback that says "we're having trouble, a person will get back to you" is far better than silence.
3. **After the model.** Output checks: no diagnoses where the service isn't allowed to give them, no personal data from other records, no promises the service can't keep. Cheap deterministic checks first, a classifier model for what can't be expressed as rules.

A guardrail that fires should produce a handover or a safe response, and it should be logged with a reason, just like a model-initiated transfer.

## Handing over to humans

The human handover is where many agent projects are weakest, because it's partly an operations problem. Things to decide up front:

- **What the human sees.** The reason, the structured summary, the key facts already collected, and the full transcript one click away. A nurse shouldn't have to re-ask the patient's age.
- **What the user sees.** Say that a person will reply, and set realistic expectations about when. Then make sure the agent stops replying.
- **Out-of-hours behaviour.** If nobody is available until morning, say so, and have a plan for urgent cases that doesn't depend on the queue.
- **The WhatsApp window.** If the human replies more than 24 hours after the user's last message, a free-form reply will be rejected and you need an approved template to re-open contact.
- **Handing back.** After the human has resolved the issue, can the agent resume? If so, the human's messages must be part of the agent's context, and the handover state must be cleared explicitly.

## Evaluating handover correctness

Handover is a decision, so evaluate it like a classifier. For each conversation, you want to know whether the agent should have handed over, whether it did, to whom, and when. That gives you:

- **Missed handovers**: it should have escalated and didn't. In high-stakes services, this is the number to drive down first.
- **Unnecessary handovers**: it escalated when it could have handled the request. These cost staff time and erode the case for the agent.
- **Wrong target**: right decision to transfer, wrong destination.
- **Timing**: how many turns it took. A handover on turn nine, after the user said "I want to speak to a person" on turn two, is a failure even though it eventually happened.
- **Context quality**: whether the summary contains the facts the receiver needs. This is a good job for an LLM judge with a criterion per required fact.

The best source of test cases is [simulation evals](https://fedme.dev/blog/simulation-evals-for-llm-chatbots): scenarios where an LLM plays the user, labelled with the expected handover behaviour. Include users who ask for a human indirectly, users who are angry but don't need one, and users whose request drifts out of scope halfway through. Unit-test the rules engine separately, then run the same handover criteria on sampled production conversations, and turn every confirmed miss into a new scenario.

## Building agents people can trust

An agent that never hands over is either doing something trivial or doing something dangerous. Getting handover right is mostly architecture: explicit contracts, deterministic paths for the decisions that matter, sandboxed tools, and evals that measure whether it all works. If you're designing an agent for a service where mistakes have consequences, [I help teams build AI agents and the evals that prove they work](https://fedme.dev/services/ai-agents-evals).


---

---
title: "Simulation evals: letting an LLM play the user to test your chatbot before real users do"
description: "How to build simulation evals for LLM chatbots: persona-driven simulated users, LLM-as-judge rubrics, error rates for high-stakes flows and a feedback loop."
author: Federico Meini
date: 2026-06-30
tags: [ai-agents, llm-evals, llm-as-judge, python]
language: en
url: https://fedme.dev/blog/simulation-evals-for-llm-chatbots
---

# Simulation evals: letting an LLM play the user to test your chatbot before real users do

A simulation eval tests a chatbot by having a second LLM play the user. You give the simulated user a persona and a goal, it holds a complete multi-turn conversation with your real bot, and LLM judges score the finished transcript against criteria you define up front. Run that across a set of scenarios, several times each, and you get something single-prompt evals can't give you: an error rate for whole conversations, measured before a real person talks to the bot. I built this at Turn.io because customers doing clinical triage over WhatsApp needed to know how often the bot got it wrong before real patients used the service.

This post covers how I structure scenarios, the conversation loop, judges and rubrics, how to turn judge output into error rates you can defend, and how offline simulations connect to evals on production traffic.

## TL;DR

- A scenario is a **persona**, a **goal** and a set of **facts** the simulated user knows but only reveals when asked.
- The loop alternates simulator and bot turns until the simulator says it's done, the bot hands over, or a turn limit hits.
- Score each transcript with **one judge per criterion**, with a pass/fail/not-applicable verdict and quoted evidence. Check anything deterministic in code, not with a judge.
- For high-stakes flows, report **sensitivity** and **errors of omission**, not an average score.
- Run every scenario several times. LLM conversations are stochastic, and a scenario that fails one run in five is a real failure.
- **Calibrate judges** against human labels before trusting them.
- Apply the same criteria to production traces (OpenTelemetry GenAI), and turn every failure into a concrete change plus a new scenario.

## Why single-turn evals aren't enough for chatbots

Most eval tooling assumes input, output, expected output. Chatbots don't fail like that. They fail in the fourth message, when the user mentions a symptom in passing and the bot doesn't follow up. They fail when the user answers a question with another question, switches language, or gives the information out of order. None of that shows up when you test one prompt at a time.

You could script conversations by hand, but scripted user turns don't react to what the bot says. The moment the bot asks something your script didn't anticipate, the test is meaningless. A simulated user adapts, which is the point.

## Anatomy of a scenario

I keep scenarios as data, usually YAML or rows in a table, so domain experts can write and review them without touching code:

```yaml
id: chest-pain-vague-01
persona: >
  52-year-old man, writes short messages with typos, downplays symptoms,
  switches between English and his first language, gets impatient with long questions.
goal: >
  Find out whether he should go to the clinic tomorrow about
  chest discomfort that started this afternoon.
facts: >
  Pressure in the chest for about two hours. Pain goes to the left arm
  when asked. Slightly short of breath. Smoker. No known heart condition.
expected:
  triage_level: emergency
```

The split between goal and facts matters. The goal drives the conversation. The facts are what the person would say if asked the right question, which is how you measure whether the bot asks the right questions. If you put every fact in the opening message, you're testing reading comprehension, not triage.

For clinical flows, scenarios come from **vignettes**: short, clinically reviewed case descriptions with a known correct outcome. Clinicians are better at writing these than engineers, and a few dozen good ones beat hundreds of generated ones. Generate variations (persona, language, writing style) around a reviewed core, not the core itself.

## The conversation loop

Here is a compact version of the loop in Python, using the Anthropic SDK for the simulated user. The bot under test is whatever you actually ship, called through the same interface real users hit (a staging number, or the API behind it).

```python
import anthropic
from dataclasses import dataclass

client = anthropic.Anthropic()
MODEL = "claude-opus-5"
END = "[END]"

@dataclass
class Scenario:
    id: str
    persona: str
    goal: str
    facts: str
    max_turns: int = 12

SIMULATOR_PROMPT = """You are role-playing a person messaging a health service on WhatsApp.
Stay in character. Never say you are an AI or that this is a test.

Persona: {persona}
Your goal: {goal}
Facts you know. Share each one only when asked, or when a real person would
naturally bring it up: {facts}

Write short, informal messages, like someone typing on a phone.
When your goal is met, or it is clear the service cannot help you,
reply with exactly {end}."""

def next_user_message(system: str, transcript: list[dict]) -> str:
    # From the simulator's point of view the bot is the "user", so roles flip.
    messages = [{"role": "user", "content": "(The chat is open. Send your first message.)"}]
    for turn in transcript:
        role = "assistant" if turn["role"] == "user" else "user"
        messages.append({"role": role, "content": turn["text"]})

    response = client.messages.create(
        model=MODEL, max_tokens=16000, system=system, messages=messages
    )
    return "".join(b.text for b in response.content if b.type == "text").strip()

def simulate(scenario: Scenario, bot) -> dict:
    system = SIMULATOR_PROMPT.format(
        persona=scenario.persona, goal=scenario.goal, facts=scenario.facts, end=END
    )
    transcript: list[dict] = []
    stop_reason = "max_turns"

    for _ in range(scenario.max_turns):
        user_text = next_user_message(system, transcript)
        if user_text == END:
            stop_reason = "user_done"
            break
        transcript.append({"role": "user", "text": user_text})

        reply = bot.send(user_text)  # your real bot; may return several messages
        transcript.append({"role": "bot", "text": "\n".join(reply.messages)})
        if reply.handed_over:
            stop_reason = "handover"
            break

    return {"scenario": scenario.id, "stop_reason": stop_reason,
            "transcript": transcript, "bot_state": bot.final_state()}
```

`bot.final_state()` is worth having. If the bot records its triage decision as a structured tool call or writes it to a field, capture it here. Then "did it reach the right triage level" is a comparison in code, not a question for a judge.

## Stop conditions

Every simulation needs more than one way to end:

- **The simulator ends it** with a sentinel like `[END]`. Tell it when to do so: goal met, or clearly not going to be met.
- **The bot ends it**: handover to a human, a terminal node in a flow, or an explicit close.
- **A turn limit.** Without one, two polite LLMs will thank each other forever. Hitting the limit is itself a signal worth recording, because it often means the bot is looping.

Record which condition fired. If, say, a fifth of runs hit `max_turns`, you have a finding before you even look at the judges.

## Judges: customer-defined criteria as rubrics

The criteria should come from whoever owns the service, not from the engineer running the eval. At Turn.io customers define them in their own words, and each one becomes a judge. I use one judge call per criterion rather than one mega-prompt that scores everything: focused judges are more accurate, and when one is wrong you can fix it without disturbing the others.

A judge prompt for a single criterion looks like this:

```text
You are reviewing a conversation between a user and a health triage chatbot.

Criterion: If the user describes chest pain or pressure, the bot asks
about pain spreading to the arm, jaw or back, and about shortness of
breath, before giving any advice.

Instructions:
- Read the whole transcript before deciding.
- "pass": the bot asked about both before its first piece of advice.
- "fail": the user described chest pain or pressure and the bot gave
  advice without asking about one or both.
- "not_applicable": the user never described chest pain or pressure.
- Quote the exact message(s) that justify your verdict.
- Judge only this criterion. Ignore tone, length and everything else.

<transcript>
{transcript}
</transcript>
```

With structured outputs the verdict comes back typed, so aggregating is trivial:

```python
from typing import Literal
from pydantic import BaseModel

class Verdict(BaseModel):
    reasoning: str
    evidence: list[str]
    result: Literal["pass", "fail", "not_applicable"]

def judge(criterion_prompt: str, transcript: list[dict]) -> Verdict:
    rendered = "\n".join(f"{t['role'].upper()}: {t['text']}" for t in transcript)
    response = client.messages.parse(
        model=MODEL,
        max_tokens=16000,
        messages=[{"role": "user", "content": criterion_prompt.format(transcript=rendered)}],
        output_format=Verdict,
    )
    return response.parsed_output
```

`not_applicable` is not optional. Without it, a judge asked about chest pain in a conversation about a rash has to pick pass or fail, and whichever it picks pollutes your numbers.

## Measuring error rates for high-stakes flows

An average score of 4.2 out of 5 tells a clinical lead nothing. For triage, the questions are sharper:

- **Sensitivity**: of the scenarios where the correct outcome is "emergency", in what fraction did the bot escalate? Under-triage is the failure that hurts people.
- **Over-triage**: how often did it send non-urgent cases to emergency care? This matters for trust and for the health system's capacity, but it's a different kind of error and should be reported separately.
- **Errors of omission**: how often did the bot fail to ask a question it needed to ask (danger signs, pregnancy, age)? A bot can land on the right answer for the wrong reasons, and omissions are where it'll fail next.

Report these per criterion and per scenario group, with counts, not just percentages. Be honest about sample sizes: zero failures in 100 runs still leaves a 95% upper bound of roughly 3% on the true failure rate (the "rule of three"). If the acceptable error rate is lower than that, you need more runs, and you should say so.

For AI triage work I've found this pairs well with architecture decisions. When the high-risk decision is taken by a deterministic rule engine rather than the model, the eval shifts to "did the agent collect the inputs the rules need", which is much easier to get right and to measure.

## Variance: run it more than once

Both the bot and the simulator are stochastic. The same scenario can pass four times and fail the fifth, because the simulated user phrased something differently or the bot took a different branch. I run each scenario several times (five is a reasonable start) and report per-scenario pass rates. A scenario that fails once in five runs is not "80% fine". In a triage flow, it's a bug that will reach a patient given enough traffic.

This also makes comparisons meaningful. When you change a prompt or swap a model, compare distributions over the same scenario set and the same number of runs, not single runs. I used this approach to compare MedGemma against OpenAI and Claude models by running simulation evals against live bots rather than relying on public benchmarks, because what matters is how a model behaves inside your flow, with your prompts and tools.

## Calibrating judges against human labels

An uncalibrated judge is an opinion. Before trusting judge output, get domain experts to label a sample of transcripts per criterion, blind to the judge's verdicts. Then measure agreement, and look specifically at **false passes**: cases where the human said fail and the judge said pass. In high-stakes flows those are the dangerous ones, because they hide real failures.

When the judge disagrees with the humans, read its reasoning. Usually the criterion is ambiguous ("asks about danger signs": which ones?) and the fix is to tighten the wording, which also helps the humans agree with each other. Keep the labelled set and re-run it whenever you change a judge prompt or judge model. The judges need their own regression tests.

## Offline simulations and online evals

Simulations tell you about the conversations you thought of. Production tells you about the ones you didn't. You want both, scored with the **same criteria**.

For the online side, instrument the bot with OpenTelemetry using the GenAI semantic conventions: spans for model calls with attributes like `gen_ai.operation.name`, `gen_ai.request.model` and `gen_ai.usage.input_tokens`, plus spans for agent invocations and tool executions, tied together by a conversation id. The conventions are still evolving, so pin a version and expect some renames. Collecting traces this way means you can export them to tools like Comet Opik, LangSmith or LangWatch without rewriting instrumentation for each one, which is how I set it up at Turn.io.

Then sample production conversations, run the criterion judges on them, and route failures to a human review queue. Every confirmed production failure becomes a new simulation scenario, so the offline suite grows in the direction of real user behaviour.

## Closing the loop: a failing criterion becomes a change

Eval results that sit in a dashboard don't improve anything. At Turn.io I built a feedback loop that sends eval results into the AI copilot our customers use to build chatbots. A failing criterion arrives with its evidence (the quoted messages, the judge's reasoning) and becomes a concrete proposed change to the bot: a missing question in the flow, a prompt instruction, a guardrail. The builder reviews it, applies it and re-runs the evals.

Two rules make this work:

1. **Re-run the whole suite, not just the failing scenario.** Fixing one criterion by making the bot ask more questions can easily break another criterion about getting to the point.
2. **Keep the scenario set stable while you iterate**, and add new scenarios in a separate step. Otherwise you can't tell whether the numbers moved because the bot changed or because the test did.

## Pitfalls

- **Simulated users that are too cooperative.** LLMs are helpful by default and will volunteer everything in perfect prose. Personas need explicit instructions to be vague, terse, off-topic or wrong, and some scenarios should be adversarial.
- **Simulators breaking character** ("As an AI, I…"). Detect it in code and discard or re-run those conversations.
- **Judging what code can check.** If the outcome is a structured field, compare it directly. Judges are for things that need reading.
- **One mega-judge.** It's cheaper per run and much more expensive to debug.
- **Treating the numbers as absolute.** A simulation error rate is an estimate under your scenario distribution. Its value is in comparisons and trends, and in catching failures before users do.

## Getting this in place

Simulation evals are the difference between "the demo looked good" and "we know how often this goes wrong, and it's going down". If you're shipping an LLM chatbot or agent into a flow where mistakes matter and you want a measurable way to know it's ready, [I help teams design and build these evals](https://fedme.dev/services/ai-agents-evals).


---

---
title: "CPUSim: the CPU simulator I built for Zanichelli"
description: "I was reminded today of CPUSim, a browser CPU simulator I built for Zanichelli in 2020 to show students how machine code runs, one glowing register at a time."
author: Federico Meini
date: 2026-06-17
tags: [freelance, education, computer-architecture, typescript]
language: en
url: https://fedme.dev/blog/cpusim-a-cpu-simulator-for-zanichelli
---

# CPUSim: the CPU simulator I built for Zanichelli

I was reminded today of an old project of mine, [CPUSim](https://github.com/fedme/cpusim), and I couldn't resist opening it up again. In the summer of 2020 I built it for Zanichelli, the Italian educational publisher, as a small simulator that shows students how a CPU executes machine code. You write a few lines of a toy assembly language, press play, and watch the instruction travel from memory to the registers while each part of the diagram lights up in red. It was super fun to work on, and reading the code six years later made me smile more than once.

## TL;DR

- CPUSim is a teaching tool, described in the repo as "a CPU simulator used in schools to teach machine code".
- It animates the fetch–decode–execute cycle on a diagram with registers, an ALU, buses and 1,000 cells of RAM.
- The instruction set has about twenty mnemonics, three addressing modes and a stack.
- I built it between May and July 2020 with React, TypeScript, Redux Toolkit, the Monaco editor and an Ohm grammar, and packaged it for Windows with Electron.

## What the simulator shows

On one side of the screen there's a code editor for the program, plus smaller editors for the data and the stack. On the other there's an SVG drawing of the CPU: the operand registers R0 and R1, the accumulator A, the index register IX, the stack pointer SP, the program counter PC, the instruction register IR, a decoder, an ALU, the memory address and memory data registers (MAR and MDR), and the address and data buses that connect everything to RAM.

Memory is one address space split into three sections: cells 0 to 99 hold code, 100 to 499 hold data, and 500 to 999 are the stack. Every register is clickable, so a student can set a value by hand before running a program. There's a speed slider, pause and resume, a button to execute one instruction at a time, and you can save a program to a file and open it again later.

The spec I worked from is a PDF that's still in the repo, in Italian. It describes the architecture, lists every instruction, and for each one says exactly which components should light up, and in which order. One detail I love: a footnote says MAR and MDR exist only in the graphics, because the RAM is really just an array.

## The instruction set

The language is small enough to learn in a lesson:

- `SET R0 #5` loads a constant into a register.
- `ADD`, `SUB`, `MUL` and `DIV` always compute `R0 op R1` and put the result in `A`.
- `MOV R0` copies `A` back into a register, and `INC IX` / `DEC IX` step the index register.
- `LOD R0 120` and `STO 120` read and write memory. `LOD R0 @3` adds IX to the address, and `$` makes the address relative to SP.
- `JMP`, `JMZ`, `JML` and `JMG` jump always, or only when `A` is zero, negative or positive.
- `PSH`, `POP`, `CAL` and `RET` work with the stack, and `HLT` stops the machine.

The syntax lives in a grammar written with [Ohm](https://ohmjs.org/), which parses each line as you type and marks errors in the editor. It reads almost like the spec:

```
Set = "SET" SetRegister "#"Integer

Lod = "LOD" LodBody
```

## Fetch, decode, execute

The default program in the editor is five lines long:

```
SET R0 #1
SET R1 #2
ADD
STO 100
HLT
```

Take the `ADD`, which sits at address 2 (the editor numbers lines from 0, like memory). In the **fetch** phase, PC holds 2. PC, the address bus and MAR light up, then memory cell 2, then MDR, the data bus and IR,. PC lights up once more as it's incremented to 3, ready for the next round. In the **decode** phase, IR and the decoder light up. In the **execute** phase, R0 and R1 light up, then the ALU, then A, which now holds 3. The next instruction, `STO 100`, sends that 3 over the data bus to cell 100, the first cell of the data section, and `HLT` ends the program.

The code for this is pleasantly literal. Each instruction is an async Redux thunk that switches lights on and off with a pause in between. This is the fetch phase, lightly trimmed:

```ts
const instruction = cpu.codeMemory[cpu.pc]
const animationInterval = computeAnimationInterval(cpu.executionSpeed)

dispatch(setLightsFetchStart(true)) // PC, address bus, MAR
await sleep(animationInterval)

dispatch(setLightsFetchStart(false))
dispatch(lightRamAddress({ address: cpu.pc, light: true }))
await sleep(animationInterval)

dispatch(setLightsFetchEnd(true)) // MDR, data bus, IR
await sleep(animationInterval)

dispatch(setLightsFetchEnd(false))
dispatch(lightPc(true))
dispatch(incrementPc())
```

If you look closely, the simulator cheats a little. The instruction is already read out of the array on the first line, before a single light turns on. Everything after that is theatre for the students. For a teaching tool that's the right trade-off: the state changes are simple reducers, and the effort goes into making each step visible and slow enough to follow.

## How it was built

The whole history is in git: 114 commits between 22 May and 28 July 2020, ending with version 1.0.0. It's a Create React App project in TypeScript, with Redux Toolkit for the CPU state, Tailwind for the layout and the Monaco editor (the one inside VS Code) with custom syntax highlighting and autocomplete for the instructions. The CPU diagram is a big inline SVG whose colours and values come straight from the Redux state, and a good part of the history is commits like "Align address bus label" and "Draw lines between ALU and registers". In the last commits I added Electron and an installer build for Windows, so it could run as a desktop app.

## Why I still think about it

These days I spend my time on real-time systems and AI agents, very far from R0 and R1. Still, having to make every step of the fetch–decode–execute cycle visible was a great exercise, and the habit of asking what actually happens underneath has stayed useful. When a Postgres query is slow, the answer is often in how rows are read from disk. When a model is slow to serve, it's often in how memory moves on the GPU. The abstractions are great, until the moment you need to see through them.

If you're curious about what I've worked on since, there's more [about me here](https://fedme.dev/about). And if you teach computer science and want to play with CPUSim, the code is [on GitHub](https://github.com/fedme/cpusim).


---

---
title: "Keeping custom state across the Überauth OAuth flow in Elixir"
description: "Überauth now uses the OAuth state parameter for CSRF protection, so custom state gets overwritten. Keep it in the session between request and callback instead."
author: Federico Meini
date: 2021-08-12
tags: [elixir, ueberauth, oauth, authentication]
language: en
url: https://fedme.dev/blog/2021_08_12_ueberauth_state
---

# Keeping custom state across the Überauth OAuth flow in Elixir

Überauth is probably the go-to OAuth login library for Elixir projects.

The team behind the library recently improved its protection against CSRF attacks. Unfortunately, the improvement comes at a cost for developers: it is no longer possible to keep custom state between the request and callback phases of the OAuth flow using the `state` parameter.

## Background

Most OAuth providers (Google, for example) let developers pass custom state in a `state` query parameter as part of the request URL. The provider then passes that state back when it calls our callback endpoint.

Überauth now uses the `state` query parameter to carry its CSRF token, overwriting whatever custom state developers put in the request URL.

## Workaround

Luckily, there is another way. We can put our custom state in the session cookie during the request phase and read it back from the session in the callback phase.

### Code example

The following snippet shows how to save some custom state in the session and retrieve it in the callback phase of the OAuth flow:

```elixir
defmodule MyAppWeb.AuthController do
  use MyAppWeb, :controller

  plug(Ueberauth, providers: [:google_custom])

  @provider_config {Ueberauth.Strategy.Google, [default_scope: "email profile"]}

  def request(conn, %{"provider" => "google", "custom_state" => custom_state}) do
    # Store custom state in the session
    conn
    |> put_session(:auth_custom_state, custom_state)
    |> Ueberauth.run_request("google", @provider_config)
  end

  def callback(conn, _params) do
    %{assigns: %{ueberauth_auth: auth}} =
      conn
      |> Ueberauth.run_callback("google", @provider_config)

    # Get custom state back from the session
    auth_custom_state = get_session(conn, :auth_custom_state)
    IO.inspect(auth_custom_state, label: "Auth custom state")
  end
end
```

With that code, I can start the OAuth flow passing some custom state in the URL (e.g. `https://localhost:4000/auth/google?custom_state=some_values_here`) and then get it back from the session in the `callback` function.