Federico MeiniReal-time systems, AI agents & more
IT
Work with me(opens in a new tab)

Real-time & distributed systems

Pipelines that bend under load instead of breaking.

I build high-throughput, real-time backends in Elixir and OTP. They include send engines that push millions of messages, job systems that never lose work, and live interfaces that update as things happen.

Talk about your project (opens in a new tab)
bulk-send pipeline0 msg/sbuffer 0dropped 0retries 0p95 –
Try it. Push the producer rate past what the workers and the rate limit can handle, then switch backpressure off and watch what happens.

The problem with “it worked in staging”

Real-time systems rarely fail at average load. They fail when a campaign goes out and ten thousand people reply in the same minute, when a downstream API starts throttling, or when a retry storm doubles the traffic you were already struggling with.

At Turn.io I built the bulk messaging engine used for national public-health campaigns, including one that reached 14.7 million people, and for national vaccination reminders. It combines rate limiting, backpressure, delivery tracking and retries. I also built the job and data pipeline infrastructure on Oban Pro and an IFTTT-style trigger system with a rules engine, all in Elixir and Phoenix on Google Cloud and Kubernetes.

What I build

High-throughput pipelines. Demand-driven stages (GenStage, Broadway) where consumers pull work, queues stay bounded and throughput converges on whatever the slowest dependency allows.

Rate limiting and retries done properly. Token buckets per sender or tenant, retries with exponential backoff and jitter, idempotency keys so a retry never sends the same message twice.

Durable job systems. Oban for work that must not be lost: scheduled sends, webhooks to third parties, data exports, cron-style automations, with uniqueness and per-queue rate limits.

Rules and trigger engines. Event-driven automations (“when X happens and Y is true, do Z”) that non-engineers can configure safely.

Live interfaces. Phoenix LiveView and PubSub for dashboards and inboxes that update in real time without a separate frontend team.

How I approach it

  1. Model the load. Averages, peaks, bursts, and the rate limits you don’t control.
  2. Make every queue bounded and every stage demand-driven. Decide up front what happens when you are overloaded: slow down, shed, or defer.
  3. Instrument before optimising. Telemetry events on every stage, so you can see where time goes.
  4. Load test with realistic traffic shapes, including the failure modes.

Ways to engage

01

Scaling review

One to two weeks profiling a system under load, finding the real bottlenecks and writing a prioritised plan.

02

Build

Design and build a pipeline, job system or real-time feature end to end, with load tests before launch.

03

Embedded senior engineer

Part-time in your team to lead a scaling effort and level up the team on OTP.

Questions

Why Elixir for real-time systems?

The BEAM gives you millions of cheap, isolated processes, supervision trees that restart what fails, and message passing that maps naturally onto messaging workloads. It makes per-user ordering, backpressure and graceful degradation much easier to get right than in most runtimes.

What is backpressure and why does it matter?

Backpressure means that downstream stages tell upstream stages how much work they can take. Without it, a traffic spike fills memory and queues until something falls over. With it, the system slows down gracefully and recovers on its own.

Can you work on a system that isn't in Elixir?

Yes. The ideas carry over (bounded queues, demand, idempotency, rate limiting), and I also write Python and TypeScript. I'm most productive in Elixir, though.

Do you do load testing?

Yes. I load test before launch and reproduce production-shaped traffic, including bursts, slow downstream APIs and retries, because that is where the surprises are.

Related writing

Other services