# Real-time and distributed systems engineering (Elixir, OTP, Phoenix)

> Freelance Elixir and distributed systems engineer. High-throughput messaging pipelines, backpressure, rate limiting, job systems on Oban, Phoenix LiveView, and Kubernetes.

URL: https://fedme.dev/services/realtime-systems
Provided by: Federico Meini (federico.meini@gmail.com)

## Pipelines that bend under load instead of breaking.

I build high-throughput, real-time backends in Elixir and OTP. They include send engines that push millions of messages, job systems that never lose work, and live interfaces that update as things happen.

## What you get

- Throughput you can reason about, with backpressure, bounded queues and rate limits designed in rather than bolted on.
- Durable background work (retries, scheduling, uniqueness, rate-limited queues) with Oban.
- Live, collaborative interfaces with Phoenix LiveView and PubSub.
- Observability that explains incidents: telemetry, Prometheus metrics, Grafana dashboards, useful alerts.

## The problem with "it worked in staging"

Real-time systems rarely fail at average load. They fail when a campaign goes out and ten thousand people reply in the same minute, when a downstream API starts throttling, or when a retry storm doubles the traffic you were already struggling with.

At Turn.io I built the **bulk messaging engine** used for national public-health campaigns, including one that reached 14.7 million people, and for national vaccination reminders. It combines rate limiting, backpressure, delivery tracking and retries. I also built the **job and data pipeline infrastructure on Oban Pro** and an **IFTTT-style trigger system with a rules engine**, all in Elixir and Phoenix on Google Cloud and Kubernetes.

## What I build

**High-throughput pipelines.** Demand-driven stages (GenStage, Broadway) where consumers pull work, queues stay bounded and throughput converges on whatever the slowest dependency allows.

**Rate limiting and retries done properly.** Token buckets per sender or tenant, retries with exponential backoff and jitter, idempotency keys so a retry never sends the same message twice.

**Durable job systems.** Oban for work that must not be lost: scheduled sends, webhooks to third parties, data exports, cron-style automations, with uniqueness and per-queue rate limits.

**Rules and trigger engines.** Event-driven automations ("when X happens and Y is true, do Z") that non-engineers can configure safely.

**Live interfaces.** Phoenix LiveView and PubSub for dashboards and inboxes that update in real time without a separate frontend team.

## How I approach it

1. **Model the load.** Averages, peaks, bursts, and the rate limits you don't control.
2. **Make every queue bounded and every stage demand-driven.** Decide up front what happens when you are overloaded: slow down, shed, or defer.
3. **Instrument before optimising.** Telemetry events on every stage, so you can see where time goes.
4. **Load test with realistic traffic shapes,** including the failure modes.

## Related

- [WhatsApp Business Platform engineering](https://fedme.dev/services/whatsapp-business-platform)
- [PostgreSQL and Elasticsearch at scale](https://fedme.dev/services/postgres-elasticsearch): where all those messages end up.

## Ways to engage

- **Scaling review**: One to two weeks profiling a system under load, finding the real bottlenecks and writing a prioritised plan.
- **Build**: Design and build a pipeline, job system or real-time feature end to end, with load tests before launch.
- **Embedded senior engineer**: Part-time in your team to lead a scaling effort and level up the team on OTP.

## Stack

Elixir, Erlang/OTP, Phoenix & LiveView, GenStage / Broadway, Oban Pro, PostgreSQL, Google Cloud, Kubernetes, Prometheus & Grafana

## FAQ

### Why Elixir for real-time systems?

The BEAM gives you millions of cheap, isolated processes, supervision trees that restart what fails, and message passing that maps naturally onto messaging workloads. It makes per-user ordering, backpressure and graceful degradation much easier to get right than in most runtimes.

### What is backpressure and why does it matter?

Backpressure means that downstream stages tell upstream stages how much work they can take. Without it, a traffic spike fills memory and queues until something falls over. With it, the system slows down gracefully and recovers on its own.

### Can you work on a system that isn't in Elixir?

Yes. The ideas carry over (bounded queues, demand, idempotency, rate limiting), and I also write Python and TypeScript. I'm most productive in Elixir, though.

### Do you do load testing?

Yes. I load test before launch and reproduce production-shaped traffic, including bursts, slow downstream APIs and retries, because that is where the surprises are.

## Contact

Email federico.meini@gmail.com · https://fedme.dev/contact