Serving MedGemma 27B on Modal: FP8, vLLM sleep mode and 21-second cold starts

You can serve a 27B-parameter medical model on serverless GPUs, pay only while it’s answering, and still get a cold start of about 21 seconds. The recipe that worked for me:

1. Quantise MedGemma 27B to FP8 ahead of time.
2. Serve it with vLLM on Modal.
3. Put vLLM to sleep at level 2 before Modal takes a GPU memory snapshot, so the snapshot holds about 1 GiB of runtime state instead of 32 GiB of weights.
4. On wake, reload the weights from a cached volume.

Getting there took two vLLM bug fixes, a custom tool-call parser and a few environment variables that I would not have guessed.

I built this at Turn.io, where we run AI agents for health services on WhatsApp. We wanted to know whether an open medical model could replace frontier models for some of that traffic. To find out, we ran it against the same simulation evals we use for production bots rather than public benchmarks. The quantised model is public: turnio/medgemma-27b-text-it-FP8-Dynamic. This post covers the serving side.

TL;DR

• FP8 on disk, not at runtime. Quantising BF16 weights to FP8 when vLLM loads them runs out of memory while the snapshot is being created. So I quantise once with llm-compressor (FP8_DYNAMIC, no calibration data), which halves the weights from ~54 GiB to ~27 GiB, and publish the result.
• Snapshot the process, not the weights. vLLM sleep level 2 discards the weights before Modal snapshots GPU memory. The snapshot shrank from ~32 GiB to ~1 GiB, and cold starts fell from ~100 s to ~21–24 s.
• Level 2 needed vLLM 0.16.0 plus a one-line patch. One bug corrupted FP8 weights on reload; the other stopped vLLM from freeing the weights at all.
• Gemma 3 doesn’t speak OpenAI tool calls. A small vLLM parser plugin and chat template translate its tool_code blocks into standard tool_calls, including when streaming.
• Size the GPU for context, not for weights. The model fits on an L40S, but the full 131K context only fits on an H100. That roughly doubles the warm-hour cost, from about $1.95 to $3.95.

Why self-host a medical model at all

Frontier APIs are the right default for most agents. They are good, cheap per call and someone else keeps them running. Our reasons to try an open model were specific:

• MedGemma is trained for medical text.
• We wanted a model we control end to end.
• We wanted to compare it with the OpenAI and Claude models our customers already use, on the conversations they actually have.

That last point matters most. We didn’t decide on MedQA scores; we ran the same multi-turn simulation evals against live bots with each model behind them. The results now feed a routing policy between models.

The catch with a 27B model is cost. An always-on GPU for traffic that comes in bursts is money burned while nobody is talking. Modal lets containers scale to zero and bills by the second, so the whole design question became: how fast can a scaled-to-zero container answer its first request?

The shape of the deployment

The app is one Modal class that starts vllm serve as a subprocess and exposes its port as a web endpoint. That gives clients an OpenAI-compatible /v1/chat/completions API, and any OpenAI SDK works by changing base_url.

@app.cls(
    image=vllm_image,
    gpu="H100",
    scaledown_window=10 * MINUTES,  # scale to zero after 10 min idle
    min_containers=0,
    max_containers=3,               # cost safety cap
    volumes={
        "/root/.cache/huggingface": hf_cache_vol,
        "/root/.cache/vllm": vllm_cache_vol,
    },
    secrets=[modal.Secret.from_name("huggingface-secret"),
             modal.Secret.from_name("medgemma-api-key")],
    enable_memory_snapshot=True,
    experimental_options={"enable_gpu_snapshot": True},
)
@modal.concurrent(max_inputs=16)  # small queue above vLLM's max_num_seqs=8
class VllmServer:
    ...

A few choices worth calling out:

• The weights live on a Modal volume (the Hugging Face cache), not in the image. Images stay small, and the weights are downloaded once.
• The API key is enforced by vLLM itself (--api-key), read from a Modal secret. The endpoint is public, so this is not optional.
• max_containers=3 is a cost cap, not a performance setting. A runaway client should hit a queue, not your credit card.

Step 1: quantise to FP8 yourself

The official checkpoint is BF16, about 54 GiB. My first attempt used vLLM’s runtime FP8 quantisation (--quantization fp8). It ran out of memory during snapshot creation, because the BF16 and FP8 copies briefly coexist.

The fix is to store FP8 weights on disk. I briefly used a community FP8 checkpoint, then replaced it with our own. For a medical model I wanted to know exactly how the weights were produced, and to be able to re-run it for new MedGemma releases. Quantisation is a one-shot Modal function with llm-compressor:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained("google/medgemma-27b-text-it", dtype="auto")

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",   # per-channel weight scales, per-token dynamic activations
    ignore=["lm_head"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(save_dir, save_compressed=True)

FP8_DYNAMIC needs no calibration dataset. Weight scales are computed per output channel, and activations are quantised per token at inference time. The result is about 27 GiB in the compressed-tensors format. It fits a 48 GB L40S with room for KV cache, and vLLM picks it up with --dtype auto.

Per-channel scales are slightly more precise than the per-tensor scales of the older “native” FP8 format. For a model this size the difference is negligible in practice. Our real quality check was the simulation evals anyway, not perplexity.

Step 2: make cold starts fast with GPU snapshots

Without snapshots, a fresh container downloads nothing (the volume is warm) but still spends minutes loading weights, compiling and warming up. It took about 5 minutes end to end. Modal’s GPU memory snapshots let you do that work once, freeze the process, and restore it on every cold start.

The obvious approach is to snapshot the whole loaded model. It works, but the snapshot is huge. vLLM’s sleep mode is designed for exactly this: POST /sleep prepares the engine to be frozen, and POST /wake_up brings it back. It has two levels:

	Level 1	Level 2
On sleep	Weights copied GPU → CPU (and so into the snapshot)	Weights discarded
On wake	CPU → GPU copy	Reload weights from disk
Snapshot size	~32 GiB	~1 GiB
Snapshot restore	~95 s	~8 s
Wake	~2.5 s	~13–16 s
Total cold start	~100 s	~21–24 s

Level 2 wakes more slowly because it has to read 27 GiB from the volume. But restoring a 32 GiB snapshot is so much slower that the net saving is almost 80 seconds. The bottleneck moves from snapshot I/O to volume I/O, and the volume is faster.

The lifecycle maps neatly onto Modal’s two kinds of @modal.enter hooks:

@modal.enter(snap=True)          # runs once; Modal snapshots right after it
def start(self):
    self.vllm_proc = subprocess.Popen(cmd)
    wait_ready(self.vllm_proc)
    warmup()                     # one tiny request to trigger lazy init
    vllm_sleep(level=2)          # drop the weights before the snapshot

@modal.enter(snap=False)         # runs on every restore
def wake(self):
    wake_up()                    # remap GPU memory, no weights yet
    reload_weights()             # POST /collective_rpc {"method": "reload_weights"}
    reset_prefix_cache()         # don't serve stale prefix-cache entries
    wait_ready(self.vllm_proc)

Two flags helped keep the snapshot small:

• --enforce-eager skips CUDA graph capture. That removes 10–20 GiB of GPU memory from the snapshot and 10–20 s of init time, at the price of slightly higher per-token latency, which is fine for a chatbot.
• --gpu-memory-utilization 0.85, which is also a ceiling. The layerwise weight reload after level 2 sleep needs temporary GPU buffer space, so going higher breaks the wake.

The two vLLM bugs behind level 2

On paper, level 2 is one argument. In practice my first attempt produced garbage output after waking, and I reverted to level 1 the same afternoon. Two separate bugs were involved.

Bug 1: quantised weights reloaded in the wrong layout

When vLLM first loads a quantised model, it runs process_weights_after_loading() on each layer, which repacks checkpoint tensors into the layout the kernels want. The old reload_weights() path skipped that step. It loaded checkpoint-format tensors into kernel-format parameters, and for compressed-tensors FP8 this didn’t raise an error. It just produced nonsense. This is vllm#28606.

The fix is layerwise reloading, which re-runs the post-processing per layer (vllm#32133). It shipped in vLLM 0.16.0, so step one was upgrading.

Bug 2: an and where a comma should be

On 0.16.0, level 2 sleep reported that it freed about 5 GiB and that 27.88 GiB was still in use. The weights weren’t being released at all. The cause turned out to be one character in gpu_worker.py:

# vLLM 0.16.0
with self.mem_allocator.use_memory_pool(tag="weights"
    ) and set_current_vllm_config(self.vllm_config):

# fixed upstream in vllm#32947
with self.mem_allocator.use_memory_pool(tag="weights"
    ), set_current_vllm_config(self.vllm_config):

a and b evaluates to b when a is truthy, so only the second context manager is entered. The memory pool that tags weight allocations never becomes active. Sleep can’t free what the allocator never tracked.

The fix (vllm#32947) was merged upstream after 0.16.0 was released, so I patch it at image build time:

.run_commands(
    "sed -i 's/) and set_current_vllm_config(/), set_current_vllm_config(/' "
    "/usr/local/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py",
)

With the patch, sleep freed 34.48 GiB and left 0.9 GiB in use. Snapshot creation went from about 2 minutes to 15 seconds. The KV cache also grew from 13,680 to 17,872 tokens on the same GPU, a free bonus from tracking memory properly.

Smaller things that broke snapshots

None of these took long to fix once found, but every one of them cost a deploy cycle to find:

• Hugging Face Xet. Newer huggingface_hub versions download through the Xet backend, which writes files into the cache volume that are gone by the time Modal restores the snapshot. The restore failed with vfs.CompleteRestore errors. HF_HUB_DISABLE_XET=1 fixed it. The older HF_HUB_ENABLE_HF_TRANSFER=0 no longer controls this.
• fastsafetensors. I tried --load-format fastsafetensors to speed up the weight reload with GPU Direct Storage. Modal volumes don’t support GDS, so it went back out.
• cuBLAS mismatch. vLLM 0.16.0’s torch bundles cuBLAS 12.8, which caused CUBLAS_STATUS_INVALID_VALUE on a CUDA 12.9 base image. Pinning nvidia-cublas-cu12==12.9.1.4 with --no-deps fixed it.
• Snapshot-safe compilation. Set TORCHINDUCTOR_COMPILE_THREADS=1 (required for memory snapshots), and TORCH_CUDA_ARCH_LIST to the GPU you actually run so you don’t compile for every architecture.
• NCCL heartbeat noise. /sleep shuts down a TCPStore that the NCCL heartbeat thread keeps pinging, which floods the logs with “broken pipe” warnings. Setting TORCH_NCCL_COORD_CHECK_MILSEC to ten minutes makes the thread sleep through the wake cycle.

Tool calling: teaching vLLM to read Gemma 3

Our agents use tools, so a model without OpenAI-style tool_calls is not usable behind them. Gemma 3, and therefore MedGemma, was trained to call tools with Python syntax inside a fenced block:

```tool_code
print(calculate_bmi(height_cm=180, weight_kg=75))
```

vLLM doesn’t parse that, so the deployment ships two extra files, both loaded through vLLM’s own extension points:

• A chat template that renders the request’s tools as Python function signatures with docstrings. It tells the model to use tool_code blocks for calls and plain text for everything else.
• A tool parser plugin (--tool-parser-plugin, --tool-call-parser medgemma). It finds tool_code blocks, strips the print(...) wrapper, parses the keyword arguments and returns standard ToolCall objects.

The parsing is mostly regex plus a small character-by-character scanner for arguments, because values can contain commas inside quotes or nested parentheses. The interesting parts are the quirks:

• Plain text wrapped as a tool call. Sometimes the model “replies” with print("Sure, here's…") inside a tool_code block. The parser recognises a bare string literal and returns it as normal content instead of a broken tool call.
• Leaked thinking tokens. Gemma 3 occasionally emits its internal thinking markers (<unused94>…<unused95>), sometimes only half of the pair. They’re stripped in both streaming and non-streaming modes.
• Markers split across stream chunks. In streaming mode ``` can arrive in one delta and tool_code in the next. The parser holds back any suffix that could be the start of the marker, so clients never see stray backticks before a tool call is detected.

The parser has unit tests and a streaming integration test, with CI running them on every push. Streaming parsers are exactly the kind of code that works in the demo and fails on the third token boundary.

Sizing: the GPU is for the context, not the weights

The FP8 model fits comfortably on an L40S (48 GB). For months that’s where it ran, with the context capped at 8K tokens, at roughly half the price of an H100.

The cap existed because of the KV cache, not the weights. Gemma 3 uses hybrid attention: 10 global layers attend over the full sequence, while 52 sliding-window layers only keep roughly a window’s worth of tokens. A full 131K-token sequence needs about 13.7 GiB of KV cache, about 10 GiB of it for the 10 global layers. The L40S had around 9 GiB left for KV cache after weights and buffers. The H100 has around 35 GiB, which fits about three full-length requests at once.

Moving to the H100 took warm-hour cost from about $1.95 to $3.95. With scale-to-zero, that’s the cost per hour the container is actually awake, not per hour of the month. Serving settings for the H100:

--max-model-len 131072          # full context
--max-num-seqs 8                # concurrent sequences
--max-num-batched-tokens 8192   # per engine step; chunked prefill splits longer prompts
--gpu-memory-utilization 0.85   # ceiling set by the layerwise reload buffer

One unglamorous middleware

After launch, some callers got 404s without ever reaching the model. One client had set its base URL to the full completions URL, so the SDK appended the path again: /v1/chat/completions/chat/completions. Another probed /v1/health, while vLLM serves health at /health.

Rather than chase every integration, I added a 20-line ASGI middleware, loaded with vLLM’s --middleware flag, that rewrites those two paths. It runs before vLLM’s auth middleware, which only guards /v1 paths, so the rewrite doesn’t open anything up.

It’s not elegant, but it’s honest about how clients behave in the wild.

What I’d tell someone doing this next

• Measure every phase of the cold start separately. I added [cold-start] timing logs to each wake step (restore, wake, reload, prefix-cache reset, ready). Without them I would have optimised the wrong thing, because the snapshot restore dominated and not the model load.
• Read the sleep log line. “Sleep mode freed X GiB, Y GiB still in use” is the most useful number in the whole setup. If Y is large, your snapshot is carrying weights you think you dropped.
• Own your quantised checkpoint. It’s a one-off job, it’s reproducible, and it removes a third party from the supply chain of a medical model.
• Decide on your own evals, not on benchmarks. A model that is cheap to host and good on MedQA can still be the wrong one for your conversations. Simulation evals on real bots answered the question benchmarks couldn’t.

If you want an open model behind your agents, with cold starts, tool calling and evals handled properly, that’s the kind of work I do.