DeviAI
All work

Source available · v1.0.0APIs & MicroservicesStudio infrastructure — not client work

A WhatsApp webhook that survives being killed mid-delivery.

Meta sends the same message twice, in a shape nobody wants to parse, to a URL anyone can find. This sits in front of your app and turns all of that into one clean event.

github.com/harinazrekar/whatsapp-normalizer
docker compose up · whatsapp-normalizerrecorded run
47s · no audio

Not an animation of an idea. Every log line, status code, id and figure on screen was printed by the stack while it was running — the handshake, the rejections, the kill, the reclaim.

197

tests passing

98%

statement coverage

0

events lost under kill -9

22s

to reclaim and redeliver

9ms

webhook acknowledged

Free

for noncommercial use, v1.0.0

01What goes wrong

If you take customer messages on WhatsApp, Meta POSTs them to a URL of yours. That URL is public, so anyone who finds it can post to it. Meta retries, so the same message arrives more than once. It expects an answer in milliseconds, and if your endpoint is slow or errors, it backs off and eventually stops delivering to you at all. And the moment your own code takes an event out of a queue to forward it, a crash means the event is simply gone — no error, no trace, a customer who never got a reply.

02How it works

One small service between Meta and your app. It checks the signature on the raw request bytes before parsing anything, flattens Meta’s nested payload into one flat event, drops repeats by message id, and writes the event to a Redis Stream — not a list, because a list pops the entry and forgets it. A separate worker delivers it to you, retries with backoff, and dead-letters what it can’t deliver. Because the stream keeps an entry pending until it is acknowledged, killing the worker mid-delivery loses nothing: another one reclaims the entry and finishes the job.

Built with

FastAPI

Answers Meta in single-digit milliseconds and does the work behind it

Redis Streams

Consumer groups keep unacknowledged entries recoverable

httpx

Async delivery to the downstream, with a real timeout

Prometheus

Queue depth, delivery outcomes, retries, dead letters

Docker Compose

Split networks; Redis publishes no port and has no route off the host

Caddy

Automatic HTTPS — WhatsApp will not call an endpoint without a valid certificate

pytest

197 tests, including the three bugs only real Redis surfaced

03What the recording shows

Five things happen, and one of them is a crash.

00:05

Two forged requests, rejected

A POST with no signature and a POST with a wrong one. Both get 403 before the body is parsed — the signature is checked against the raw bytes, because re-serialising JSON changes them and breaks the comparison.

00:09

The real one, acknowledged in 9ms

A correctly signed delivery returns 200 {"received":1,"queued":1,"duplicates":0} and is written to the stream. Meta gets its answer immediately; the actual forwarding happens behind it.

00:15

The same message again, dropped

Meta redelivers. The service recognises the message id and answers duplicates:1. Your system never sees the second copy.

00:21

kill -9 while an event is in flight

The worker is killed hard while it is mid-POST to the downstream. Nothing acknowledged the event; Redis reports one entry still pending.

00:26

A second worker reclaims it

entries_reclaimed count=1, then delivered status=200. Twenty-two seconds after the kill, the event arrived — the downstream got 3 requests carrying 2 distinct ids, which is the correct answer, not a bug.

04Engineering decisions

The calls that decide whether an event ever arrives.

01

A stream with a consumer group, not a list

BLPOP removes the entry the instant it is read. If the process dies between that read and a successful delivery, the event is gone and nothing anywhere records that it existed. A consumer group leaves the entry pending until it is explicitly acknowledged, which is the only reason a kill is survivable at all.

02

The delivery claim has two states, not one

A single “this is being handled” flag is ambiguous after a crash: was the downstream reached or not? Delivering again risks a duplicate; skipping risks losing it. A short-lived in-flight claim and a long-lived completed marker tell those two cases apart — only a completed delivery retires an entry without re-sending.

03

The claim is held across the backoff sleep

Releasing it before sleeping lets a reclaiming worker start a parallel attempt on an entry this one still holds. Every retry round would then double the copies in flight. Found by running it, not by reading it.

04

The worker serves its own metrics

Prometheus counters are process-local. Delivery, retry, dead-letter and reclaim counts all live in the worker, which served no HTTP at all — so six metrics were incremented into a registry nothing could scrape, and an alert on failed deliveries could never have fired. Giving it a port fixed the metrics and gave the process its first liveness probe.

05

The reclaim timing is validated, not commented

If the reclaim window is shorter than the worst-case delivery attempt, a live attempt gets reclaimed by another worker and the customer is messaged twice. Three separate settings control that relationship, so the service refuses to start when they contradict each other rather than trusting a comment nobody reads.

What it has not done yet

  • No live Meta payload has hit it yet — that needs a Meta developer app, and everything up to that boundary is exercised against real Redis and real Docker.
  • The blocking read path cannot be covered by the test suite: fakeredis does not implement blocking XREADGROUP. It is verified by running the stack instead.

We list this because the alternative is letting you assume otherwise. The code, the tests and the deployment guide are all in the repository — you can run the same recording yourself.

PolyForm Noncommercial — free to read, run and modify for anything noncommercial. Running it inside a business needs a licence, which takes one conversation.