Reliability Topic

Reliability

Reliable systems are built from unreliable parts. These posts cover the patterns that absorb failure rather than amplify it: retry budgets, idempotency keys, circuit breakers, webhook delivery, and what it actually takes to ship something that does not fall over at 3am.

Backend

Posted on May 28, 2026

Why Your Distributed Lock Doesn't Lock

Distributed locks don't provide mutual exclusion. Fencing tokens, GC pauses, clock drift, and why the lock you wrote is actually a polite hint at best.

The Thundering Herd Problem

Cache stampedes, retry storms, reconnect floods: three failure modes with the same root cause. Synchronized behavior under load amplifies failures every time.

Webhook Reliability: The Lost Art

Webhooks break predictably: duplicate events, missed deliveries, retry storms. Here is what it actually takes to build receivers that hold up in production.

Rate Limiting Is Harder Than It Looks

Token bucket, sliding window, fixed counter: rate limiting algorithms all sound simple until you actually implement them correctly across distributed systems.

Monitoring Is Not a Dashboard

Real monitoring is not a Grafana dashboard. It is knowing which questions to ask, which signals answer them, and what to do when the answer is unexpected.

The Deploy That Took Down Friday

Friday deploys have a reputation for a reason. Here's why they go wrong, what guardrails actually help, and when it's okay to ship on a Friday anyway.