arrow_back Return to Deep Dives

Post-Mortem:
The Caching Nightmareit burns!

• Severity: HIGH

Abstract

A simple attempt to speed up the dashboard endpoint using Redis turned into a distributed locking catastrophe. We implemented caching without fully understanding the invalidation lifecycle across multiple container instances, leading to phantom data, race conditions, and an eventual cascading failure of the entire staging environment. Solution: Embrace simplicity, avoid distributed locks unless strictly necessary, and use a TTL strategy that makes sense.

The Hypothesis

"If we cache the expensive aggregation query in Redis for 5 minutes, dashboard load times will drop from 3s to <100ms, and everyone will be happy."

The Reality

We cached it. But when a user updated their data, Instance A invalidated the cache while Instance B simultaneously repopulated it with stale data.

Oops.

The Initial Implementation

The goal was straightforward. The dashboard was struggling under the weight of a horrific set of JOINs. I figured a quick Redis implementation would act as a band-aid until we could properly optimize the database schema.

I wrapped the service call in a basic caching decorator. It checked Redis, returned if found, or executed the query and stored the result. Simple, right?[1]

"It works on my machine."

— Famous last words Narrator: It didn't.

The Race Condition

Things started failing immediately in staging. The issue manifested as a flicker. You'd update a setting, see the change, refresh, and the old setting was back.

Because we were running three instances of the backend API behind a load balancer, we didn't account for concurrent writes we built a beautifully efficient system for serving stale data rapidly. When an update occurred, we fired off an invalidation command. But if another instance was mid-query to populate the cache, it would write the old data back *after* the invalidation happened.

The Fix

We initially tried distributed locks (Redlock)[2], which only added complexity and latency, defeating the purpose of the cache. Ultimately, we moved to a simpler pattern: Versioned Cache Keys.

Instead of explicitly invalidating, every time a user updates their settings, we increment a version integer in Redis. The cache key for their dashboard is tied to that version. Old data just falls out via TTL. No race conditions, no locks, just peace.