Hi Pythonistas!
You have multiple servers now.Load balancer distributing traffic.System handling more requests.
But every request still does this:
Request comes in
↓
App server processes it
↓
Query the database
↓
Database reads from disk
↓
Returns data
↓
App server sends response
Every. Single. Request.Hits the database.Database reads from disk. Disk is slow.
Memory access: ~100 nanoseconds
SSD access: ~100 microseconds (1,000x slower)
HDD access: ~10 milliseconds (100,000x slower)
And most requests ask for the same data.Homepage. Popular posts. User profiles.Same query. Same result.
Millions of times a day.Why go to the database every time?Store the result somewhere fast.Next time skip the database entirely. That's caching.
What Is a Cache?
A cache is a fast temporary storage layer.Sits between your app and your database.
Request comes in
↓
Check cache first
↓
Found? → return immediately
Not found? → query database → store in cache → return
Cache stores data in memory (RAM).RAM is orders of magnitude faster than disk.
Database query (disk): 10-100ms
Cache lookup (RAM): <1ms
100x faster.
Cache Hit and Cache Miss
Two outcomes when you check the cache:
Cache Hit - Data is in the cache.
Request → Cache → Found → Return immediately
Fast. Database not touched.
Cache Miss - Data is not in the cache.
Request → Cache → Not found → Database → Store in cache → Return
Slower this time.But next time → cache hit.
Hit Rate
Percentage of requests served from cache.
Hit rate = Cache hits / Total requests × 100
High hit rate = good.
90% hit rate → only 10% of requests hit the database. Database load reduced by 90%.
Where to Cache?
Cache can live at multiple levels.
Client-side cache
Browser caches responses.Response header:Cache-Control: max-age=3600.Browser stores response for 1 hour.Next request for same resource → served from browser.No network request at all.Fastest possible cache.
CDN cache
Content Delivery Network.Caches static files at edge servers worldwide.User in Kerala → gets files from nearest CDN node.
Not from your server in the US.More on CDN in the next post.
Application cache
Cache inside your app server.In-memory dictionary.Fast. But:each server has its own cache.Server1's cache ≠ Server2's cache.
Inconsistent across servers.
Distributed cache
Shared cache across all servers.
Server1 ─┐
Server2 ─┼──→ Redis (shared cache)
Server3 ─┘
All servers read from and write to the same cache.Consistent. Scalable.Most common in production.
Redis is the most popular distributed cache.
import redis
r = redis.Redis(host='localhost', port=6379)
# store with expiry
r.setex('user:123', 3600, json.dumps(user_data))
# retrieve
user = json.loads(r.get('user:123'))The Layers Together
A real production system has caching at every level:
Browser cache
↓ (miss)
CDN cache
↓ (miss)
Load balancer
↓
App server
↓ (miss)
Redis cache
↓ (miss)
Database
Each layer catches what the layer above missed.
Database only gets hit when no layer has the answer.
Rough hit rates at each layer:
Browser cache → 30-40% of requests
CDN cache → 40-50% of remaining
Redis cache → 80-90% of remaining
Database → only 5-10% of all requests
Database load reduced by 90-95%.
Cache Invalidation
Caching has one hard problem.When do you remove stale data?User updates their profile.Database updated.Cache still has old data.Next request → cache hit → returns old data.User is confused.This is called stale data.
There's a famous saying in computer science:"There are only two hard things in computer science:
cache invalidation and naming things." - Phil Karlton
Several strategies to handle this.
Strategy 1 - TTL (Time To Live)
Every cached item has an expiry time.
r.setex('user:123', 3600, json.dumps(user_data))After 1 hour → cache miss → fresh data from database.Simple. No extra logic needed.
But:
data can be stale for up to TTL duration.
Choose TTL based on how fresh data needs to be:
Stock prices → TTL: 1 second
News articles → TTL: 5 minutes
User profiles → TTL: 1 hour
Product catalog → TTL: 24 hours
Static content → TTL: 30 days
Strategy 2 - Cache Aside (Lazy Loading)
Most common pattern.App checks cache first.
Miss → load from database → store in cache.
def get_user(user_id):
cached = r.get(f'user:{user_id}')
if cached:
return json.loads(cached) # cache hit
user = db.query( # cache miss
"SELECT * FROM users WHERE id = ?",
user_id
)
r.setex(f'user:{user_id}', 3600, json.dumps(user))
return userGood:
- only caches what's actually requested.
- no wasted memory on unused data.
Bad:
- first request always a cache miss.
- stampede problem - more on this soon.
Strategy 3 - Write Through
Every database write also writes to cache.
def update_user(user_id, data):
db.query("UPDATE users SET ... WHERE id = ?", user_id)
r.setex(f'user:{user_id}', 3600, json.dumps(data))Cache always fresh.
Good:
- no stale data.
- high cache hit rate immediately.
Bad:
- every write hits both database AND cache.
- slower writes.
- cache fills with data that might never be read.
Strategy 4 - Write Behind (Write Back)
App writes to cache only.Cache writes to database asynchronously later.
App → Cache (immediate)
Cache → Database (async, later)
Good:
- extremely fast writes.
- database not hit on every write.
Bad:
- cache crashes before writing to database → data lost.
- used in specific high-write scenarios.
- not common for general use.
Strategy 5 - Invalidate on Write
When data changes → delete from cache.
def update_user(user_id, data):
db.query("UPDATE users SET ... WHERE id = ?", user_id)
r.delete(f'user:{user_id}')
# next request → cache miss → fresh data from databaseSimple. Guaranteed fresh data after update.
But:
next request after update → always a miss.stampede risk.
Cache Eviction Policies
Cache has limited memory.What happens when it's full?You need to remove something.
Which item to remove?
LRU - Least Recently Used
Remove the item not accessed the longest.Most common policy.Assumption: recently used data will be used again soon.
LFU - Least Frequently Used
Remove the item accessed the fewest times.Better for skewed access patterns.Popular items stay in cache longer.
FIFO - First In First Out
Remove the oldest item in cache.Regardless of access pattern.Simple. Not very smart.Random Remove a random item.
Surprisingly effective in practice.Simple to implement.
How to choose:
General use → LRU
Popularity matters → LFU
Simplicity needed → FIFO
Unpredictable access → Random
The Thundering Herd ProblemAlso called cache stampede.Imagine:Popular data cached.TTL expires.
1,000 users request it simultaneously.All get cache miss.All hit database at once.Database overwhelmed.
Three solutions:
1. Mutex / Lock
First request gets the lock.Queries database. Updates cache. Releases lock.Other requests wait.Lock released → all read from cache.
2. Staggered TTLs
Add random variation to TTL:
import random
ttl = 3600 + random.randint(-300, 300)
r.setex(key, ttl, data)Different items expire at different times.No mass expiration at the same moment.
3. Probabilistic Early Expiration
Before TTL expires randomly refresh.Some requests that still get a hit will refresh anyway.Cache never fully expires for everyone at once.
What Should You Cache?
Not everything is worth caching.
Good candidates:
- Expensive database queries → user profiles, product details
- Computation results → recommendation scores, aggregates
- Frequently read, rarely written → config, static content
- Session data → user login state
Bad candidates:
- Highly personalized data → different for every user, low reuse
- Rapidly changing data → stock prices, live scores
- Sensitive data → passwords, payment info
- Large objects → huge files, use CDN instead
Mental Model
Cache → fast temporary storage, sits before database
Cache hit → data found, fast
Cache miss → data not found, go to database
Hit rate → percentage served from cache
TTL → how long data stays in cache
Cache aside → lazy loading, most common pattern
Write through → write to cache and database together
Write behind → write to cache, async to database
Invalidate → delete from cache when data changes
LRU → evict least recently used
LFU → evict least frequently used
FIFO → evict oldest item
Redis → most popular distributed cache
Stampede → mass cache miss hitting database simultaneously
Jitter → random TTL variation to prevent stampede
Stale data → cache has old data after database update
What's Coming Next
Now you know what caching is and why it works.But how does LRU actually remove the right item in O(1)?
What data structure makes that possible?
We'll implement LRU, LFU, FIFO from scratch in Python.See you there.
