Hi Pythonistas!
Your system has multiple services.
API Server → Payment Service → Bank API
Bank API gets slow.
Takes 10 seconds to respond.
Payment Service waits.
API Server waits for Payment Service.
User request → API Server (waiting...)
↓
Payment Service (waiting...)
↓
Bank API (slow... 10 seconds)
Meanwhile:
more requests come in.
all waiting for Bank API.
threads pile up.
memory fills up.
API Server runs out of threads.
New requests → API Server → no threads → timeout → error
Now API Server is down.
Not because of its own problem.
Because of Bank API.
One slow service took down everything.
This is called a cascading failure.
The Electrical Analogy
Your home has electrical circuits.
One circuit per section.
Kitchen. Bedroom. Living room.
Too much electricity in kitchen:
circuit breaker trips.
Kitchen loses power.
But:
bedroom still works.
living room still works.
The rest of the house is protected.
Without circuit breaker:
too much electricity everywhere.
entire house loses power.
Fire risk.
Software circuit breaker works the same way.
One service fails:
circuit breaker trips.
That service is isolated.
Rest of system keeps working.
What Is a Circuit Breaker?
A pattern that:
monitors calls to a service.
detects failures.
stops calls when failure rate is too high.
lets the service recover.
automatically resumes when healthy.
Three States
CLOSED - normal operation
Everything working.
Requests flow normally.
Circuit breaker monitors failure rate.
Request → Service → Response ✅
Request → Service → Response ✅
OPEN - failure detected
Too many failures.
Circuit trips.
Requests immediately rejected.
No calls to failing service.
Request → Circuit Breaker → Error immediately ❌
(service not called at all)
HALF-OPEN - testing recovery
After timeout:
allows a few test requests.
Success → back to CLOSED.
Failure → back to OPEN.
Test request → Service → Success ✅ → CLOSED
Test request → Service → Failure ❌ → OPEN again
State Transitions
failures exceed threshold
CLOSED ──────────────────────────→ OPEN
↑ │
│ │ timeout expires
│ ↓
└────────────────────────── HALF-OPEN
test requests succeedSimple Implementation
import time
from enum import Enum
class State(Enum):
CLOSED = 'closed'
OPEN = 'open'
HALF_OPEN = 'half_open'
class CircuitBreaker:
def __init__(
self,
failure_threshold=5,
timeout=60,
success_threshold=2
):
self.failure_threshold = failure_threshold
self.timeout = timeout
self.success_threshold = success_threshold
self.state = State.CLOSED
self.failure_count = 0
self.success_count = 0
self.last_failure_time = None
def call(self, func, *args, **kwargs):
if self.state == State.OPEN:
if time.time() - self.last_failure_time > self.timeout:
self.state = State.HALF_OPEN
self.success_count = 0
else:
raise Exception('Circuit breaker is OPEN')
try:
result = func(*args, **kwargs)
self._on_success()
return result
except Exception as e:
self._on_failure()
raise e
def _on_success(self):
if self.state == State.HALF_OPEN:
self.success_count += 1
if self.success_count >= self.success_threshold:
self.state = State.CLOSED
self.failure_count = 0
elif self.state == State.CLOSED:
self.failure_count = 0
def _on_failure(self):
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = State.OPENUsing It
payment_cb = CircuitBreaker(
failure_threshold=5,
timeout=60,
success_threshold=2
)
def process_payment(amount, card):
try:
return payment_cb.call(
bank_api.charge,
amount=amount,
card=card
)
except Exception as e:
if 'Circuit breaker is OPEN' in str(e):
return {
'status': 'pending',
'message': 'Payment queued. Will process shortly.'
}
raise eImportant Question
You might think:
"If two services depend on each other isolating one doesn't help. The feature is broken anyway."
You're right.
And this is critical to understand.
Circuit Breaker's Real Job
Circuit breaker solves:
cascading failures.
It does NOT solve:
feature unavailability.
Without circuit breaker:
Payment Service down
→ Order Service hangs waiting
→ threads fill up
→ Order Service crashes
→ BOTH services down
→ everything broken
With circuit breaker:
Payment Service down
→ circuit opens
→ Order Service gets immediate error
→ Order Service stays alive
→ only payment features broken
→ everything else works
Circuit breaker saved Order Service.
The feature is still broken.
But that's a different problem.
Solutions for Tightly Dependent Services
When a feature genuinely needs both services
you have an architectural problem.
Several ways to handle it.
Solution 1 - Partial Response
Return what you have.
Show incomplete data.
Better than nothing.
def get_order_history(user_id):
orders = order_service.get_orders(user_id)
for order in orders:
try:
payment = payment_cb.call(
payment_service.get_status,
order.payment_id
)
order.payment_status = payment.status
except:
order.payment_status = 'status unavailable'
return ordersUser sees orders.
Payment status shows "unavailable."
Not ideal.
Better than blank page.
Solution 2 - Cached Data
Show last known state.
def get_payment_status(payment_id):
try:
status = payment_cb.call(
payment_service.get_status,
payment_id
)
redis.set(f'payment:{payment_id}', status, ex=3600)
return status
except:
cached = redis.get(f'payment:{payment_id}')
if cached:
return {
'status': cached,
'note': 'Last known status (may be outdated)'
}
return {'status': 'unknown'}Stale data often acceptable.
User sees last known status.
With a note it might be outdated.
Solution 3 - Async Events
This is the architectural fix.
Instead of real-time calls:
Order Service calls Payment Service every request
→ tight coupling
→ one down = feature broken
Use event-driven approach:
# Payment Service publishes events
queue.publish('payment.status.updated', {
'payment_id': 123,
'order_id': 456,
'status': 'completed'
})
# Order Service consumes and stores locally
@queue.subscribe('payment.status.updated')
def handle_payment_update(message):
db.update_order(
order_id=message['order_id'],
payment_status=message['status']
)
# Order Service reads from its OWN database
def get_order_history(user_id):
# no call to Payment Service at runtime
return db.get_orders_with_payment_status(user_id)Order Service has its own copy of payment status.
Payment Service down?
Order Service reads from its own DB.
Feature works completely.
This is called data denormalization across services.
Solution 4 - Feature Flags
Hide broken parts cleanly.
def get_order_history(user_id):
orders = order_service.get_orders(user_id)
if payment_cb.state == State.OPEN:
return {
'orders': orders,
'show_payment_status': False,
'message': 'Payment status temporarily unavailable'
}
for order in orders:
order.payment_status = payment_service.get_status(
order.payment_id
)
return {
'orders': orders,
'show_payment_status': True
}UI hides payment column.
User sees order list.
Understands payment is temporarily down.
Clean. Honest.
Solution 5 - Rethink the Boundary
Sometimes the real problem is:
two services always needed together
should be one service.
Order Service + Payment Service
always needed together?
→ maybe wrong boundary
→ merge them
→ or payment status lives in Order Service
Microservice boundaries are not sacred.
If two services are always coupled:
the boundary is wrong.
The Honest Truth About Microservices
This dependency problem
is the biggest criticism of microservices.
Monolith:
→ Order and Payment in same codebase
→ no network calls
→ no dependency failures
→ simpler
Microservices:
→ separate services
→ network calls between them
→ circuit breakers, retries, fallbacks
→ much more complex
Microservices solve some problems.
Create others.
This is why some teams go back to monoliths.
Or use a modular monolith separate modules in same codebase.
No network calls.
No dependency failures.
We'll discuss this in Phase 5 - Architecture Patterns.
Fallback Strategies
When circuit is OPEN don't just return an error. have a plan.
Return cached data → show last known state
Return default response → generic popular items
Queue for later → process when service recovers
Fail fast clearly → honest error message
Hide the feature → feature flag
Circuit Breaker vs Retry
Different tools. Different purposes.
Retry:
for attempt in range(3):
try:
result = service.call()
break
except:
time.sleep(2 ** attempt)
Good for: temporary blips, network hiccups.
Bad when: service is completely down.
Retrying 1000 times on dead service makes it worse.
Circuit Breaker:
Service consistently failing.
Stop calling it.
Wait for recovery.
Without CB:
1000 users × 3 retries = 3000 calls to dead service
All waiting. All timing out. Making it worse.
With CB:
5 failures → circuit opens
995 users → immediate error (no wait)
Dead service → zero calls → can recover
python
@circuit_breaker
@retry(attempts=3)
def call_service():
return service.call()
Retry for temporary failures.
Circuit breaker for persistent failures.
Always Set Timeouts
Circuit breaker needs timeouts to work.
Without timeout:
Request → Service (hangs forever)
Thread waits forever
Circuit breaker never sees failure
With timeout:
response = requests.post(
'https://payment.internal/charge',
json=data,
timeout=3 # 3 second timeout
)
Timeout raises exception.
Circuit breaker counts it as failure.
After threshold → circuit opens.
Always set timeouts.
Never call external service without one.
Bulkhead Pattern
Related concept.
Separate resource pools per service.
Like watertight compartments in a ship.
Without bulkhead:
Thread pool: 100 threads (shared)
Payment slow → 100 threads waiting → system unresponsive
With bulkhead:
Payment pool: 20 threads
User pool: 20 threads
Search pool: 20 threads
Other: 40 threads
Payment slow → 20 threads waiting
→ other 80 threads still working
→ rest of system responsive
from concurrent.futures import ThreadPoolExecutor
payment_executor = ThreadPoolExecutor(max_workers=20)
user_executor = ThreadPoolExecutor(max_workers=20)
def charge_payment(data):
future = payment_executor.submit(
payment_service.charge, data
)
return future.result(timeout=3)
def get_user(user_id):
future = user_executor.submit(
user_service.get, user_id
)
return future.result(timeout=3)
Payment pool exhausted.
User service still works.
Graceful Degradation
System keeps working.
With reduced functionality.
Full functionality:
→ Personalized feed ✅
→ Recommended movies ✅
→ Notifications ✅
→ Social features ✅
Recommendation service down:
→ Personalized feed ✅
→ Popular movies (fallback) ⚠️
→ Notifications ✅
→ Social features ✅
Core functionality works.
Non-critical features degrade.
Users notice less than total outage.
Netflix calls this graceful degradation.
It's what Hystrix and Resilience4j enable.
Using pybreaker
import pybreaker
payment_breaker = pybreaker.CircuitBreaker(
fail_max=5,
reset_timeout=60
)
@payment_breaker
def charge_card(amount, card):
return bank_api.charge(amount, card)
try:
result = charge_card(100, card_details)
except pybreaker.CircuitBreakerError:
return fallback_response()
Don't implement from scratch.
Use battle-tested libraries.
Mental Model
Cascading failure → one slow service takes down everything
Circuit breaker → isolates failing service
CLOSED → normal, requests flow
OPEN → failing, requests rejected immediately
HALF_OPEN → testing recovery
failure_threshold → failures before opening
Fallback → what to do when circuit is open
Retry → for temporary failures
Circuit breaker → for persistent failures
Timeout → required for CB to detect failures
Bulkhead → separate resource pools
Graceful degradation → keep working with reduced functionality
CB solves → cascading failures
CB does NOT solve → feature unavailability
Tight dependency fix → async events, cached data, partial response
Wrong boundary → merge services or redesign ownership
What's Coming Next
Now your system handles failures gracefully.
But what about operations that must happen
across multiple services?
Transfer money from Service A to Service B.
Both must succeed or neither should.
How do you guarantee that
across distributed systems?
Idempotency and Distributed Transactions.
How to make operations safe to retry.
And how to coordinate across services.
