Hi Pythonistas!
Last post we learned 2PC.
It tries to make distributed systems
behave like a single database.
Strict. Synchronous. Blocking.
The problem:
- distributed systems are not single databases.
- Networks fail.
- Services go down.
- Things take time.
2PC fights that reality.
Saga accepts it.
The Core Idea
Don't try to make everything atomic.
Instead:
break the big operation into small steps.
each step does one thing.
in one service.
in one database.
If a step fails:
undo the previous steps.
Using compensating transactions.
Step 1: Create order → Compensate: Cancel order
Step 2: Reserve inventory → Compensate: Release inventory
Step 3: Charge payment → Compensate: Refund payment
No locks across services.
No coordinator waiting for everyone.
Each step independent.
Simple Analogy
You're planning a trip.
Step 1: Book flight
Step 2: Book hotel
Step 3: Book rental car
Flight booked.
Hotel booked.
Rental car - fully booked.
What do you do?
Cancel the hotel.
Cancel the flight.
Start over.
That's a Saga.
Each step can be undone.
You don't need all three to lock simultaneously.
Compensation vs Rollback
Important difference.
Database rollback:
transaction never happened.
instant.
no trace.
Saga compensation:
step DID happen.
you undo it with a new action.
takes time.
leaves a trace.
Database rollback:
INSERT → rollback → row never existed
Saga compensation
INSERT order → order exists
later: UPDATE order SET status='cancelled'
order still exists. just cancelled.
Compensation is a business action.
Not a database operation.
Two Types of Saga
Choreography - services talk via events. no coordinator.
Orchestration - central coordinator tells each service what to do.
Choreography
No coordinator.
Services communicate via events.
Each service listens. reacts. publishes.
Happy path:
Order Service creates order
→ publishes: "order.created"
Inventory Service listens
→ reserves inventory
→ publishes: "inventory.reserved"
Payment Service listens
→ charges payment
→ publishes: "payment.completed"
Order Service listens
→ confirms order ✅
Failure path:
Payment Service fails
→ publishes: "payment.failed"
Inventory Service listens
→ releases inventory
→ publishes: "inventory.released"
Order Service listens
→ cancels order ✅
Code:
# Order Service
@queue.subscribe('order.create.requested')
def create_order(message):
order = db.create_order(
user_id=message['user_id'],
items=message['items'],
amount=message['amount']
)
queue.publish('order.created', {
'order_id': order.id,
'items': message['items'],
'amount': message['amount']
})
@queue.subscribe('payment.failed')
def handle_payment_failed(message):
db.cancel_order(message['order_id'])
queue.publish('order.cancelled', {
'order_id': message['order_id']
})
# Inventory Service
@queue.subscribe('order.created')
def reserve_inventory(message):
try:
db.reserve_items(
order_id=message['order_id'],
items=message['items']
)
queue.publish('inventory.reserved', {
'order_id': message['order_id'],
'amount': message['amount']
})
except InsufficientStockError:
queue.publish('inventory.reservation.failed', {
'order_id': message['order_id']
})
@queue.subscribe('payment.failed')
def release_inventory(message):
db.release_reservation(message['order_id'])
# Payment Service
@queue.subscribe('inventory.reserved')
def charge_payment(message):
try:
payment = payment_gateway.charge(
order_id=message['order_id'],
amount=message['amount']
)
queue.publish('payment.completed', {
'order_id': message['order_id'],
'payment_id': payment.id
})
except PaymentFailedError:
queue.publish('payment.failed', {
'order_id': message['order_id']
})
Good:
no central coordinator.
services independent.
loosely coupled.
Bad:
hard to track where saga is.
hard to debug.
business logic spread everywhere.
Orchestration
Central coordinator.
Called Saga Orchestrator.
All business logic in one place.
Flow:
Orchestrator:
Step 1 → "Order Service, create order" ✅
Step 2 → "Inventory Service, reserve" ✅
Step 3 → "Payment Service, charge" ❌ failed
Compensate in reverse:
Step 2 undo → "Inventory, release" ✅
Step 1 undo → "Order Service, cancel" ✅
Code:
class OrderSagaOrchestrator:
def place_order(self, user_id, items, card, amount):
state = SagaState(saga_id=str(uuid.uuid4()))
state.input_data = {
'user_id': user_id,
'items': items,
'card': card,
'amount': amount
}
state.status = 'started'
state.save() # save: started
try:
# Step 1
if state.current_step < 1:
order = order_service.create_order(
saga_id=state.saga_id, # idempotency key
user_id=user_id,
items=items,
amount=amount
)
state.order_id = order.id
state.current_step = 1
state.save() # save: step 1 done
# Step 2
if state.current_step < 2:
inventory_service.reserve(
saga_id=state.saga_id,
order_id=state.order_id,
items=items
)
state.current_step = 2
state.save() # save: step 2 done
# Step 3
if state.current_step < 3:
payment = payment_service.charge(
saga_id=state.saga_id,
order_id=state.order_id,
card=card,
amount=amount
)
state.payment_id = payment.id
state.current_step = 3
state.save() # save: step 3 done
state.status = 'completed'
state.save()
return {
'status': 'success',
'order_id': state.order_id
}
except InventoryError:
self._compensate_step1(state)
state.status = 'failed'
state.save()
raise OrderFailedError('Items not available')
except PaymentError:
self._compensate_step2(state)
self._compensate_step1(state)
state.status = 'failed'
state.save()
raise OrderFailedError('Payment failed')
def _compensate_step1(self, state):
try:
order_service.cancel_order(state.order_id)
except Exception as e:
logger.error(f'Compensation step 1 failed: {e}')
self._alert_manual_intervention(state, step=1)
def _compensate_step2(self, state):
try:
inventory_service.release(state.order_id)
except Exception as e:
logger.error(f'Compensation step 2 failed: {e}')
self._alert_manual_intervention(state, step=2)
Saga State - Save After Every Step
CREATE TABLE order_saga_state (
saga_id VARCHAR(255) PRIMARY KEY,
current_step INTEGER DEFAULT 0,
status VARCHAR(50) DEFAULT 'started',
order_id INTEGER,
payment_id INTEGER,
input_data JSONB,
error_message TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
Why database?
Not memory. Not Redis.
Memory → orchestrator crashes → state gone → can't resume
Database → orchestrator crashes → state safe → resume exactly where stopped
Crash at Step 2?
Restart. Load state. See current_step = 2. Resume Step 3.
Not from beginning.
Multiple Saga Types
Each business operation gets its own table.
Or use a generic table:
CREATE TABLE sagas (
saga_id VARCHAR(255) PRIMARY KEY,
saga_type VARCHAR(100), -- 'order_placement'
current_step INTEGER DEFAULT 0,
status VARCHAR(50),
payload JSONB, -- all data here
error TEXT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- compensation log shared across all saga types
CREATE TABLE saga_compensation_log (
id SERIAL PRIMARY KEY,
saga_id VARCHAR(255),
step INTEGER,
action VARCHAR(100),
status VARCHAR(50),
error TEXT,
created_at TIMESTAMP DEFAULT NOW()
);
Compensation Rules
Rule 1 - Must be idempotent:
Rule 2 - Must always be reversible:
Creating → cancel or delete
Reserving → release
Charging → refund
Rule 3 - Some things can't be undone:
Sent email → send correction email
Sent SMS → send update SMS
Design around these.
Rule 4 — Compensate in reverse:
Step 1 ✅ Step 2 ✅ Step 3 ❌
Compensate: Step 2 first → Step 1 second
Always reverse order.
What If Compensation Fails?
Compensation might fail.
Step 3 failed: payment failed
Compensate Step 2: release inventory → FAILED ❌
Options:
Retry with backoff:
def compensate_with_retry(self, action, max_retries=5):
for attempt in range(max_retries):
try:
action()
return
except Exception:
time.sleep(2 ** attempt)
self.alert_manual_intervention()
Dead letter queue:
queue.publish('saga.compensation.failed', {
'saga_id': state.saga_id,
'step': 2,
'action': 'release_inventory',
'order_id': state.order_id
})
Manual intervention:
Human fixes it.
Support team resolves.
This happens.
Rare. But must be handled.
Choreography vs Orchestration
Choreography Orchestration
────────────────────────────────────────────────
Coordinator None Central
Business logic Spread out One place
Debugging Hard Easier
Visibility Low High
Coupling Loose Tighter
Best for Simple flows Complex flows
2PC vs Saga
2PC Saga
────────────────────────────────────────────
Consistency Strong (ACID) Eventual
Availability Lower Higher
Blocking Yes No
Speed Slow Faster
Rollback Automatic Manual (compensation)
Complexity Implementation Design
Use case Financial, DB Microservices
Temporary Inconsistency
Saga has a tradeoff.
Between Step 1 and Step 3:
data temporarily inconsistent.
Step 1: Order created (status: pending)
Step 2: Inventory reserved
Step 3: Payment processing...
Another user queries:
→ sees pending order
→ that's OK
Eventually consistent.
Step 3 completes → order confirmed.
Step 3 fails → compensation runs → order cancelled.
Either way eventually consistent.
This is the tradeoff.
Not immediately consistent.
But more available.
More resilient.
Honest Truth - This Is Complex
Simple database transaction:
with db.transaction():
deduct_balance(user_a, 100)
add_balance(user_b, 100)
# 3 lines. ACID handles everything.
Same as Saga:
- saga_state table
- saga_compensation_log table
- SagaOrchestrator class
- Resume logic
- Compensation logic
- Idempotency everywhere
- Dead letter queues
- Monitoring for stuck sagas
- Alert system
- Manual intervention process
Ten times more code.
Ten times more things to go wrong.
When Do You Actually Need Saga?
Most apps don't.
One database → normal transactions. no saga.
Two databases → maybe redesign. maybe saga.
Many microservices → saga. but use a framework.
Reality:
90% of startups → one PostgreSQL database→ normal transactions→ no saga needed
Growing company → a few services → most operations still in one DB→ saga for cross-service only (rare)
Large company → many microservices→ saga needed→ but they use Temporal or Conductor→ dedicated platform team handles it
Don't Build From Scratch - Use a Framework
Temporal (recommended):
from temporalio import workflow, activity
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_data):
# Temporal handles:
# → state persistence automatically
# → crash recovery automatically
# → retries automatically
# you just write the steps
order = await workflow.execute_activity(
create_order,
order_data,
start_to_close_timeout=timedelta(seconds=10)
)
try:
await workflow.execute_activity(
reserve_inventory,
order.id
)
await workflow.execute_activity(
charge_payment,
order.id
)
except Exception:
await workflow.execute_activity(
cancel_order,
order.id
)
raise
Temporal handles state.
Crash recovery.
Retries.
You write business logic.
Nothing else.
Other frameworks:
Temporal → most popular, Python/Go/Java
Conductor → Netflix open-sourced
Camunda → Java ecosystem
Before Building Saga - Ask First
Can I redesign to avoid this?
Can these operations live in one service?
Can this data live in one database?
Can I accept eventual consistency without saga?
Most of the time:
yes.
you can redesign.
Saga is last resort.
Not first solution.
Mental Model
Saga → steps + compensations
Compensation → business action to undo a step
Choreography → event-driven, no coordinator
Orchestration → central coordinator
Saga state → saved in DB after every step
Compensation log → tracks what compensations ran
Idempotency → required everywhere
Temporary inconsistency → acceptable tradeoff
Compensation fails → retry, DLQ, manual intervention
Don't build → from scratch, use Temporal
Saga is complex → last resort, not first solution
What's Coming Next
Phase 4 - Reliability is complete.
You now understand:
Post 16 → Message queues
Post 17 → Rate limiting
Post 18 → Circuit breakers
Post 19 → Idempotency
Post 20A → Distributed transactions (2PC)
Post 20B → Saga pattern
Next - Phase 5: Architecture Patterns.
How do you structure large systems?
Monolith vs microservices.
When to split. When not to.
The decisions that define
how your system grows.
See you in Post 21.