System Design

System Design Adventure-20B

Saga Pattern: Designing for Failure

Posted by Afsal on 20 Nov 2026

Hi Pythonistas!

Last post we learned 2PC.

It tries to make distributed systems

behave like a single database.

Strict. Synchronous. Blocking.

The problem:

  • distributed systems are not single databases.
  • Networks fail.
  • Services go down.
  • Things take time.

2PC fights that reality.

Saga accepts it.

The Core Idea

Don't try to make everything atomic.

Instead:

break the big operation into small steps.

each step does one thing.

in one service.

in one database.

If a step fails:

undo the previous steps.

Using compensating transactions.

Step 1: Create order      → Compensate: Cancel order

Step 2: Reserve inventory → Compensate: Release inventory

Step 3: Charge payment    → Compensate: Refund payment

No locks across services.

No coordinator waiting for everyone.

Each step independent.

Simple Analogy

You're planning a trip.

Step 1: Book flight

Step 2: Book hotel

Step 3: Book rental car

Flight booked.

Hotel booked.

Rental car - fully booked.

What do you do?

Cancel the hotel.

Cancel the flight.

Start over.

That's a Saga.

Each step can be undone.

You don't need all three to lock simultaneously.

Compensation vs Rollback

Important difference.

Database rollback:

transaction never happened.

instant.

no trace.

Saga compensation:

step DID happen.

you undo it with a new action.

takes time.

leaves a trace.

Database rollback:

INSERT → rollback → row never existed

Saga compensation

INSERT order → order exists

later: UPDATE order SET status='cancelled'

order still exists. just cancelled.

Compensation is a business action.

Not a database operation.

Two Types of Saga

Choreography - services talk via events. no coordinator.

Orchestration - central coordinator tells each service what to do.

Choreography

No coordinator.

Services communicate via events.

Each service listens. reacts. publishes.

Happy path:

Order Service creates order

→ publishes: "order.created"

Inventory Service listens

→ reserves inventory

→ publishes: "inventory.reserved"

Payment Service listens

→ charges payment

→ publishes: "payment.completed"

Order Service listens

→ confirms order ✅

Failure path:

Payment Service fails

→ publishes: "payment.failed"

Inventory Service listens

→ releases inventory

→ publishes: "inventory.released"

Order Service listens

→ cancels order ✅

Code:

# Order Service

@queue.subscribe('order.create.requested')
def create_order(message):
    order = db.create_order(
        user_id=message['user_id'],
        items=message['items'],
        amount=message['amount']
    )

    queue.publish('order.created', {
        'order_id': order.id,
        'items':    message['items'],
        'amount':   message['amount']
    })


@queue.subscribe('payment.failed')
def handle_payment_failed(message):
    db.cancel_order(message['order_id'])
    queue.publish('order.cancelled', {
        'order_id': message['order_id']
    })


# Inventory Service
@queue.subscribe('order.created')
def reserve_inventory(message):
    try:
        db.reserve_items(
            order_id=message['order_id'],
            items=message['items']
        )

        queue.publish('inventory.reserved', {
            'order_id': message['order_id'],
            'amount':   message['amount']
        })

    except InsufficientStockError:
        queue.publish('inventory.reservation.failed', {
            'order_id': message['order_id']
        })


@queue.subscribe('payment.failed')
def release_inventory(message):
    db.release_reservation(message['order_id'])


# Payment Service
@queue.subscribe('inventory.reserved')
def charge_payment(message):
    try:
        payment = payment_gateway.charge(
            order_id=message['order_id'],
            amount=message['amount']
        )

        queue.publish('payment.completed', {
            'order_id':  message['order_id'],
            'payment_id': payment.id
        })

    except PaymentFailedError:
        queue.publish('payment.failed', {
            'order_id': message['order_id']
        })

Good:

no central coordinator.

services independent.

loosely coupled.

Bad:

hard to track where saga is.

hard to debug.

business logic spread everywhere.

Orchestration

Central coordinator.

Called Saga Orchestrator.

All business logic in one place.

Flow:

Orchestrator:

Step 1 → "Order Service, create order"      ✅

Step 2 → "Inventory Service, reserve"       ✅

Step 3 → "Payment Service, charge"          ❌ failed

Compensate in reverse:

Step 2 undo → "Inventory, release"          ✅

Step 1 undo → "Order Service, cancel"       ✅

Code:

class OrderSagaOrchestrator:

    def place_order(self, user_id, items, card, amount):
        state = SagaState(saga_id=str(uuid.uuid4()))
        state.input_data = {
            'user_id': user_id,
            'items':   items,
            'card':    card,
            'amount':  amount
        }
        state.status = 'started'
        state.save()                    # save: started

        try:
            # Step 1
            if state.current_step < 1:
                order = order_service.create_order(
                    saga_id=state.saga_id,  # idempotency key
                    user_id=user_id,
                    items=items,
                    amount=amount
                )
                state.order_id     = order.id
                state.current_step = 1
                state.save()            # save: step 1 done

            # Step 2
            if state.current_step < 2:
                inventory_service.reserve(
                    saga_id=state.saga_id,
                    order_id=state.order_id,
                    items=items
                )
                state.current_step = 2
                state.save()            # save: step 2 done

            # Step 3
            if state.current_step < 3:
                payment = payment_service.charge(
                    saga_id=state.saga_id,
                    order_id=state.order_id,
                    card=card,
                    amount=amount
                )

                state.payment_id   = payment.id
                state.current_step = 3
                state.save()            # save: step 3 done

            state.status = 'completed'
            state.save()

            return {
                'status':   'success',
                'order_id': state.order_id
            }

        except InventoryError:
            self._compensate_step1(state)
            state.status = 'failed'
            state.save()
            raise OrderFailedError('Items not available')

        except PaymentError:
            self._compensate_step2(state)
            self._compensate_step1(state)
            state.status = 'failed'
            state.save()
            raise OrderFailedError('Payment failed')

    def _compensate_step1(self, state):
        try:
            order_service.cancel_order(state.order_id)
        except Exception as e:
            logger.error(f'Compensation step 1 failed: {e}')
            self._alert_manual_intervention(state, step=1)

    def _compensate_step2(self, state):
        try:
            inventory_service.release(state.order_id)
        except Exception as e:
            logger.error(f'Compensation step 2 failed: {e}')
            self._alert_manual_intervention(state, step=2)

Saga State - Save After Every Step

CREATE TABLE order_saga_state (
    saga_id         VARCHAR(255) PRIMARY KEY,
    current_step    INTEGER DEFAULT 0,
    status          VARCHAR(50) DEFAULT 'started',
    order_id        INTEGER,
    payment_id      INTEGER,
    input_data      JSONB,
    error_message   TEXT,
    created_at      TIMESTAMP DEFAULT NOW(),
    updated_at      TIMESTAMP DEFAULT NOW()
);

Why database?

Not memory. Not Redis.

Memory → orchestrator crashes → state gone → can't resume

Database → orchestrator crashes → state safe → resume exactly where stopped

Crash at Step 2?

Restart. Load state. See current_step = 2. Resume Step 3.

Not from beginning.

Multiple Saga Types

Each business operation gets its own table.

Or use a generic table:

CREATE TABLE sagas (
    saga_id      VARCHAR(255) PRIMARY KEY,
    saga_type    VARCHAR(100),      -- 'order_placement'
    current_step INTEGER DEFAULT 0,
    status       VARCHAR(50),
    payload      JSONB,             -- all data here
    error        TEXT,
    created_at   TIMESTAMP DEFAULT NOW(),
    updated_at   TIMESTAMP DEFAULT NOW()
);

-- compensation log shared across all saga types

CREATE TABLE saga_compensation_log (
    id         SERIAL PRIMARY KEY,
    saga_id    VARCHAR(255),
    step       INTEGER,
    action     VARCHAR(100),
    status     VARCHAR(50),
    error      TEXT,
    created_at TIMESTAMP DEFAULT NOW()
);

Compensation Rules

Rule 1 - Must be idempotent:

Rule 2 - Must always be reversible:

Creating → cancel or delete

Reserving → release

Charging → refund

Rule 3 - Some things can't be undone:

Sent email     → send correction email

Sent SMS       → send update SMS

Design around these.

Rule 4 — Compensate in reverse:

Step 1 ✅ Step 2 ✅ Step 3 ❌

Compensate: Step 2 first → Step 1 second

Always reverse order.

What If Compensation Fails?

Compensation might fail.

Step 3 failed: payment failed

Compensate Step 2: release inventory → FAILED ❌

Options:

Retry with backoff:

def compensate_with_retry(self, action, max_retries=5):
    for attempt in range(max_retries):
        try:
            action()
            return
        except Exception:
            time.sleep(2 ** attempt)
    self.alert_manual_intervention()

Dead letter queue:

queue.publish('saga.compensation.failed', {
    'saga_id':  state.saga_id,
    'step':     2,
    'action':   'release_inventory',
    'order_id': state.order_id
})

Manual intervention:

Human fixes it.

Support team resolves.

This happens.

Rare. But must be handled.

Choreography vs Orchestration

               Choreography    Orchestration

────────────────────────────────────────────────

Coordinator     None            Central

Business logic  Spread out      One place

Debugging       Hard            Easier

Visibility      Low             High

Coupling        Loose           Tighter

Best for        Simple flows    Complex flows

2PC vs Saga

               2PC             Saga

────────────────────────────────────────────

Consistency     Strong (ACID)   Eventual

Availability    Lower           Higher

Blocking        Yes             No

Speed           Slow            Faster

Rollback        Automatic       Manual (compensation)

Complexity      Implementation  Design

Use case        Financial, DB   Microservices

Temporary Inconsistency

Saga has a tradeoff.

Between Step 1 and Step 3:

data temporarily inconsistent.

Step 1: Order created (status: pending)

Step 2: Inventory reserved

Step 3: Payment processing...

Another user queries:

→ sees pending order

→ that's OK

Eventually consistent.

Step 3 completes → order confirmed.

Step 3 fails → compensation runs → order cancelled.

Either way  eventually consistent.

This is the tradeoff.

Not immediately consistent.

But more available.

More resilient.

Honest Truth - This Is Complex

Simple database transaction:

with db.transaction():
    deduct_balance(user_a, 100)
    add_balance(user_b, 100)

# 3 lines. ACID handles everything.

Same as Saga:

  1. saga_state table
  2. saga_compensation_log table
  3. SagaOrchestrator class
  4. Resume logic
  5. Compensation logic
  6. Idempotency everywhere
  7. Dead letter queues
  8. Monitoring for stuck sagas
  9. Alert system
  10. Manual intervention process

Ten times more code.

Ten times more things to go wrong.

When Do You Actually Need Saga?

Most apps don't.

One database      → normal transactions. no saga.

Two databases     → maybe redesign. maybe saga.

Many microservices → saga. but use a framework.

Reality:

90% of startups   → one PostgreSQL database→ normal transactions→ no saga needed

Growing company   → a few services → most operations still in one DB→ saga for cross-service only (rare)

Large company     → many microservices→ saga needed→ but they use Temporal or Conductor→ dedicated platform team handles it

Don't Build From Scratch - Use a Framework

Temporal (recommended):

from temporalio import workflow, activity

@workflow.defn

class OrderWorkflow:

    @workflow.run
    async def run(self, order_data):
        # Temporal handles:
        # → state persistence automatically
        # → crash recovery automatically
        # → retries automatically
        # you just write the steps

        order = await workflow.execute_activity(
            create_order,
            order_data,
            start_to_close_timeout=timedelta(seconds=10)
        )

        try:
            await workflow.execute_activity(
                reserve_inventory,
                order.id
            )

            await workflow.execute_activity(
                charge_payment,
                order.id
            )

        except Exception:
            await workflow.execute_activity(
                cancel_order,
                order.id
            )
            raise

Temporal handles state.

Crash recovery.

Retries.

You write business logic.

Nothing else.

Other frameworks:

Temporal   → most popular, Python/Go/Java

Conductor  → Netflix open-sourced

Camunda    → Java ecosystem

Before Building Saga - Ask First

Can I redesign to avoid this?

Can these operations live in one service?

Can this data live in one database?

Can I accept eventual consistency without saga?

Most of the time:

yes.

you can redesign.

Saga is last resort.

Not first solution.

Mental Model

Saga              → steps + compensations

Compensation      → business action to undo a step

Choreography      → event-driven, no coordinator

Orchestration     → central coordinator

Saga state        → saved in DB after every step

Compensation log  → tracks what compensations ran

Idempotency       → required everywhere

Temporary inconsistency → acceptable tradeoff

Compensation fails → retry, DLQ, manual intervention

Don't build       → from scratch, use Temporal

Saga is complex   → last resort, not first solution

What's Coming Next

Phase 4 - Reliability is complete.

You now understand:

Post 16 → Message queues

Post 17 → Rate limiting

Post 18 → Circuit breakers

Post 19 → Idempotency

Post 20A → Distributed transactions (2PC)

Post 20B → Saga pattern

Next - Phase 5: Architecture Patterns.

How do you structure large systems?

Monolith vs microservices.

When to split. When not to.

The decisions that define

how your system grows.

See you in Post 21.

← Previous Post

Recent posts