System Design Adventure-16

Hi Pythonistas!

Your system is working.
Users signing up.
Orders being placed.
Everything running fine.
Then you add a new feature.

When user signs up:
1. Save user to database
2. Send welcome email
3. Send SMS verification
4. Create default settings
5. Notify analytics
6. Set up storage bucket
Six things. One request.

def signup(request):
    user = db.save_user(request.data)        # 50ms
    email.send_welcome(user.email)            # 300ms
    sms.send_verification(user.phone)         # 200ms
    db.create_default_settings(user.id)       # 50ms
    analytics.track_signup(user.id)           # 150ms
    s3.create_user_bucket(user.id)            # 400ms
    return {'success': True}                  # total: 1150ms

User waits 1.15 seconds to sign up.
And if the email service is down?
Signup fails.
User can't create an account.
Because of an email.
There's a better way.

Synchronous vs Asynchronous

Synchronous:

Do one thing.
Wait for it to finish.
Do the next.
Task 1 → wait → Task 2 → wait → Task 3 → done
Simple. But slow.
One failure breaks everything.

Asynchronous:

Start a task.
Don't wait.
Move on.
Task runs in background.
Task 1 → return response
Task 2, 3, 4 run in background
Fast response.
Tasks run independently.
One failure doesn't break others.

What Is a Message Queue?

A buffer between:
the part that creates work (producer).
and the part that does work (consumer).
Producer → [Queue] → Consumer
Producer puts message in queue.
Consumer picks it up and processes it.
They never talk directly.
Completely decoupled.

The Restaurant Analogy

Customer (producer) → gives order to waiter.
Waiter puts order on kitchen counter (queue).
Chef (consumer) picks up and cooks.
Customer doesn't wait at the counter.
Sits down.
Gets food when ready.
Kitchen processes at their own pace.
One chef sick → others keep working.
Kitchen overwhelmed → orders wait in queue.
Nobody loses their order.

How It Solves the Signup Problem

def signup(request):
    # save user (must be synchronous)
    user = db.save_user(request.data)    # 50ms

    # everything else → queue
    queue.publish('user.signup', {
        'user_id': user.id,
        'email': user.email,
        'phone': user.phone
    })

    return {'success': True}             # total: 55ms

Response in 55ms instead of 1150ms.

20x faster.
Background worker handles the rest:

@queue.subscribe('user.signup')
def handle_signup(message):
    email.send_welcome(message['email'])
    sms.send_verification(message['phone'])
    db.create_default_settings(message['user_id'])
    analytics.track_signup(message['user_id'])
    s3.create_user_bucket(message['user_id'])

Email service down?
Message stays in queue.
Retried automatically.
User already signed up successfully.

Core Concepts

Producer: Creates messages, Your app server. Your API.

Consumer (Worker): Picks up and processes messages. Separate process.

Message: Unit of work. Usually JSON.

Queue:  Buffer between producer and consumer.

Persistent: survives restarts.

Acknowledgement (ACK): Consumer tells queue: "processed successfully." Queue removes message.

No ACK → queue retries.

def process_message(message):
    try:
        send_email(message)
        message.ack()    # done ✅
    except Exception:
        message.nack()   # failed, retry ❌

What Happens Without ACK
Consumer picks up message.
Starts processing.
Consumer crashes mid-way.
Without ACK system:
message gone.
work never completed.
With ACK system:
queue notices: no ACK received.
message put back.
another consumer picks it up.
work eventually completes.
This is why ACK exists.

Can Critical Items Go to Queue?

Important question. Answer: it depends.

What Makes Something Critical:
Must happen before response     → synchronous
User needs it immediately       → synchronous
Financial reservation           → synchronous
Data consistency required now   → synchronous

Can happen later                → queue
User doesn't need it now        → queue
Eventual consistency ok         → queue

Real example order placement

def place_order(request):
    with db.transaction():
        # SYNCHRONOUS — user needs these now
        validate_payment(request.payment)    # must work now
        reserve_inventory(request.items)     # must work now
        charge_payment(request.payment)      # must work now
        order = create_order(request)        # must work now

        # write to outbox (same transaction)
        write_to_outbox([
            {'event': 'order.confirmation_email'},
            {'event': 'order.warehouse_notify'},
            {'event': 'order.analytics'},
        ])

    # fast response to user
    return {'order_id': order.id, 'status': 'confirmed'}

    # background → email, warehouse, analytics

Payment = synchronous.
Notifications = queue.

The Outbox Pattern

Critical async work needs special handling.
What if queue crashes after payment?
Message lost.
User charged. No order created.
Disaster.
Solution Outbox Pattern:
Write to database AND outbox table in same transaction.

with db.transaction():
    order = db.save_order(data)

    # same transaction
    db.save_outbox_message({
        'event': 'order.created',
        'order_id': order.id
    })
    # if transaction fails → both rolled back
    # if transaction succeeds → both saved

Separate process publishes outbox messages to queue:

while True:
    messages = db.get_pending_outbox_messages()
    for message in messages:
        queue.publish(message.event, message.data)
        db.delete_outbox_message(message.id)

Database and queue always in sync.
Message never lost.
Even if queue temporarily down.

The Decision Framework

Does user need result immediately?
YES → synchronous
NO  → queue

If this fails, does whole operation fail?
YES → synchronous
NO  → queue with retry

Is financial reservation involved?
YES → synchronous + DB transaction
NO  → queue

Can money or inventory be duplicated?
YES (cannot) → synchronous + outbox pattern
NO           → queue is fine

Simple rule:

Synchronous:
→ payment, auth, inventory reservation
→ anything user needs RIGHT NOW

Queue:
→ emails, SMS, push notifications
→ analytics and logging
→ search indexing
→ generating reports
→ cache invalidation
→ webhooks

Popular Message Queues

RabbitMQ

• Traditional message broker.
• Complex routing support.
• Good for: task queues, fan-out.

AWS SQS

• Managed. No infrastructure.
• Scales automatically.
• Good for: AWS systems, serverless.

Redis (as queue)

Simple. Already in your stack.
Good for: low volume, simple tasks.

Kafka

• Not just a queue.
• Event streaming platform.
• Messages stored permanently.
• Multiple consumers read same messages independently.
• High throughput.

More on Kafka soon.

Patterns

Task Queue

Work distributed across workers.
Each message processed by ONE worker.
Producer → [Queue] → Worker1
                  → Worker2
                  → Worker3
Used for: emails, image processing, reports.

Pub/Sub

One producer. Multiple consumers.
Each gets a copy.
Producer → [Topic] → Consumer A (email)
                  → Consumer B (analytics)
                  → Consumer C (notifications)
Used for: event broadcasting, microservices.

Dead Letter Queue (DLQ)

Message fails repeatedly.
Move to DLQ instead of losing it.
[Main Queue] → Consumer (fails 3 times)
                    ↓
             [Dead Letter Queue]
                    ↓
             investigate / fix / retry

Never lose a failed message.

Fan-out

One event → multiple queues → multiple consumers.
Order placed
    ↓
SNS Topic
    ↓          ↓            ↓
Email Queue  SMS Queue  Analytics Queue
    ↓          ↓            ↓
Email Worker SMS Worker Analytics Worker
Each service independent.
One slow → doesn't affect others.

At Least Once vs Exactly Once

At Least Once:

Message delivered at least once.
Might be delivered multiple times.
Network hiccup:
Consumer processes message.
ACK not received.
Queue retries.
Consumer processes again.
User gets two welcome emails.
Fix - idempotent consumer:

Exactly Once:

Message delivered exactly once.
Hard to achieve.
Kafka supports with transactions.
Most systems: at-least-once + idempotent consumers.

Message Queues in Real Systems

E-commerce:

Order placed → queue → inventory update
                    → payment processing
                    → shipping notification
                    → analytics

Social media:

Post created → queue → notify followers
                    → update feed
                    → index for search
                    → content moderation

Ride sharing:

Trip completed → queue → process payment
                      → update driver earnings
                      → send rating request
                      → update analytics

Benefits

Decoupling:

Without queue: Email service down → signup fails

With queue: Email service down → message waits → signup succeeds → email sent when recovered

Buffering:

10,000 signups in 1 minute. Email service handles 100/minute.

Without queue: email service overwhelmed
With queue:    messages wait, processed at own pace

Scalability:

Queue growing too fast? Add more consumers. They automatically pick up messages.

Retry:

Processing fails? Message goes back to queue. Retried automatically.

Mental Model

Producer       → creates messages
Consumer       → processes messages
Queue          → buffer between them
ACK            → confirms successful processing
NACK           → signals failure, triggers retry
DLQ            → failed messages after max retries
Decoupling     → producer and consumer independent
Buffering      → absorbs traffic spikes
Outbox pattern → write DB + queue atomically
At least once  → might process twice
Exactly once   → complex, Kafka transactions
Idempotency    → safe to process twice
RabbitMQ       → task queues, routing
Kafka          → event streaming, replay
SQS            → managed, AWS ecosystem
Pub/Sub        → one message, many consumers
Fan-out        → one event, multiple queues

Synchronous    → payment, auth, inventory
Queue          → everything else

What's Coming Next

Now your system handles failures gracefully.
But what about protecting it from too much traffic?
100,000 requests per second hitting your API.
Your server can handle 1,000.
What do you do?
Rate Limiting.
How to protect your system
from being overwhelmed.
By accident or by attack