Preventing Split-Brain Checkouts Across Stripe, Shopify, and Custom ERPs

When enterprise brands decouple their storefront from a single monolith, checkout stops being a single database transaction. A typical composable purchase touches at least three external systems: Shopify Plus for order records, Stripe or Adyen for payment capture, and an external ERP or WMS for warehouse allocation.

That separation works great until network weather hits at the bottom of the funnel.

If a shopper submits an order and the payment gateway captures funds, but the warehouse reservation call times out after 5 seconds, you have a split-brain checkout. The customer has a credit card charge on their phone, while operations has no order in the WMS. Wrapping three synchronous HTTP calls in a basic try/catch block won't save you if the Node.js worker restarts mid-flight or a retry double-charges the card.

Here is how we use Orchestrated Sagas and edge-enforced idempotency keys to make multi-service checkouts self-healing.

The Problem: The Fallacy of Two-Phase Commits

In a traditional relational database, we ensure consistency using ACID transactions. If one step fails, the entire transaction rolls back. In a distributed headless architecture, we interact with external HTTP APIs (Shopify Storefront/Admin APIs, Stripe, proprietary ERPs). These systems do not share a database, and they do not support the Two-Phase Commit (2PC) protocol.

To maintain consistency without distributed locks, we must embrace Eventual Consistency and the Saga Pattern.

The Saga Pattern in E-Commerce

A Saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction in the saga. If a local transaction fails because it violates a business rule, the saga executes a series of compensating transactions that undo the changes made by the preceding local transactions.

For checkout orchestration, we favor Command/Orchestration over Choreography. A central Orchestrator (a Node.js microservice backed by PostgreSQL) manages the state machine of the checkout process.

sequenceDiagram
    participant Client as Next.js Client
    participant API as Checkout Orchestrator (Node.js)
    participant Stripe as Payment Gateway
    participant ERP as Inventory (WMS)
    participant Shopify as Commerce Engine

    Client->>API: POST /checkout (Idempotency-Key)
    API->>Stripe: 1. Charge Card
    Stripe-->>API: Success (Charge ID)
    API->>ERP: 2. Reserve Inventory
    alt Inventory Failed
        ERP-->>API: 500 Internal Error
        API->>Stripe: 3. COMPENSATE: Refund Charge
        Stripe-->>API: Refunded
        API-->>Client: 409 Conflict (Out of Stock)
    else Inventory Succeeded
        ERP-->>API: Success (Reservation ID)
        API->>Shopify: 3. Create Order
        Shopify-->>API: Success (Order ID)
        API-->>Client: 200 OK (Order Confirmed)
    end

Step 1: Enforcing Edge-Aware Idempotency

Before we can safely retry or compensate transactions, our orchestration API must be strictly idempotent. If a user on a flaky 5G connection taps "Pay Now" twice, we cannot charge them twice.

We implement idempotency at the API gateway/middleware level using Redis. When a request arrives, we check the Idempotency-Key header.

// middleware/idempotency.ts
import { Redis } from 'ioredis';
import { Request, Response, NextFunction } from 'express';
import crypto from 'crypto';

const redis = new Redis(process.env.REDIS_URL);

export const idempotencyMiddleware = async (req: Request, res: Response, next: NextFunction) => {
  const idempotencyKey = req.headers['x-idempotency-key'];
  if (!idempotencyKey) {
    return res.status(400).json({ error: 'x-idempotency-key header is required' });
  }

  const cacheKey = `idempotency:${idempotencyKey}`;
  const cachedState = await redis.get(cacheKey);

  if (cachedState) {
    const { status, response, isProcessing } = JSON.parse(cachedState);
    
    // If another request is currently processing this key, return a 409 Conflict 
    // or hold the connection (polling) depending on UX requirements.
    if (isProcessing) {
      return res.status(409).json({ error: 'Request already in progress' });
    }
    
    // Return the cached response of the successful original request
    return res.status(status).json(response);
  }

  // Mark as processing with a TTL to prevent deadlocks if the server crashes
  await redis.set(cacheKey, JSON.stringify({ isProcessing: true }), 'EX', 60);

  // Capture the original res.json to intercept the response and cache it
  const originalJson = res.json.bind(res);
  res.json = ((body: any) => {
    redis.set(
      cacheKey, 
      JSON.stringify({ isProcessing: false, status: res.statusCode, response: body }), 
      'EX', 
      86400 // Store for 24 hours
    );
    return originalJson(body);
  }) as any;

  next();
};

Step 2: Modeling the Saga State Machine in PostgreSQL

The Orchestrator needs a persistent state machine. If the Node.js process crashes halfway through a checkout, a background worker must be able to pick up the pending saga and either complete it or run compensations.

We use PostgreSQL to track the state of the saga.

-- schema.sql
CREATE TYPE saga_status AS ENUM (
  'PENDING', 
  'PAYMENT_SECURED', 
  'INVENTORY_RESERVED', 
  'ORDER_CREATED', 
  'COMPENSATING', 
  'FAILED', 
  'COMPLETED'
);

CREATE TABLE checkout_sagas (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key VARCHAR(255) UNIQUE NOT NULL,
    status saga_status NOT NULL DEFAULT 'PENDING',
    cart_payload JSONB NOT NULL,
    stripe_charge_id VARCHAR(255),
    erp_reservation_id VARCHAR(255),
    shopify_order_id VARCHAR(255),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

Step 3: Implementing the Orchestrator and Compensating Transactions

The core checkout function executes the steps sequentially. If any step throws an exception, the catch block triggers the compensate() routine.

Notice how we update the PostgreSQL database before making the external API calls. This write-ahead logging (WAL) approach ensures that if the process dies, our database accurately reflects the last attempted state.

// services/checkoutOrchestrator.ts
import { db } from '../db';
import { stripeAPI, erpAPI, shopifyAPI } from '../external-clients';

export async function processCheckoutSaga(idempotencyKey: string, cart: CartPayload) {
  // 1. Initialize Saga
  let saga = await db.checkout_sagas.create({
    data: { idempotency_key: idempotencyKey, cart_payload: cart }
  });

  try {
    // 2. Execute Payment
    const charge = await stripeAPI.charge(cart.total, cart.paymentMethodId, idempotencyKey);
    saga = await updateSagaStatus(saga.id, 'PAYMENT_SECURED', { stripe_charge_id: charge.id });

    // 3. Reserve Inventory
    const reservation = await erpAPI.reserve(cart.items, idempotencyKey);
    saga = await updateSagaStatus(saga.id, 'INVENTORY_RESERVED', { erp_reservation_id: reservation.id });

    // 4. Create Shopify Order
    const order = await shopifyAPI.createOrder(cart, idempotencyKey);
    saga = await updateSagaStatus(saga.id, 'COMPLETED', { shopify_order_id: order.id });

    return { success: true, orderId: order.id };

  } catch (error) {
    // 5. Orchestrate Compensations on Failure
    await updateSagaStatus(saga.id, 'COMPENSATING');
    await executeCompensations(saga);
    await updateSagaStatus(saga.id, 'FAILED');
    
    throw new Error(`Checkout failed: ${error.message}. All systems rolled back safely.`);
  }
}

async function executeCompensations(saga: SagaRecord) {
  // Compensate in reverse order of execution
  
  if (saga.erp_reservation_id) {
    try {
      await erpAPI.releaseInventory(saga.erp_reservation_id);
    } catch (err) {
      // Log critical alert. In a real system, push this to a Dead Letter Queue (DLQ)
      // for manual intervention or background retry.
      console.error('CRITICAL: Failed to release inventory', err);
    }
  }

  if (saga.stripe_charge_id) {
    try {
      await stripeAPI.refund(saga.stripe_charge_id);
    } catch (err) {
      console.error('CRITICAL: Failed to refund payment', err);
    }
  }
}

Where Teams Get Burned in Production

A textbook Saga diagram looks clean on a whiteboard, but production failure modes happen in the edges between steps.

1. Dead Letter Queues (DLQ) When Compensations Fail

What happens if the compensating transaction itself fails? Suppose the Shopify Order API returns a 503, your orchestrator calls stripeAPI.refund(), and Stripe's endpoint times out too.

Never swallow a failed compensation in a console.error. Failed rollbacks must be written to a persistent retry queue (or polled directly from checkout_sagas WHERE status = 'COMPENSATING' AND updated_at < NOW() - INTERVAL '2 minutes') with exponential backoff. If a saga stays stuck in COMPENSATING after 5 retries, page on-call and surface the record in an internal ops dashboard.

2. Rejecting Bad Payloads Before Touching Postgres

Running a multi-step saga takes 400ms–1,200ms because it waits on third-party round trips. To keep malformed requests and bots from tying up Postgres connections, validate structural rules at the edge before forwarding to the orchestrator:

• Reject missing or malformed x-idempotency-key headers immediately.
• Verify HMAC cart signatures at the CDN worker so tampered quantities never start a saga.
• Short-circuit known out-of-stock SKUs from edge cache before creating a PENDING saga row.

3. Correlating Logs Across Vendor Boundaries

When finance asks why a specific order was refunded 800ms after capture, grepping unstructured logs across three services is painful. Pass a deterministic correlation ID—derived from the idempotency_key—into Stripe's metadata.saga_id, the ERP's reference field, and Shopify's note_attributes. Having the exact same UUID recorded inside all three third-party dashboards turns a 45-minute reconciliation hunt into a single search.

For merchants running native Shopify checkouts who want plug-and-play order protection and automated stockout handling without building custom middleware, check out **CheckoutShield** and **Smart Backorder Manager**.