Engineering6 min read2026-10-29

How Out-of-Order Shopify Webhooks Silently Corrupt App Databases

Shopify guarantees at-least-once delivery for webhooks but makes no promises about chronological order. Relying on basic application-level upserts guarantees race conditions during bulk merchant updates.

V
Varsion Studio
Engineering Team

A merchant installs your B2B wholesale app and decides to bulk-edit the tags on 15,000 products using Shopify's admin interface. Ten minutes later, support tickets start rolling in. Custom pricing rules aren't applying correctly because the product tags in your database don't match the tags in Shopify.

You check the logs. Your webhook endpoints returned HTTP 200 OK for every single products/update payload. The background workers processed the queue without throwing any exceptions.

Yet, your database is holding stale data.

This is the silent database corruption that plagues almost every scaling Shopify app. It happens because developers inherently trust the network to deliver events chronologically. In distributed systems, relying on webhook arrival time is a guaranteed path to data inconsistency.

The Illusion of Chronology

Under the hood, Shopify routes webhooks through massive distributed message brokers (like AWS EventBridge or Google Cloud Pub/Sub) before they hit your endpoints. These systems guarantee at-least-once delivery. They do not guarantee exactly-once delivery, nor do they guarantee FIFO (First-In, First-Out) ordering.

When a merchant triggers a bulk update, Shopify fires thousands of webhooks concurrently. Here is what network latency and worker queuing can do to your timeline:

Real-World Time Merchant Action Webhook Fired Webhook Arrives at Your App
10:00:01.100 Adds tag "B2B" v1 (Has "B2B") 10:00:03.500
10:00:01.400 Adds tag "VIP" v2 (Has "B2B", "VIP") 10:00:02.100

Notice the arrival times. Webhook v2 arrived a full second before v1.

If your application logic blindly upserts the incoming payload into PostgreSQL, worker A processes v2 and writes both tags. A second later, worker B processes v1 and overwrites the row, erasing the "VIP" tag. The merchant sees the tag in Shopify, but your app has permanently lost it.

The Naive ORM Trap

Most teams start by writing a standard upsert using an ORM like Prisma or TypeORM. The code looks clean and logical:

// ❌ DANGEROUS: Blind upsert vulnerable to out-of-order payloads
export async function handleProductUpdate(webhookPayload: ShopifyProduct) {
  await prisma.product.upsert({
    where: { shopifyId: webhookPayload.id },
    create: {
      shopifyId: webhookPayload.id,
      title: webhookPayload.title,
      tags: webhookPayload.tags,
      updatedAt: new Date(webhookPayload.updated_at),
    },
    update: {
      title: webhookPayload.title,
      tags: webhookPayload.tags,
      updatedAt: new Date(webhookPayload.updated_at),
    },
  });
}

To fix the chronology issue, developers often try to implement a "read-then-write" check in the application layer: Fetch the product, check if the webhook's updated_at is newer than the database's updated_at, and only save if true.

This introduces a classic Time-of-Check to Time-of-Use (TOCTOU) race condition. If two webhook workers process v1 and v2 at the exact same millisecond, they both read the old database state, both pass the time check, and whichever database transaction commits last "wins"—even if it's the older data.

Pushing Conditional Logic to PostgreSQL

To solve this, the concurrency check must be atomic. The database itself needs to reject the update if the incoming payload is older than the existing row.

Unfortunately, standard ORM upsert methods (including Prisma's) do not support conditional UPDATE clauses based on the existing row's data. You have to drop down to raw SQL to leverage PostgreSQL's INSERT ... ON CONFLICT with a WHERE condition.

Here is how we handle webhook upserts atomically at Varsion:

// ✅ SAFE: Atomic conditional upsert using raw SQL
export async function handleProductUpdateAtomic(webhookPayload: ShopifyProduct) {
  const incomingUpdatedAt = new Date(webhookPayload.updated_at).toISOString();

  await prisma.$executeRaw`
    INSERT INTO "Product" ("shopifyId", "title", "tags", "updatedAt")
    VALUES (
      ${webhookPayload.id}, 
      ${webhookPayload.title}, 
      ${webhookPayload.tags}, 
      ${incomingUpdatedAt}::timestamp
    )
    ON CONFLICT ("shopifyId") DO UPDATE 
    SET 
      "title" = EXCLUDED."title",
      "tags" = EXCLUDED."tags",
      "updatedAt" = EXCLUDED."updatedAt"
    WHERE "Product"."updatedAt" < EXCLUDED."updatedAt";
  `;
}

This single query eliminates the TOCTOU race condition. If an older webhook arrives late, PostgreSQL attempts the update, evaluates the WHERE clause, finds it false, and safely ignores the stale data without throwing an error.

The Phantom Order Anomaly

Out-of-order delivery doesn't just affect updates to the same record; it affects the entire lifecycle of an entity.

A frequent source of bugs in order-management apps is receiving an orders/updated or orders/paid webhook before the orders/create webhook. If your system expects every update to have a pre-existing order, it will throw a NotFoundError and fail.

If you configure your queue to retry on failure, the worker will just keep failing until the orders/create webhook finally arrives. But what if orders/create was dropped entirely due to a temporary DNS blip? Your app is now permanently stuck, unable to process payment updates for an order it refuses to acknowledge.

We design our database schemas to accept "Skeleton Upserts".

Every webhook—whether it's a create, update, or delete—is treated as a partial state replication. If orders/paid arrives first, we insert a skeleton row containing just the ID, the payment status, and the updated_at timestamp.

-- Creating a skeleton record from an out-of-order 'orders/paid' event
INSERT INTO "Order" ("shopifyId", "financialStatus", "updatedAt")
VALUES (123456789, 'paid', '2026-10-29T10:05:00Z')
ON CONFLICT ("shopifyId") DO UPDATE 
SET 
  "financialStatus" = EXCLUDED."financialStatus",
  "updatedAt" = EXCLUDED."updatedAt"
WHERE "Order"."updatedAt" < EXCLUDED."updatedAt";

When the orders/create webhook eventually arrives, it simply fills in the missing columns (like lineItems and shippingAddress) without overwriting the newer financialStatus we already received.

Dropping the Noise at the Edge

While PostgreSQL can easily handle conditional updates, forcing your database to process thousands of duplicate webhooks during a bulk operation wastes valuable connection pool resources and spikes your RDS CPU.

Shopify provides a unique x-shopify-webhook-id header with every request. Because Shopify guarantees at-least-once delivery, they will often send the exact same webhook (with the same ID) multiple times if they don't receive an immediate 200 OK, or if their internal routing stutters.

We intercept and deduplicate these at the edge using Redis before they ever reach our backend workers or database.

import { Redis } from '@upstash/redis';
import { headers } from 'next/headers';

const redis = Redis.fromEnv();

export async function POST(req: Request) {
  const webhookId = headers().get('x-shopify-webhook-id');
  
  if (!webhookId) {
    return new Response('Missing webhook ID', { status: 400 });
  }

  // SETNX (Set if Not eXists) with a 24-hour TTL
  // Returns 1 if set, 0 if it already existed
  const isNewEvent = await redis.set(`webhook:${webhookId}`, 'processed', {
    nx: true,
    ex: 86400 
  });

  if (!isNewEvent) {
    // We have already queued this exact payload. 
    // Return 200 immediately to satisfy Shopify.
    return new Response('Duplicate webhook acknowledged', { status: 200 });
  }

  // Proceed to verify HMAC and push to background worker queue...
}

By combining Redis edge deduplication (to drop identical retries) with PostgreSQL conditional upserts (to handle chronologically mixed payloads), the application becomes entirely immune to webhook storms.

Designing for Hostile Networks

Building resilient Shopify apps requires abandoning the assumption that APIs behave linearly. The network is fundamentally hostile to state. Webhooks will arrive late, they will arrive twice, and they will arrive out of order.

If your application relies on the sequence in which HTTP requests hit your server to determine the truth of your database, you are ticking down to a data corruption incident. By treating webhooks not as sequential commands, but as timestamped state propositions, you force the database to act as the final arbiter of truth.

#PostgreSQL#Shopify Apps#Webhooks#System Design#Concurrency
Recommended Shopify AppBuilt by Varsion Studio

Smart Backorder Manager: Never lose a sale to a stockout again. Automatically convert out-of-stock SKUs into transparent backorders.

Capture 100% of customer demand during supply chain delays or viral product drops. Smart Backorder Manager dynamically displays estimated restock dates, tags split-fulfillment orders, and keeps customers informed.

Custom Engineering

Need a bespoke headless architecture or custom Shopify Plus build?

Partner directly with our engineering team for custom integrations and sub-second storefronts.

Related Engineering Insights

View All (10)
Architecture7 min read

Migrating 500 Million Shopify Webhook Records Without Dropping a Single Insert

Standard ALTER TABLE statements lock PostgreSQL tables and cause massive webhook ingestion failures during high-traffic merchant events. Using the expand-and-contract pattern ensures schema changes never interrupt live multi-tenant Shopify apps.

Read Article
Engineering6 min read

Beating Shopify Function CPU Limits with Optimized Rust WebAssembly

Shopify restricts custom checkout extensions to strict execution time and memory limits. Replacing JavaScript runtimes with natively compiled Rust WebAssembly prevents timeouts and enables complex B2B pricing algorithms.

Read Article