n8n idempotency failures – prevent duplicate executions on retry

n8n Production Failure Patterns / Leave a Comment

Step by Step Guide to solve n8n idempotency retry failures
Step by Step Guide to solve n8n idempotency retry failures

I shipped a Stripe payment workflow on a Friday. By Monday morning, 11 customers had been charged twice. Support tickets were piling up. The workflow hadn’t crashed – it had done exactly what I built it to do: retry on failure.

The real bug wasn’t in the code. It was that I’d never made the workflow idempotent. This guide exists so you don’t have that Monday.

Who this is for: n8n developers and automation engineers who need reliable, production‑grade workflows that avoid duplicate side‑effects. We cover this in detail in the n8n Production Failure Patterns Guide.

🔍 Debug Decision Tree: Find Your Fix in 60 Seconds

Most devs get stuck here: they know something duplicated, but have no systematic way to pinpoint where the guard failed. Use this tree before doing anything else.

Are you seeing duplicate records / emails / charges?
│
├── YES — Did the same n8n execution run more than once?
│         │
│         ├── YES (check Execution Log: same execution_id ran 2×)
│         │         └── ➜ Webhook re-delivery problem
│         │               Fix: Add idempotency gate BEFORE first side-effect node
│         │               (see Section 2: Webhook Re-delivery below)
│         │
│         └── NO (different execution_ids, same payload)
│                   │
│                   ├── Did a node retry after a timeout / 5xx?
│                   │         └── YES ➜ Missing idempotency key on HTTP Request node
│                   │                   Fix: Section 3.3 — attach Idempotency-Key header
│                   │
│                   └── Did you manually re-run the workflow?
│                             └── YES ➜ No pre-call DB guard in place
│                                       Fix: Section 3.2 — INSERT ON CONFLICT pattern
│
└── NO — Getting "pending" status stuck in idempotency_log?
          │
          ├── Older than 30 min → external API never returned 2xx
          │         Fix: Section 5 — Grafana query to find stale keys
          │
          └── Race condition (two parallel branches wrote same key simultaneously)
                    Fix: Section 3.5 — Circuit Breaker pattern below

Quick Diagnosis

Problem: Re‑running a failed n8n workflow (or a node inside it) writes the same data twice, creates orphan records, or triggers side‑effects such as duplicate emails or payments.

Featured‑snippet solution: Add an idempotency key to every external call, store the key with a status flag, and configure the node’s Retry options to “Skip on success.” This guarantees that a second attempt sees the previous successful execution and aborts without writing again.

1. What “Idempotency” Means in n8n?

If you encounter any n8n partial failure handling issues, resolve them before continuing with the setup.

Term n8n Context Typical Outcome
Idempotent operation A node that can be executed multiple times with the same input and produce the exact same result (no extra side‑effects). Safe retries, no duplicate records.
Non‑idempotent operation Nodes that trigger external actions (e.g., HTTP Request, Send Email, Stripe Charge) that create resources on each call. Duplicate emails, double‑charged payments, extra DB rows.
Idempotency key A deterministic identifier (UUID, hash of payload, or business key) attached to the request. Used by the downstream system to detect repeats.

Note: Many SaaS APIs (Stripe, PayPal, SendGrid) require the key in a specific header (Idempotency‑Key). If the API does not support it, implement a local guard (e.g., DB lock) to achieve the same effect.

2. Common Scenarios That Trigger Idempotency Failures

Scenario Why it fails
Automatic retries (default 3×) on a node that creates a record Each retry repeats the POST request → two rows in DB.
Manual re‑run of a failed workflow The workflow starts from the first node again, replaying all side‑effects.
Parallel branches writing to the same resource Race condition: both branches write before the other sees the commit.
Webhook re‑delivery (e.g., Stripe webhook retried after 30 s) n8n receives the same event twice and forwards it downstream.

How Webhook Re-delivery Creates Duplicates – The Exact Flow

This is where most devs get stuck: Stripe or Shopify retries the webhook because your workflow took too long to respond. n8n treats it as a brand-new execution. No built-in deduplication kicks in.

Stripe                    n8n Webhook               Your Workflow
  │                            │                          │
  │── POST /webhook ──────────>│                          │
  │                            │── triggers execution ───>│
  │                            │                          │── charges card ──> ✅ SUCCESS
  │                            │                          │
  │  (timeout / slow response) │                          │
  │<── 504 Gateway Timeout ────│ │ │ │ │ │ (Stripe retries 30s later)│ │ │── POST /webhook ──────────>│                          │
  │                            │── NEW execution ────────>│
  │                            │                          │── charges card ──> ❌ DUPLICATE

Fix: Respond 200 OK immediately, THEN process.
  → n8n setting: Webhook node → "Respond" → "Immediately"
  → Guard checks idempotency key BEFORE any side-effect node runs.

3. Building an Idempotent n8n Workflow

3.1 Generate a Stable Idempotency Key

Create a Set node that derives a deterministic key from business data:

{
  "name": "Set Key",
  "type": "n8n-nodes-base.set",
  "parameters": {
    "values": [
      {
        "name": "idempotencyKey",
        "value": "={{ $json['orderId'] + '_' + $json['customerId'] }}",
        "type": "string"
      }
    ]
  }
}

Tip: Use a business‑unique attribute (order ID, email) plus a static prefix. For pure randomness, replace the value with {{ $uuid }}.

Production-grade alternative: SHA256 fingerprinting

When you don’t have a stable business key (e.g. processing raw webhook payloads), generate a collision-resistant fingerprint from the payload content instead. Add a Code node immediately after your Webhook node:

// Code node — runs before any side-effect node
const crypto = require('crypto');
const input = $input.first().json;

// Hash only the stable fields — exclude volatile ones like timestamps
const stablePayload = {
  event_type: input.type,
  resource_id: input.data?.object?.id,
  amount:      input.data?.object?.amount
};

const idempotencyKey = crypto
  .createHash('sha256')
  .update(JSON.stringify(stablePayload))
  .digest('hex');

return [{ json: { ...input, idempotencyKey } }];

⚠️ Do not include volatile fields like Date headers, request_id, or random salts in the hash. The key must be identical across all retry attempts for the same logical event.

3.2 Store the Key Before the External Call

Choose a storage backend that matches your load:

Storage option When to use Considerations
n8n DB (SQLite) Small volume, low traffic SQLite locks can deadlock under high concurrency — add a busy_timeout.
External DB (PostgreSQL / MySQL) High‑throughput, multi‑instance n8n Use SERIALIZABLE isolation to avoid phantom reads.
Cache (Redis) Fast look‑ups, TTL‑based expiry TTL must exceed the longest possible retry window.

⚠️ Most devs get stuck here: They write the idempotency key to the DB after the external call succeeds. This is backwards. If the external call times out midway and n8n retries, the key was never written — so the guard misses it and the duplicate fires anyway.

Always write the key with status pending BEFORE the external call.

Insert‑if‑not‑exists (PostgreSQL) — this query creates a log entry only once:

INSERT INTO idempotency_log (key, status, created_at)
VALUES ($1, 'pending', NOW())
ON CONFLICT (key) DO UPDATE
  SET status = EXCLUDED.status
RETURNING status;

If the returned status is completed, skip the downstream call entirely.
If you encounter any n8n long running workflow failures, resolve them before continuing.

What a duplicate-causing retry looks like in the n8n Execution Log

This log pattern is the tell-sign that idempotency is broken — two executions, same input, both showing success:

[2026-01-14 02:47:11] Execution #4821 — STARTED
  Trigger: Webhook POST /stripe-charge
  Payload: { orderId: "ord_9kX2", customerId: "cus_Rp7", amount: 4900 }

[2026-01-14 02:47:12] Node "HTTP Request: Charge Stripe" — SUCCESS
  Response: { id: "ch_3Oa1", status: "succeeded", amount: 4900 }

[2026-01-14 02:47:12] Execution #4821 — FINISHED (1.2s)

────────────────────────────────────────────────────────────────

[2026-01-14 02:47:41] Execution #4822 — STARTED   ← Stripe retry after 30s timeout
  Trigger: Webhook POST /stripe-charge
  Payload: { orderId: "ord_9kX2", customerId: "cus_Rp7", amount: 4900 }   ← SAME PAYLOAD

[2026-01-14 02:47:42] Node "HTTP Request: Charge Stripe" — SUCCESS
  Response: { id: "ch_3Oa2", status: "succeeded", amount: 4900 }   ← SECOND CHARGE 💥

[2026-01-14 02:47:42] Execution #4822 — FINISHED (1.1s)

If you see two FINISHED executions with identical payloads and different ch_ IDs, your guard is either missing or placed after the charge node — not before it.

3.3 Attach the Key to the External Call

Map the key to the header expected by the target API:

API Header name n8n node field
Stripe Idempotency-Key Headers → Idempotency-Key
SendGrid X-Message-Id Headers → X-Message-Id
Custom REST Idempotency-Key (or custom) Headers → Idempotency-Key

Add the header inside the HTTP Request node configuration:

{
  "name": "Create Order",
  "type": "n8n-nodes-base.httpRequest",
  "parameters": {
    "url": "https://api.example.com/orders",
    "method": "POST",
    "jsonParameters": true,
    "options": {
      "bodyContentType": "json",
      "bodyParametersJson": "={{ $json }}",
      "headerParametersJson": "{\"Idempotency-Key\":\"{{ $json.idempotencyKey }}\"}"
    }
  }
}

3.4 Mark Success / Failure After the Call

After the request returns, use a Set node to update the log record:

Outcome Action
2xx (success) Update idempotency_log.status = 'completed'.
4xx / 5xx (error) Keep status = 'pending' so the next retry can attempt it again.

3.5 Circuit Breaker — Stop the Cascade Before It Starts

Idempotency guards protect individual requests. But when your downstream API goes down entirely, n8n keeps retrying every queued execution — burning budget and flooding a dead service. A circuit breaker detects this and routes executions to a dead-letter queue before the thundering-herd failure begins.

Add a Code node before your HTTP Request node:

// Circuit Breaker — place this Code node before any external API call
const service          = 'stripe_api';
const failureThreshold = 3;       // open circuit after 3 failures
const windowMs         = 300000;  // 5-minute rolling window

// Read recent failures from n8n static data (persists across executions)
const state = $getWorkflowStaticData('global');
const now   = Date.now();

// Remove failures that have aged out of the window
state[service] = (state[service] || []).filter(t => now - t < windowMs); if (state[service].length >= failureThreshold) {
  // Circuit is OPEN — skip the API call
  return [{
    json: {
      circuit:     'open',
      action:      'dead_letter_queue',
      retryAfter:  'manual_review',
      message:     `${service} circuit open — ${state[service].length} failures in last 5 min`
    }
  }];
}

// Circuit is CLOSED — safe to proceed
return [{ json: { ...$input.first().json, circuit: 'closed' } }];

Then add an IF node immediately after: if circuit = 'open' → route to a Slack alert and stop. If circuit = 'closed' → proceed to the HTTP Request node with the idempotency key attached.

4. Safe Retry Configuration in n8n

Follow these four steps inside each side-effect node:

  1. Open the node → click the Retry tab.
  2. Set Maximum Retries to 5.
  3. Set Retry Strategy to Exponential Backoff.
  4. Enable “Skip on Success (Idempotent)” — n8n stops further retries once the node reports success and the guard confirms completed.

Retry Backoff Matrix

Retry # Backoff (ms) Notes
1 0 Immediate first attempt
2 500 After 0.5 s
3 1,500 After 1.5 s
4 3,500 After 3.5 s
5 7,500 Final attempt

Warning: Exponential backoff can cause a thundering herd when many identical workflows fail simultaneously (e.g., during a downstream API outage). Mitigate by adding a random jitter of ±200 ms to the backoff formula.

5. Detecting & Alerting on Idempotency Failures

If you encounter any n8n silent failures with no logs, resolve those before setting up alerting here.

Tool Metric to watch Alert condition
n8n Execution Log status = "error" on node “Create Order” Send Slack alert if > 3 errors in 5 min.
Prometheus Exporter n8n_node_retry_total{node="Create Order", result="failed"} Alert if rate > 0.2 rps.
Datadog idempotency_log.status = "pending" older than 30 min Trigger incident automatically.

Grafana dashboard query — surface pending keys older than 30 minutes:

SELECT
  key,
  status,
  DATE_PART('epoch', NOW() - created_at) AS age_seconds
FROM idempotency_log
WHERE status      = 'pending'
  AND age_seconds > 1800;

One‑Line Fix for Duplicate Writes

Add an idempotency key, store it in a log table with a unique constraint, and set each node’s retry policy to “Skip on Success.” This guarantees that a second execution sees the key marked completed and aborts without re‑sending the external request.

⚡ If This → Do This: Quick-Fix Reference

Bookmark this table. When a duplicate incident hits at 2 AM, you want the fix in 30 seconds — not after re-reading the whole guide.

If you see this… The cause is… Do this immediately
Two executions, same payload, both show success Webhook re-delivery — no idempotency gate Add Code node with SHA256 key + DB guard as the first node after Webhook
One execution shows error then success on retry Auto-retry fired without idempotency header Add Idempotency-Key header to HTTP Request node using a stable business key
idempotency_log rows stuck in pending > 30 min External API returned non-2xx; key never marked completed Run Grafana query from Section 5, investigate API error, manually mark or retry
Duplicate rows from two parallel branches Race condition — both branches wrote before either committed Use PostgreSQL SERIALIZABLE isolation + ON CONFLICT DO NOTHING in both branches
Entire downstream API is down; executions piling up No circuit breaker — n8n retrying a dead service Add circuit breaker Code node (Section 3.5) — route to dead-letter queue until API recovers
Manual re-run creates duplicates Log was cleared between runs; guard missed the completed row Never truncate idempotency_log without archiving. Keep rows for at least 72 hours.

6. Frequently Asked Questions

Question Answer
Do I need to update every node? Only side‑effect nodes (HTTP Request, Send Email, Database Write). Pure transformation nodes are already idempotent by nature.
What if the external API has no idempotency support? Implement a local guard: check the log before the call and wrap it in a DB transaction that rolls back on failure.
Can I reuse the same key across different workflows? Yes, if the key represents a business entity (e.g., order ID). Ensure the uniqueness scope matches the downstream system’s expectations.
How does n8n’s “Execute Workflow” node handle retries? It inherits the parent workflow’s retry settings. Wrap it with the same idempotency guard to prevent nested duplicate runs.

Prepared by the senior SEO & n8n technical team — authoritative, production‑ready guidance for eliminating idempotency failures.

Must Read

Leave a Comment

Your email address will not be published. Required fields are marked *