n8n idempotency failures – prevent duplicate executions on retry

 

Who this is for: n8n developers and automation engineers who need reliable, production‑grade workflows that avoid duplicate side‑effects. We cover this in detail in the n8n Production Failure Patterns Guide

Quick Diagnosis

Problem: Re‑running a failed n8n workflow (or a node inside it) writes the same data twice, creates orphan records, or triggers side‑effects such as duplicate emails or payments.

Featured‑snippet solution: Add an idempotency key to every external call, store the key with a status flag, and configure the node’s Retry options to “Skip on success.” This guarantees that a second attempt sees the previous successful execution and aborts without writing again.

1. What “Idempotency” Means in n8n?

If you encounter any n8n partial failure handling resolve them before continuing with the setup.

Term	n8n Context	Typical Outcome
Idempotent operation	A node that can be executed multiple times with the same input and produce the exact same result (no extra side‑effects).	Safe retries, no duplicate records.
Non‑idempotent operation	Nodes that trigger external actions (e.g., HTTP Request, Send Email, Stripe Charge) that create resources each call.	Duplicate emails, double‑charged payments, extra DB rows.
Idempotency key	A deterministic identifier (UUID, hash of payload, or business key) attached to the request.	Used by the downstream system to detect repeats.

EEFA note: Many SaaS APIs (Stripe, PayPal, SendGrid) require the key in a specific header (Idempotency‑Key). If the API does not support it, implement a local guard (e.g., DB lock) to achieve the same effect.

2. Common Scenarios That Trigger Idempotency Failures

Scenario	Why it fails
Automatic retries (default 3×) on a node that creates a record	Each retry repeats the POST request → two rows in DB.
Manual re‑run of a failed workflow	The workflow starts from the first node again, replaying all side‑effects.
Parallel branches writing to the same resource	Race condition: both branches write before the other sees the commit.
Webhook re‑delivery (e.g., Stripe webhook retried after 30 s)	n8n receives the same event twice and forwards it downstream.

3. Building an Idempotent n8n Workflow

3.1 Generate a Stable Idempotency Key

Create a Set node that derives a deterministic key from business data:

{
  "name": "Set Key",
  "type": "n8n-nodes-base.set",
  "parameters": {
    "values": [
      {
        "name": "idempotencyKey",
        "value": "={{ $json['orderId'] + '_' + $json['customerId'] }}",
        "type": "string"
      }
    ]
  }
}

*Tip:* Use a business‑unique attribute (order ID, email) plus a static prefix. For pure randomness, replace the value with {{$uuid}}.

3.2 Store the Key Before the External Call

Choose a storage that matches your load:

Storage option	When to use	EEFA considerations
n8n DB (SQLite)	Small volume, low latency	SQLite locks can cause dead‑locks under high concurrency – add a busy_timeout.
External DB (PostgreSQL, MySQL)	High‑throughput, multi‑instance n8n	Use SERIALIZABLE isolation to avoid phantom reads.
Cache (Redis)	Fast look‑ups, TTL‑based expiry	TTL must exceed the longest possible retry window.

Insert‑if‑not‑exists (PostgreSQL) – this query creates a log entry only once:

INSERT INTO idempotency_log (key, status, created_at)
VALUES ($1, 'pending', NOW())
ON CONFLICT (key) DO UPDATE
SET status = EXCLUDED.status
RETURNING status;

If the returned status is completed, skip the downstream call. If you encounter any n8n long running workflow failures resolve them before continuing with the setup.

3.3 Attach the Key to the External Call

Map the key to the header expected by the target API:

API	Header name	n8n node field
Stripe	Idempotency-Key	Headers → Idempotency-Key
SendGrid	X-Message-Id	Headers → X-Message-Id
Custom REST	Idempotency-Key (or custom)	Headers → Idempotency-Key

Add the header in an HTTP Request node:

{
  "name": "Create Order",
  "type": "n8n-nodes-base.httpRequest",
  "parameters": {
    "url": "https://api.example.com/orders",
    "method": "POST",
    "jsonParameters": true,
    "options": {
      "bodyContentType": "json",
      "bodyParametersJson": "={{ $json }}",
      "headerParametersJson": "{\"Idempotency-Key\":\"{{ $json.idempotencyKey }}\"}"
    }
  }
}

3.4 Mark Success / Failure

After the request, use a Set node to update the log:

Outcome	Action
2xx (success)	Update idempotency_log.status = 'completed'.
4xx/5xx (error)	Keep status = 'pending' so the next retry can try again.

4. Safe Retry Configuration in n8n

1. Open the node → Retry tab.
2. Set Maximum Retries (e.g., 5).
3. Choose Retry Strategy = Exponential Backoff.
4. Enable “Skip on Success (Idempotent)” – n8n stops further retries once the node reports success and the idempotency guard reports completed.

Retry Matrix

Retry Count	Backoff (ms)	When to abort automatically
1	0	Immediate first attempt
2	500	After 0.5 s
3	1 500	After 1.5 s
4	3 500	After 3.5 s
5	7 500	After 7.5 s (final)

EEFA warning: Exponential backoff can cause a thundering herd when many identical workflows fail simultaneously (e.g., downstream API outage). Mitigate by adding a random jitter (±200 ms) to the backoff formula.

5. Detecting & Alerting on Idempotency Failures

If you encounter any n8n silent failures no logs resolve them before continuing with the setup.

Tool	Metric	Alert Condition
n8n Execution Log	status = “error” & node = “Create Order”	Send Slack alert if > 3 errors in 5 min.
Prometheus Exporter	n8n_node_retry_total{node=”Create Order”,result=”failed”}	Alert on rate > 0.2 rps.
External Monitoring (Datadog)	idempotency_log.status = “pending” older than 30 min	Trigger incident.

Grafana dashboard snippet – list pending keys older than 30 min:

SELECT
  key,
  status,
  DATE_PART('epoch', NOW() - created_at) AS age_seconds
FROM idempotency_log
WHERE status = 'pending'
  AND age_seconds > 1800;

One‑Line Fix for Duplicate Writes

Add an idempotency key, store it in a log table with a unique constraint, and set each node’s retry policy to “Skip on Success.” This guarantees that a second execution sees the key marked completed and aborts without re‑sending the external request.

6. Frequently Asked Questions

Question	Answer
Do I need to change every node?	Only side‑effect nodes (HTTP Request, Send Email, Database Write). Pure transformation nodes are already idempotent.
What if the external API has no idempotency support?	Implement a local guard: check the log before the call and use a DB transaction that rolls back on failure.
Can I reuse the same key across different workflows?	Yes, if the key represents a business entity (e.g., order ID). Ensure the uniqueness scope matches the downstream system.
How does n8n’s built‑in “Execute Workflow” node handle retries?	It inherits the parent workflow’s retry settings. Wrap it with the same idempotency guard to avoid nested duplicate runs.