n8n queue mode duplicate execution – why workflows run twice and how to fix it

n8n Queue Mode Errors / Leave a Comment
Step by Step Guide to solve n8n queue mode duplicate execution
Step by Step Guide to solve n8n queue mode duplicate execution

Who this is for: n8n administrators and workflow engineers who see the same workflow fire multiple times for a single trigger and need a reliable, production‑grade solution. We cover this in detail in the n8n Queue Mode Errors Guide.

Quick Diagnosis

Symptom: One trigger generates two or more identical workflow runs, causing duplicate records or API calls.

Root cause: The queue worker processes the same executionId more than once because the queue is mis‑configured (e.g., queueMode set to parallel with too many concurrent workers, or the Redis lock TTL is too short).

One‑line fix: Switch to queueMode: "fifo", use a short execution lock (executionTimeout: 30s), limit concurrency, and add a deduplication guard that checks the executionId before proceeding.

Troubleshooting Decision Tree (Fix Faster)

Use this tree to find the right fix within 5 minutes rather than reading the entire article top‑to‑bottom.

Symptom First check Fix section
Workflow runs exactly N times = number of instances Worker logs: “Start Active Workflows”? Section 2a: add worker‑only env vars
Schedule trigger duplicating after n8n update Did you upgrade n8n recently? Section 2a: delete & recreate cron node
Webhook triggers 2–3× from external system Does provider have an event ID header? Section 5a: payload idempotency gate
Random duplicates under high load redis-cli TTL n8n:queue:lock:<id> < execution time? Section 3.1: raise executionTimeout
Sub‑workflow called far more times than items Execute Workflow node mode = per‑item? Section 2a: set mode to “once”
Duplicates only appear after Redis restart Is Redis using AOF persistence? Section 3.1: enable --appendonly yes
Duplicates in production but not staging Are main and worker image tags identical? Section 3a: pin both to same version

1. n8n Queue Mode Basics

If you encounter any n8n queue mode job stuck resolve them before continuing with the setup.

Setting Recommended for deduplication
queueMode fifo – guarantees first‑in‑first‑out ordering, eliminating race conditions.
maxConcurrentExecutions 1‑2 per workflow – prevents multiple workers from grabbing the same job.
executionTimeout 30‑45s – keeps the Redis lock alive for the whole run.
redisLockKeyPrefix Keep default (n8n:queue) but ensure it’s unique per instance.

EEFA note: In production always use a dedicated Redis instance with persistence. An in‑memory store loses locks on restart, instantly spawning duplicates.

<!-- ========================================================= NEW SECTION: TREE FOR QUICK FIX ==========

5d. Troubleshooting Decision Tree

Use this tree to find the right fix within 5 minutes rather than reading the entire article top‑to‑bottom.

Symptom First check Fix section
Workflow runs exactly N times = number of instances Worker logs: "Start Active Workflows"? Section 2a — add worker‑only env vars
Schedule trigger duplicating after n8n update Did you upgrade n8n recently? Section 2a — delete & recreate cron node
Webhook triggers 2–3× from external system Does provider have an event ID header? Section 5a — payload idempotency gate
Random duplicates under high load redis-cli TTL n8n:queue:lock:<id> < execution time? Section 3.1 — raise executionTimeout
Sub‑workflow called far more times than items Execute Workflow node mode = per‑item? Section 2a — set mode to "once"
Duplicates only appear after Redis restart Is Redis using AOF persistence? Section 3.1 — enable --appendonly yes
Duplicates in production but not staging Are main and worker image tags identical? Section 3a — pin both to same version

2. Why Duplicate Execution Happens?

  1. Lock expiration: If a workflow runs longer than executionTimeout, Redis releases the lock and another worker re‑processes the same executionId.
  2. Parallel dequeue: With queueMode: "parallel" and several concurrent workers, multiple workers may read the same pending job before the lock is written.
  3. Improper Redis configuration: Sharing the default localhost:6379 across clustered nodes creates lock‑key collisions.

2a. Additional Root Causes You May Be Missing

The three causes above are the most common, but production deployments regularly surface four more that are overlooked in most guides – each with a distinct fix path.

Workers Registering Active Workflows

In Docker Compose or Kubernetes setups, worker containers sometimes log “Start Active Workflows” on boot the same banner the main instance emits. When this happens every instance activates its own copy of every trigger, producing one duplicate per extra worker. The root cause is missing or overridden worker‑specific environment variables.

Required worker‑only variables:

# Add these to EVERY worker container — never to the main instance
N8N_DISABLE_ACTIVE_WORKFLOWS=true
N8N_SKIP_WEBHOOK_REGISTRATION_ON_STARTUP=true
N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN=true

EEFA: After adding these variables, redeploy the stack and confirm worker logs no longer contain the “Start Active Workflows” line. If it still appears, the variable is being overridden by a parent .env file or a Kubernetes ConfigMap – check for conflicts.

 

Main Process Also Executing Jobs

By default the main n8n instance acts as a fallback executor: if no workers are available it processes the job itself. In a multi‑worker stack this means both the main process and a worker can pick up the same queued execution, causing the workflow to run twice.

# Main instance only — disables execution on the main process
N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true

EEFA: Only set N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true when you have at least one healthy worker running. Without a worker the queue will stall entirely.

 

Ghost Cron / Schedule Triggers

A known issue in n8n queue mode: after updating n8n or restarting workers, stale cron schedule registrations are not fully cleared from memory. The result is one duplicate execution per extra worker for every scheduled workflow — and the count grows with each restart. Community reports confirm this affects n8n versions up to v1.123.7 and beyond.

Immediate workaround:

  1. Open the affected workflow in the editor.
  2. In the Schedule Trigger node, delete and re‑create the cron rule from scratch.
  3. Save and re‑activate the workflow.

Preventive long‑term fix: After every n8n version upgrade, cycle through all workflows that use a Schedule Trigger by deactivating them, waiting 30 seconds, and reactivating them – rather than simply restarting the containers. This forces a clean re‑registration.

 

Execute Workflow / Sub‑Workflow Fanout

When a parent workflow uses the Execute Workflow node to call a child workflow and passes multiple items without limiting the execution mode, the child can be invoked once per worker that picks up an item batch — far more than intended. A community‑reported case saw 181 items produce 542 child executions for the same parent ID.

Fix – set execution mode on the Execute Workflow node:

{
  "name": "Execute Workflow",
  "type": "n8n-nodes-base.executeWorkflow",
  "parameters": {
    "mode": "once",            // "once" = run once for ALL items, not once PER item
    "workflowId": "{{ $json.childWorkflowId }}"
  }
}

Use Run once with all items when the child workflow should receive the full batch in one execution. Use Run once for each item only when you deliberately want N child executions for N items, and ensure idempotency guards (Section 3.2) protect each one.

3. Step‑by‑Step Fix

3.1 Update the Queue Configuration

Queue settings (JSON) – place these in config.json or the equivalent environment file.

{
  "queueMode": "fifo",
  "maxConcurrentExecutions": 1,
  "executionTimeout": 30
}

Redis connection (JSON) – keep the lock prefix default.

{
  "redis": {
    "host": "redis-prod.mycompany.com",
    "port": 6379,
    "password": "••••••••",
    "tls": true
  }
}

Docker / K8s environment variables – preferred for containerised deployments.

N8N_QUEUE_MODE=fifo
N8N_MAX_CONCURRENT_EXECUTIONS=1
N8N_EXECUTION_TIMEOUT=30
N8N_REDIS_HOST=redis-prod.mycompany.com
N8N_REDIS_TLS=true

EEFA: Restart the n8n service after any change and verify with n8n config:get queueMode.

3.2 Insert a Deduplication Guard

Set node – capture the execution ID

{
  "name": "Set executionId",
  "type": "n8n-nodes-base.set",
  "parameters": {
    "values": [
      {
        "name": "executionId",
        "value": "={{$json[\"$executionId\"]}}"
      }
    ]
  }
}

IF node – abort if the ID was already processed

// Returns true when the ID is *not* in Redis
return await $redis.get(`executed:${$json.executionId}`) !== '1';

If the condition fails, route the flow to a **No‑Op** node that simply ends the workflow.

Function node – mark the ID as completed

await $redis.set(`executed:${$json.executionId}`, '1', 'EX', 86400); // keep for 24 h
return {};

3.3 Clean‑up on Successful Completion

Add the Function node above as the final step of the workflow. It guarantees the execution ID is stored for a day, preventing re‑processing if a lock were to expire later. If you encounter any n8n queue mode missing worker process resolve them before continuing with the setup.

3a. Complete Docker Compose Reference Configuration

The single most common cause of duplicate execution in containerised deployments is a docker-compose.yml that gives workers the same environment variables as the main instance. The reference below cleanly separates main‑only vs worker‑only variables.

version: "3.8"

services:
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes"]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  n8n:
    image: n8nio/n8n:1.50.1          # pin version — never use :latest in production
    restart: unless-stopped
    depends_on:
      redis:
        condition: service_healthy
    environment:
      # ── Shared settings (main + worker must match) ──────────────────
      - EXECUTIONS_MODE=queue
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      # ── Main‑only settings ─────────────────────────────────────────
      - N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true   # prevents main from executing jobs
    ports:
      - "5678:5678"

  n8n-worker:
    image: n8nio/n8n:1.50.1          # must match main image version exactly
    restart: unless-stopped
    depends_on:
      redis:
        condition: service_healthy
    command: ["n8n", "worker", "--concurrency=5"]
    environment:
      # ── Shared settings (must match main) ──────────────────────────
      - EXECUTIONS_MODE=queue
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - QUEUE_BULL_REDIS_HOST=redis
      - QUEUE_BULL_REDIS_PORT=6379
      # ── Worker‑only settings ─────────────────────────────────────
      - N8N_DISABLE_ACTIVE_WORKFLOWS=true
      - N8N_SKIP_WEBHOOK_REGISTRATION_ON_STARTUP=true
      - N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN=true
    deploy:
      replicas: 2

EEFA: Pin both the main and worker images to the same explicit version tag. If the main instance is on v1.50.1 and a worker is on v1.49.x, they may use incompatible job serialisation formats, causing unpredictable re‑queuing and duplicate runs.

Verification after deployment:

# Confirm main is NOT executing jobs itself
docker compose logs n8n | grep "Queue mode"

# Confirm workers are NOT activating workflows
docker compose logs n8n-worker | grep "Start Active"
# Expected: no output (the line should be absent)

# Confirm workers are picking up jobs
docker compose logs n8n-worker | grep "Worker started execution"

3b. Kubernetes Environment Variable Reference

For Kubernetes deployments the same variable split applies — main Deployment vs worker Deployment. The critical difference is that the worker Pod spec must contain the worker‑only variables and must not inherit them from a shared ConfigMap without overrides.

# worker-deployment.yaml (relevant env section only)
env:
  - name: EXECUTIONS_MODE
    value: "queue"
  - name: N8N_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: n8n-secrets
        key: encryption-key
  - name: QUEUE_BULL_REDIS_HOST
    value: "redis-service"
  # Worker‑only — prevents ghost trigger activation
  - name: N8N_DISABLE_ACTIVE_WORKFLOWS
    value: "true"
  - name: N8N_SKIP_WEBHOOK_REGISTRATION_ON_STARTUP
    value: "true"
  - name: N8N_SKIP_WEBHOOK_DEREGISTRATION_SHUTDOWN
    value: "true"
args: ["worker", "--concurrency=5"]

EEFA: If workers are deployed as a StatefulSet (required when using ReadWriteOnce persistent volumes), add a podAntiAffinity rule to spread replicas across nodes. Workers on the same node competing for the same Redis connection pool is a secondary cause of race‑condition duplicates.

4. Monitoring & Preventive Checklist

Check Verification Method
Queue mode is fifo n8n config:get queueMode → fifo
maxConcurrentExecutions ≤ 2 Inspect config.json or env vars
Execution timeout ≤ 45 s n8n config:get executionTimeout
Redis lock TTL matches timeout redis-cli TTL n8n:queue:<executionId> after a run
No duplicate executionId in DB SELECT COUNT(*) FROM executions WHERE execution_id = ?
Alert on > 1 execution per trigger Grafana/Prometheus alert on n8n_queue_duplicate_total metric

EEFA tip: Enable telemetry (N8N_TELEMETRY_ENABLED=true) only in staging; production telemetry can expose execution IDs.

4a. Extended Monitoring: Prometheus Metrics & Log Patterns

The checklist above is a point‑in‑time snapshot. For ongoing duplicate detection you need streaming metrics and log pattern alerts.

Enable n8n Prometheus Metrics

# Add to main instance environment
N8N_METRICS=true
N8N_METRICS_INCLUDE_API_ENDPOINTS=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

This exposes a /metrics endpoint compatible with Prometheus scraping. Key metrics to track for duplicate detection:

Metric What it signals
n8n_executions_total Total executions per workflow — a sudden spike for one workflow ID indicates duplicates.
n8n_queue_jobs_active Jobs currently being processed. More than concurrency × workers indicates a leak.
n8n_queue_jobs_waiting Pending queue depth. A growing backlog with duplicate completions points to lock expiry.
n8n_execution_duration_seconds p99 execution time. If this regularly exceeds executionTimeout, lock expiry duplicates are guaranteed.

Log Pattern to Watch For

Duplicate enqueue events appear in the main process log as consecutive identical execution IDs within the same second:

# Indicates a duplicate — same workflow enqueued 3 times
Enqueued execution 962695 (job 272128)
Enqueued execution 962696 (job 272129)
Enqueued execution 962697 (job 272130)

# Shell command to detect this pattern live
docker compose logs -f n8n | grep "Enqueued execution" | \
  awk '{print $3}' | sort | uniq -d

EEFA: If the above command returns any output, your main process is producing duplicate queue entries before workers even pick them up – this is the ghost trigger scenario from Section 2a, not a Redis lock issue.

 

Redis Queue Depth Check

# Check the BullMQ queue for stuck or duplicated jobs
redis-cli -h redis-prod.mycompany.com KEYS "bull:*" | wc -l

# Inspect a specific job
redis-cli -h redis-prod.mycompany.com HGETALL "bull:jobs:"

# Check active locks
redis-cli -h redis-prod.mycompany.com KEYS "n8n:queue:lock:*"

EEFA: More than one lock key for the same executionId is definitive proof of a duplicate. Clear orphaned locks with redis-cli DEL n8n:queue:lock:<executionId> during a maintenance window.

5. Advanced: External Idempotency Keys

When calling APIs that support idempotency (e.g., Stripe, Salesforce), forward the n8n executionId as the idempotency key:

POST https://api.stripe.com/v1/charges
Idempotency-Key: {{ $json.executionId }}

This adds a second line of defense— the external service itself will reject duplicates.

5a. Payload‑Level Idempotency for Webhook Triggers

When the duplicate originates from the sender — for example, a webhook provider that retries on timeout (Stripe, GitHub, Shopify all use at‑least‑once delivery) — the executionId alone is not enough because each retry receives a brand‑new executionId. You need to key on something in the payload itself.

Step 1 – Extract a stable event key from the payload

{
  "name": "Set idempotency key",
  "type": "n8n-nodes-base.set",
  "parameters": {
    "values": [
      {
        "name": "idempotencyKey",
        "value": "={{ $json.body.id ?? $json.headers['x-github-delivery'] ?? $json.body.event_id }}"
      }
    ]
  }
}

Step 2 – Check Redis before processing

// IF node condition — proceed only when key is unseen
const key = `webhook:idempotency:${$json.idempotencyKey}`;
const seen = await $redis.get(key);
return seen === null; // true = first time, false = retry → route to No-Op

Step 3 – Mark the event as handled

// At the END of successful workflow — store for 48 hours to cover delayed retries
const key = `webhook:idempotency:${$json.idempotencyKey}`;
await $redis.set(key, '1', 'EX', 172800);
return {};

Common payload‑level idempotency keys by provider:

Provider Stable event field
Stripe body.id (e.g., evt_1abc...)
GitHub headers['x-github-delivery']
Shopify headers['x-shopify-webhook-id']
Dropbox body.list_folder.cursor (hash on full payload if absent)
Generic / custom {{ $crypto.sha256($json.body | stringify) }} — hash of the full payload

EEFA: The payload hash approach works for any provider but is computationally more expensive. Use it only when the provider does not supply a native event ID.

5b. Concurrency Control: The Right Numbers

Concurrency is a direct lever for duplicate prevention. Set it wrong and you either starve the queue or create the race conditions that cause duplicates. Here is how to size it correctly.

Workflow type Recommended concurrency per worker Reasoning
I/O‑bound (API calls, webhooks, wait nodes) 10 – 20 CPU is mostly idle; high concurrency keeps the event loop busy.
CPU‑bound (data transformation, code nodes, encryption) 1 – 5 Excess concurrency causes context‑switching overhead that slows all jobs.
Mixed 5 (n8n default) Safe starting point; monitor and tune based on docker stats.

Critical formula – calculate required Postgres connections before setting concurrency:

Required DB connections = (Number of Workers × Worker Concurrency) + Main Process connections

# Example: 3 workers × concurrency 5 + 10 main connections = 25 total
# Set max_connections in postgresql.conf to at least 30 (add ~20% headroom)

EEFA: n8n recommends a minimum concurrency of 5 per worker. Setting it to 1 or 2 with many workers exhausts the database connection pool, which causes execution delays — and those delays can push jobs past their executionTimeout, which then causes lock‑expiry duplicates. Do not use very low concurrency as a deduplication strategy; it creates a different class of problem.

# Set worker concurrency at startup
n8n worker --concurrency=5

# Or via environment variable (takes precedence over --concurrency flag)
N8N_CONCURRENCY_PRODUCTION_LIMIT=5

5c. Frequently Asked Questions

Q: Why does my workflow run exactly 3 times in queue mode?

Running exactly N times (where N equals your number of instances) strongly indicates that every instance — main plus workers — is activating workflows independently. This happens when workers are missing N8N_DISABLE_ACTIVE_WORKFLOWS=true. Each instance activates the trigger and enqueues its own execution. Fix: add the worker‑only variables from Section 2a and redeploy.

Q: My Schedule Trigger was fine until I updated n8n — why is it duplicating now?

This is the ghost cron trigger issue documented in Section 2a. n8n does not always clean up old cron registrations when upgrading. The reliable fix is to delete and recreate the Schedule Trigger node in the affected workflows after every major version upgrade, then save and reactivate.

Q: Does setting N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true break anything?

Only if you have no workers online. With this flag set and zero available workers, jobs pile up in Redis indefinitely because the main instance will not execute them as a fallback. Always confirm at least one worker is healthy (check logs for “Worker started execution”) before enabling this flag in production.

Q: My webhook provider sends the same event twice. How do I stop duplicate processing?

The queue‑level deduplication in Section 3.2 will not help here because each retry from the provider generates a new executionId. You need payload‑level idempotency keyed on the provider’s native event ID (see Section 5a). Implement the Redis‑backed idempotency gate before any side‑effect nodes (database writes, API calls, emails).

Q: Should I use SQLite or Postgres in queue mode?

Postgres is required. n8n explicitly does not support SQLite with queue mode. SQLite’s file‑locking model is incompatible with multiple worker processes writing concurrently, and you will see data corruption or missed executions — not just duplicates.

Q: What is the difference between the deduplication guard and an idempotency key?

The deduplication guard (Section 3.2) operates at the queue level — it prevents the same executionId from being processed twice by workers. The idempotency key (Section 5 and 5a) operates at the trigger / side‑effect level — it prevents a workflow from producing duplicate outputs when triggered twice by external retries. You need both in a production setup.

6. Conclusion

Duplicate execution in n8n occurs when the queue worker re‑processes the same executionId, typically because queueMode is parallel and the lock TTL (executionTimeout) expires under load. Fix it by switching to queueMode: "fifo", limiting maxConcurrentExecutions to 1‑2, and setting executionTimeout to 30 seconds. Add a deduplication guard at the workflow start that aborts if the executionId already exists in Redis, and mark the ID as processed at the end. Monitor lock TTLs and execution counts to catch regressions early.

For multi‑worker deployments the most impactful additional steps are: setting N8N_DISABLE_PRODUCTION_MAIN_PROCESS=true on the main instance, adding N8N_DISABLE_ACTIVE_WORKFLOWS=true on every worker, and keying idempotency on the provider’s event ID for webhook‑driven workflows. Combined with proper concurrency sizing and pinned image versions, these measures eliminate every common class of duplicate execution documented in the n8n community.

Must Read

Leave a Comment

Your email address will not be published. Required fields are marked *