Stop Multi-Instance Sync Failures in n8n: 4 Fixes

n8n Performance & Scaling / Leave a Comment
Step by Step Guide to solve multi instance sync
Step by Step Guide to solve multi instance sync

Who this is for: DevOps engineers or platform engineers who need to run n8n in a horizontally‑scaled Docker/Kubernetes environment with deterministic execution. Keep workflow state consistent across scaled instances. We cover this in detail in the n8n Performance & Scaling Guide.

Sync Setup

Goal Minimal Config Required Services Key Env Vars
Consistent workflow execution across 3+ n8n nodes Use PostgreSQL for DB + Redis (or RabbitMQ) for queue PostgreSQL 13+, Redis 6+ (or RabbitMQ 3.8+) DB_TYPE=postgresdb
DB_POSTGRESDB_HOST=postgres
EXECUTIONS_PROCESS=queue
QUEUE_BULL_REDIS_HOST=redis
  1. Deploy a shared PostgreSQL instance.
  2. Deploy a Redis (or RabbitMQ) broker.
  3. Apply the env‑vars above on every n8n container.
  4. Restart all nodes – they now share workflow definitions and pull jobs from a single queue, guaranteeing state consistency.

Quick Diagnosis: What Problem Does This Page Solve?

If you encounter any database optimization resolve them before continuing with the setup.

When you horizontally scale n8n (Docker Swarm, Kubernetes, or multiple VMs), each node keeps its own execution cache. Without coordination the same workflow can fire on several nodes simultaneously, causing duplicate actions, race conditions, and data drift.

Solution – Centralize workflow definitions, execution state, and queue handling using a shared PostgreSQL database and a distributed job broker (Redis or RabbitMQ). The guide below shows the exact configuration, minimal code snippets, and troubleshooting steps to achieve deterministic, conflict‑free execution across any number of n8n instances.

1. Execution Model Overview

1.1 Workflow Storage

Mode Storage Backend Note
Single‑instance SQLite (local file) Not shareable across containers
Multi‑instance PostgreSQL / MySQL / MariaDB Network‑accessible DB required

1.2 Execution Queue

Mode Queue Implementation Benefit
Single‑instance In‑process (synchronous) Simple, but no deduplication
Multi‑instance Bull (Redis) or RabbitMQ Guarantees exactly‑once processing

1.3 Static Data & Credentials

Storage Location Recommendation
Local files (/root/.n8n) Container FS Do not use in scaled setups
DB‑backed tables (staticData) Shared DB Default when DB_TYPE is external

When EXECUTIONS_PROCESS=queue is set, n8n publishes every trigger event to the broker. Any node can pull the job, ensuring a single execution per trigger. If you encounter any database migration strategies resolve them before continuing with the setup.

2. Choosing a Shared Persistence Layer

2.1 Database Options

DB Engine Pros Cons
PostgreSQL Strong ACID, JSONB, SELECT FOR UPDATE locking Slightly heavier
MySQL / MariaDB Familiar to many DBAs Limited JSON querying (MySQL 5.7+)
SQLite (shared NFS) Zero‑config File‑locking issues under concurrency – not recommended

EEFA – PostgreSQL’s FOR UPDATE SKIP LOCKED prevents workers from stealing each other’s execution rows, eliminating duplicate runs.

2.2 Queue Broker Options

Broker Typical Latency Scaling Ease Persistence
Redis (Bull) < 5 ms Horizontal scaling trivial In‑memory, optional RDB/AOF
RabbitMQ ~ 10 ms Supports complex routing Durable queues
Kafka ~ 1 ms (high‑throughput) Very large clusters Log‑based persistence – overkill for most n8n workloads

EEFA – Reuse an existing Redis cache if available; otherwise RabbitMQ gives built‑in dead‑letter handling for failed jobs. If you encounter any cache layer implementation resolve them before continuing with the setup.

3. Configuring Queue Mode with Redis

3.1 Docker‑Compose: Service Definitions

PostgreSQL service (5 lines)

postgres:
  image: postgres:13
  environment:
    POSTGRES_USER: n8n
    POSTGRES_PASSWORD: secret

Redis service (5 lines)

redis:
  image: redis:6-alpine
  command: ["redis-server", "--appendonly", "yes"]

n8n service (5 lines – core env vars)

n8n:
  image: n8nio/n8n:latest
  environment:
    - DB_TYPE=postgresdb
    - DB_POSTGRESDB_HOST=postgres
    - EXECUTIONS_PROCESS=queue
    - QUEUE_BULL_REDIS_HOST=redis

n8n service – optional auth & ports (5 lines)

    - N8N_BASIC_AUTH_ACTIVE=true
    - N8N_BASIC_AUTH_USER=admin
    - N8N_BASIC_AUTH_PASSWORD=supersecret
  ports:
    - "5678:5678"

EEFA – appendonly yes in Redis ensures queued jobs survive node restarts.

3.2 Kubernetes Manifest – Deployment & Service

Deployment (first 5 lines)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n
spec:
  replicas: 3
  selector:
    matchLabels:
      app: n8n

Container spec with env (next 5 lines)

  template:
    metadata:
      labels:
        app: n8n
    spec:
      containers:
        - name: n8n
          image: n8nio/n8n:latest
          env:
            - name: DB_TYPE
              value: "postgresdb"
            - name: DB_POSTGRESDB_HOST
              value: "postgres-svc"
            - name: EXECUTIONS_PROCESS
              value: "queue"
            - name: QUEUE_BULL_REDIS_HOST
              value: "redis-svc"

Service (5 lines)

---
apiVersion: v1
kind: Service
metadata:
  name: n8n-svc
spec:
  selector:
    app: n8n
  ports:
    - port: 80
      targetPort: 5678
  type: LoadBalancer

EEFA – Use a headless Service for Redis if you need stable DNS per pod. Add podAntiAffinity to avoid co‑locating all n8n pods on a single node.

4. Synchronizing Static Data, Credentials & Secrets

4.1 Migration Checklist

Steps Action
1 Ensure DB_TYPE is not sqlite.
2 Export local credentials: n8n export:credentials --output ./creds.json.
3 Import them: n8n import:credentials --input ./creds.json.
4 Delete the local .n8n folder on each node (or mount it read‑only).
5 Restart all n8n instances.

EEFA – Credentials are encrypted with N8N_ENCRYPTION_KEY. The same key must be set on every pod; otherwise imported credentials become unreadable.

4.2 Setting a Shared Encryption Key (4 lines)

export N8N_ENCRYPTION_KEY=$(openssl rand -base64 32)
docker run -e N8N_ENCRYPTION_KEY=$N8N_ENCRYPTION_KEY \
           -e DB_TYPE=postgresdb \
           -e DB_POSTGRESDB_HOST=postgres \
           -e EXECUTIONS_PROCESS=queue \
           -e QUEUE_BULL_REDIS_HOST=redis \
           n8nio/n8n

Copy the printed key into the environment of all n8n instances.

5. Health Checks, Conflict Resolution & Idempotency

5.1 Built‑in Health Endpoint (4 lines)

livenessProbe:
  httpGet:
    path: /healthz
    port: 5678
  initialDelaySeconds: 30
  periodSeconds: 10

Configure your load balancer to route traffic only to pods that report healthy.

5.2 Idempotent Trigger Patterns

Technique Implementation
Idempotent nodes Store a unique request ID in workflow static data (Set node) and skip processing if it already exists.
Webhook signature verification Validate HMAC header before any business logic.
Debounce logic Add a Delay node with `Wait Until` to coalesce rapid repeats.

EEFA – For cron triggers, set cronTimezone to a fixed zone (e.g., UTC) on all nodes to avoid overlapping runs.

5.3 Monitoring Queue Backlog (2 lines)

redis-cli -h redis llen bull:default   # pending jobs

If the length exceeds 500, consider scaling n8n replicas or increasing Redis maxmemory.

6. Troubleshooting Common Sync Issues

Symptom Likely Cause Fix
Duplicate API calls Two nodes processing the same queue item Verify a single broker address (QUEUE_BULL_REDIS_HOST) and that EXECUTIONS_PROCESS=queue is set on all nodes.
Workflow not found after scaling Workers pointing to different DB instances Use one connection string (DB_POSTGRESDB_HOST) and confirm DNS resolves to the same service.
Credentials decryption error Mismatched N8N_ENCRYPTION_KEY across pods Export the key from a working node and propagate it uniformly.
Stale static data after pod restart Persistent volume not shared Switch to DB‑backed static data (default with external DB) or mount a ReadWriteMany PVC (e.g., NFS) across pods.
Redis connection timeout Network policy blocks port 6379 Add an allow rule for the Redis service in your cluster’s network policy.

6.1 Diagnostic Script: Part 1 (4 lines)

#!/usr/bin/env bash
echo "=== DB connectivity ==="
psql -h $DB_POSTGRESDB_HOST -U $DB_POSTGRESDB_USER -d $DB_POSTGRESDB_DATABASE -c "\dt"

6.2 Diagnostic Script: Part 2 (4 lines)

echo "=== Redis queue length ==="
redis-cli -h $QUEUE_BULL_REDIS_HOST llen bull:default

6.3 Diagnostic Script: Part 3 (4 lines)

echo "=== Encryption key consistency ==="
grep N8N_ENCRYPTION_KEY /proc/$(pgrep -f n8n)/environ | cut -d= -f2 | sort | uniq -c

Run the three parts on each node; outputs should be identical.

Conclusion

By centralizing workflow definitions in PostgreSQL and delegating trigger handling to a single Redis (or RabbitMQ) queue, every n8n instance works from the same source of truth and processes each event exactly once. Consistent N8N_ENCRYPTION_KEY, shared static‑data storage, and health‑checked pods complete the production‑ready recipe. Apply the snippets above, verify the diagnostics, and your scaled n8n fleet will run deterministically, free from duplicate actions and race conditions.

Must Read

Leave a Comment

Your email address will not be published. Required fields are marked *