n8n in-memory key-value store – store and retrieve data across nodes without a database

n8n Performance & Scaling / Leave a Comment
Step by Step Guide to solve workflow design best practices
Step by Step Guide to solve workflow design best practices

Who this is for: Developers and DevOps engineers who build production‑grade n8n automations and need to keep execution time and resource usage low. We cover this in detail in the n8n Performance & Scaling Guide.

Quick Diagnosis

Problem: A workflow runs slower than expected or spikes CPU/memory, creating bottlenecks in automated pipelines.

Featured‑snippet solution: Use lightweight nodes, batch data, limit synchronous loops, and off‑load heavy processing to external services. The checklist below shaves milliseconds off each execution and keeps the resource footprint minimal.

1. n8n’s Execution Model

Component What it does Performance impact
Node Runner Executes nodes sequentially in a single Node.js process CPU‑bound; each node blocks the event loop
Workflow Queue Stores pending executions (Redis, DB) I/O latency if the backend is slow
Credential Store Decrypts credentials per execution Minor CPU cost
Execution Context Holds data passed between nodes Memory grows with large payloads

EEFA note: n8n runs in a single‑threaded Node.js runtime. Any synchronous, CPU‑heavy operation (e.g., large for loops in a Function node) stalls the worker and impacts all concurrent workflows. If you encounter any environment variable tuning resolve them before continuing with the setup.

2. Core Design Patterns for Speed

2.1 Prefer Built‑in Nodes Over Custom JavaScript

Use‑case Recommended node
HTTP GET/POST HTTP Request
Data transformation Set, Merge, SplitInBatches
Conditional routing IF

Replace a Function node with a Set node

Purpose: Extract only the id and status fields from each item without JavaScript loops. If you encounter any cost optimization resolve them before continuing with the setup.

{
  "parameters": {
    "keepOnlySet": true,
    "values": [
      { "name": "id", "value": "={{$json[\"id\"]}}" },
      { "name": "status", "value": "={{$json[\"status\"]}}" }
    ]
  },

  "name": "Set ID & Status",
  "type": "n8n-nodes-base.set",
  "typeVersion": 1,
  "position": [400, 300]
}

EEFA warning: A Function node that iterates items.forEach to extract fields is ~3‑5× slower than the Set node’s internal C++ path.

2.2 Batch Processing with SplitInBatches

Purpose: Process large arrays (10 k+ records) in manageable chunks to limit memory spikes.

{
  "parameters": {
    "batchSize": 500,
    "continueOnFail": false
  },

  "name": "Split In Batches",
  "type": "n8n-nodes-base.splitInBatches",
  "typeVersion": 1,
  "position": [600, 300]
}

Result: Only 500 items reside in memory at a time, and downstream nodes can run in parallel when you enable *Execute Workflow* → *Run Once*.

2.3 Asynchronous Off‑loading

Purpose: Delegate CPU‑intensive work (PDF generation, image processing) to an external micro‑service.

{
  "parameters": {
    "url": "https://api.example.com/render-pdf",
    "method": "POST",
    "jsonParameters": true,

    "bodyParametersJson": "={{$json}}",
    "options": {
      "responseFormat": "json",
      "timeout": 120000,
      "allowUnauthorizedCerts": false
    }
  },

  "name": "Generate PDF (Async)",
  "type": "n8n-nodes-base.httpRequest",
  "typeVersion": 2,
  "position": [800, 300]
}

EEFA tip: Set a generous timeout and a retry strategy to avoid hanging the workflow if the service slows down.

2.4 Minimize Data Copies

  • Use Reference syntax ({{$json["field"]}}) instead of cloning objects.
  • Drop unused fields early with a Set node (keepOnlySet: true).

2.5 Parallelism via Execute Workflow

Purpose: Run a heavy branch in a sub‑workflow concurrently with the main flow.

{
  "parameters": {
    "workflowId": "123",
    "runOnce": true,
    "waitForCompletion": false
  },

  "name": "Parallel Sub‑workflow",
  "type": "n8n-nodes-base.executeWorkflow",
  "typeVersion": 1,
  "position": [1000, 300]
}

3. Data Handling & Batching Strategies

Strategy When to use
Selective field extraction Large payloads from APIs
Chunked writes Bulk DB inserts (PostgreSQL, MySQL)
Streaming CSV/JSON files > 5 MB
Cache results Re‑used lookup tables

Implementation tips

  • After an external call, immediately use Set with keepOnlySet:true.
  • Pair SplitInBatches with the target node’s *Batch Mode* (e.g., Postgres).
  • For streaming files, use Read Binary File + Write Binary File in Node.js stream mode.
  • Cache look‑ups in Redis with a TTL of 5 min.

Optimized Data Flow Checklist

  • Remove unnecessary keys after each external call.
  • Batch API calls (ids[]=1&ids[]=2 instead of one request per ID).
  • Enable continueOnFail only where failures are expected.
  • Set maxExecutionTime on the workflow to guard against runaway loops.
  • If you encounter any upgrading n8n versions resolve them before continuing with the setup.

EEFA caution: Disabling continueOnFail on a node that may receive malformed data can abort the workflow, causing missed alerts. Use granular error handling (IF → Error Trigger) instead.

4. Conditional Branch Optimization

4.1 Collapse Nested IFs

Purpose: Reduce the number of node evaluations by combining conditions.

{
  "parameters": {
    "conditions": {
      "string": [
        {
          "value1": "={{$json[\"status\"]}}",
          "operation": "equal",
          "value2": "approved"
        },

        {
          "value1": "={{$json[\"amount\"]}}",
          "operation": "greaterThan",
          "value2": 1000
        }
      ],
      "logic": "AND"
    }
  },

  "name": "Approved & High‑Value?",
  "type": "n8n-nodes-base.if",
  "typeVersion": 1,
  "position": [1200, 300]
}

Result: One evaluation replaces two sequential checks, trimming CPU cycles.

4.2 Early Exit with Stop Node

Purpose: Terminate a branch instantly when further processing isn’t required.

{
  "name": "Stop on Inactive",
  "type": "n8n-nodes-base.stop",
  "typeVersion": 1,
  "position": [1400, 300]
}

5. Error Handling Without Overhead

Technique Overhead Best practice
Try‑Catch in Function Minimal (JS try) Use only for anticipated JSON parse errors
Error Trigger node Adds a separate execution path Place at workflow end to capture unhandled errors
Retry on HTTP 429 Slight latency due to back‑off Configure exponential back‑off in HTTP Request node

Centralized Error Logging

Purpose: Forward errors to a lightweight logging workflow.

{
  "parameters": {
    "workflowId": "999",
    "runOnce": true,
    "waitForCompletion": false,
    "inputData": "={{$json}}"
  },

  "name": "Log Error",
  "type": "n8n-nodes-base.executeWorkflow",
  "typeVersion": 1,
  "position": [1600, 300]
}

EEFA note: Keep the error‑logging workflow minimal (no heavy DB writes) to avoid cascading failures.

6. Monitoring & Profiling Your Optimized Workflow

  1. Enable Execution Logging – Settings → Execution → set logLevel to debug for the target workflow.
  2. Performance tab – Built‑in view shows per‑node execution time.
  3. Prometheus integration – Export n8n_execution_time_seconds.
avg by (nodeName) (
  rate(n8n_execution_time_seconds_sum[5m]) /
  rate(n8n_execution_time_seconds_count[5m])
)
  1. CPU profiling – See the sibling guide n8n CPU profiling guide for V8 flamegraph generation.

EEFA tip: Set alert thresholds based on historical baselines; a sudden 2× increase often signals a regression in data volume or a newly added heavy node.

Conclusion

By favoring built‑in nodes, batching data, off‑loading heavy work, and pruning unnecessary payloads, you can keep n8n workflows snappy and resource‑efficient. Parallelism via Execute Workflow, collapsed condition checks, and early exits further reduce CPU load. Pair these patterns with diligent monitoring (performance tab, Prometheus, logs) and a lightweight error‑logging strategy to maintain reliability in production. Apply the checklist, adjust batch sizes to your hardware (tested on Docker with 2 CPU cores, 4 GB RAM), and your automations will stay fast, scalable, and cost‑effective.

Must Read

Leave a Comment

Your email address will not be published. Required fields are marked *