Improve Custom Node Performance in n8n: 5 Steps

n8n Performance & Scaling / Leave a Comment
Step by Step Guide to solve custom node performance
Step by Step Guide to solve custom node performance

Who this is for: n8n developers building custom nodes that must handle thousands of items in production‑grade workflows. We cover this in detail in the n8n Performance & Scaling Guide.

Quick Diagnosis

Your custom n8n node is slowing down the entire workflow because it processes data synchronously, allocates large objects in memory, or makes blocking I/O calls. The fastest fix for a featured‑snippet‑ready answer is: refactor the node to use asynchronous streams, limit in‑memory payload size, and benchmark with the built‑in profiler.

1. Execution Model Overview

If you encounter any docker performance tuning resolve them before continuing with the setup.

Phase What n8n does Typical Pitfall EEFA Note
Input this.getInputData() returns an array of items. Pulling the whole array into a temporary variable and looping synchronously. Large batches (>10 k items) can exceed the V8 heap → “JavaScript heap out of memory”.
Processing execute() runs for each item (or once for the whole batch). Heavy CPU work (e.g., JSON parsing, regex) inside the loop. Offload CPU‑intensive work to a worker thread or external service.
Output return this.prepareOutputData(items) pushes results downstream. Returning a massive array without pagination or streaming. Use this.helpers.returnJsonArray() only for ≤ 5 k items; otherwise stream.

Key takeaway: n8n expects nodes to be non‑blocking and memory‑conservative. Any deviation creates a bottleneck that propagates downstream.

2. Refactor Synchronous Loops into Asynchronous Streams

2.1 Bad Example – Blocking Loop

The node pulls all items, processes each synchronously, and returns a huge array.

// Get items and initialise results
const items = this.getInputData();
const results = [];

// Process each item synchronously
for (const item of items) {
    const processed = heavySyncFunction(item.json);
    results.push({ json: processed });
}

// Return the full result set
return this.prepareOutputData(results);

2.2 Good Example – Parallel Async Processing

Step 1 – Worker helper (4 lines). Runs heavy work in a separate thread and resolves a promise.

import { Worker } from 'worker_threads';

function runInWorker(data: any): Promise {
    return new Promise((resolve, reject) => {
        const worker = new Worker('./worker.js', { workerData: data });
        worker.on('message', resolve);
        worker.on('error', reject);
        worker.on('exit', code => {
            if (code !== 0) reject(new Error(`Worker stopped with code ${code}`));
        });
    });
}

Step 2 – Execute with limited concurrency (5 lines).

export async function execute(this: IExecuteFunctions) {
    const items = this.getInputData();
    const concurrency = Math.min(8, items.length);
    const results: any[] = [];

    for (let i = 0; i < items.length; i += concurrency) { const chunk = await Promise.all( items.slice(i, i + concurrency).map(item => runInWorker(item.json))
        );
        results.push(...chunk);
    }

    return this.prepareOutputData(results.map(r => ({ json: r })));
}

Why it works:
Each item runs in its own thread, freeing the main event loop.
Concurrency is capped (here at 8) to avoid CPU thrashing—critical on shared‑CPU containers.

EEFA warning: Worker threads increase memory usage per thread. Monitor the container’s memory (`docker stats` or `kubectl top pod`) and adjust concurrency accordingly. If you encounter any cpu profiling resolve them before continuing with the setup.

3. Reduce In‑Memory Payload Size

3.1 Trim Unnecessary Fields

Keep only the data you actually need before passing it downstream.

function prune(item: any) {
    const { id, name, status } = item; // keep only what you need
    return { id, name, status };
}

3.2 Use Chunked Processing

Batch size Avg. latency (ms) Memory (MiB)
500 items 120 45
2 000 items 460 180
5 000 items 1 200 420

Recommendation: Keep batch size ≤ 2 000 for typical 2 GiB containers. For larger volumes, split the workflow with a “SplitInBatches” node before your custom node. If you encounter any resource limiting with cgroups resolve them before continuing with the setup.

4. Leverage n8n’s Built‑In Profiling Tools

4.1 Enable Debug Logging

Add this to ~/.n8n/config to capture detailed execution timestamps.

execution:
  logLevel: debug

4.2 Log Node Duration

Place a “Code” node after your custom node to output timing.

const start = $node["CustomNode"].getExecutionData().startTime;
const end   = new Date();
console.log(`CustomNode duration: ${end - start} ms`);

4.3 Generate a Flame Graph

Run node --prof <your‑worker>.js and feed the output to node --prof-process. The resulting flame graph pinpoints hot functions inside your worker script.

5. Cache Reusable Results – When and How

Situation Cache type Implementation
Repeated API calls with identical parameters In‑memory LRU npm i lru-cache and store results for 5 min.
Large static lookup tables (e.g., country codes) Read‑only Map loaded at startup Load once in module.exports = new Map([...]).
Expensive transformations that rarely change External Redis Use ioredis and set TTL = 1 h.

5.1 LRU Cache Setup (4 lines)

import LRU from 'lru-cache';
const cache = new LRU<string, any>({ max: 500, ttl: 300_000 }); // 5 min TTL

5.2 Cached fetch helper (5 lines)

async function cachedFetch(key: string, fetcher: () => Promise) {
    if (cache.has(key)) return cache.get(key);
    const result = await fetcher();
    cache.set(key, result);
    return result;
}

5.3 Use inside execute() (4 lines)

const data = await cachedFetch(item.id, () => heavySyncFunction(item));
results.push({ json: data });

EEFA note: In Docker‑orchestrated environments each container has its own memory space; a per‑container LRU cache does not share state across replicas. For horizontal scaling, prefer an external Redis cache.

6. Test Performance Before Deployment

Test Tool What it measures
Unit benchmark benchmark.js (npm) Function‑level latency, GC pauses
End‑to‑end workflow run n8n exec --run Full‑pipeline duration, memory peak
Load simulation k6 script calling the REST API Concurrency impact, throttling behavior

6.1 Benchmark script – Suite definition (4 lines)

const { Suite } = require('benchmark');
const { runInWorker } = require('./customNode');

6.2 Benchmark test & execution (5 lines)

new Suite()
  .add('process 100 items', async () => {
    const items = Array.from({ length: 100 }, (_, i) => ({ id: i }));
    await Promise.all(items.map(i => runInWorker(i)));
  })
  .on('cycle', e => console.log(String(e.target)))
  .run({ async: true });

Run node bench.js and aim for ≤ 5 ms per item on a 2 CPU container.

7. Production‑Ready Deployment Checklist

  • Limit concurrency in the node (use env var CUSTOM_NODE_MAX_CONCURRENCY).
  • Set V8 memory flag in Dockerfile if needed: ENV NODE_OPTIONS="--max-old-space-size=1024" (adjust to container limit).
  • Expose health endpoint that runs a lightweight version of the node to verify it starts without OOM.
  • Log duration and error rates to a centralized observability platform (e.g., Grafana Loki).
  • Run the Docker‑performance‑tuning child page for container‑level tweaks: Docker performance tuning for n8n.

8. Common Errors & Quick Fixes

Symptom Root cause One‑line fix
RangeError: Maximum call stack size exceeded Recursive async calls without await Add await before each runInWorker or use Promise.allSettled.
Error: Worker stopped with code 1 Worker script throws uncaught error Wrap worker code in try/catch and parentPort.postMessage({ error }).
High memory usage after long runs Cache never expires Set appropriate TTL (cache.ttl = 300_000).
Workflow stalls at 100 % CPU Concurrency > CPU cores Set CUSTOM_NODE_MAX_CONCURRENCY = os.cpus().length.

Conclusion

By converting blocking loops to capped‑concurrency worker threads, trimming payloads, caching repeatable results, and profiling with n8n’s built‑in tools, your custom node will stay responsive even under heavy load. Apply the checklist before shipping, monitor memory and CPU, and you’ll have a production‑ready node that scales reliably across containers and clusters.

Must Read

Leave a Comment

Your email address will not be published. Required fields are marked *