Who this is for: n8n developers building custom nodes that must handle thousands of items in production‑grade workflows. We cover this in detail in the n8n Performance & Scaling Guide.
Quick Diagnosis
Your custom n8n node is slowing down the entire workflow because it processes data synchronously, allocates large objects in memory, or makes blocking I/O calls. The fastest fix for a featured‑snippet‑ready answer is: refactor the node to use asynchronous streams, limit in‑memory payload size, and benchmark with the built‑in profiler.
1. Execution Model Overview
If you encounter any docker performance tuning resolve them before continuing with the setup.
| Phase | What n8n does | Typical Pitfall | EEFA Note |
|---|---|---|---|
| Input | this.getInputData() returns an array of items. |
Pulling the whole array into a temporary variable and looping synchronously. | Large batches (>10 k items) can exceed the V8 heap → “JavaScript heap out of memory”. |
| Processing | execute() runs for each item (or once for the whole batch). |
Heavy CPU work (e.g., JSON parsing, regex) inside the loop. | Offload CPU‑intensive work to a worker thread or external service. |
| Output | return this.prepareOutputData(items) pushes results downstream. |
Returning a massive array without pagination or streaming. | Use this.helpers.returnJsonArray() only for ≤ 5 k items; otherwise stream. |
Key takeaway: n8n expects nodes to be non‑blocking and memory‑conservative. Any deviation creates a bottleneck that propagates downstream.
2. Refactor Synchronous Loops into Asynchronous Streams
2.1 Bad Example – Blocking Loop
The node pulls all items, processes each synchronously, and returns a huge array.
// Get items and initialise results
const items = this.getInputData();
const results = [];
// Process each item synchronously
for (const item of items) {
const processed = heavySyncFunction(item.json);
results.push({ json: processed });
}
// Return the full result set
return this.prepareOutputData(results);
2.2 Good Example – Parallel Async Processing
Step 1 – Worker helper (4 lines). Runs heavy work in a separate thread and resolves a promise.
import { Worker } from 'worker_threads';
function runInWorker(data: any): Promise {
return new Promise((resolve, reject) => {
const worker = new Worker('./worker.js', { workerData: data });
worker.on('message', resolve);
worker.on('error', reject);
worker.on('exit', code => {
if (code !== 0) reject(new Error(`Worker stopped with code ${code}`));
});
});
}
Step 2 – Execute with limited concurrency (5 lines).
export async function execute(this: IExecuteFunctions) {
const items = this.getInputData();
const concurrency = Math.min(8, items.length);
const results: any[] = [];
for (let i = 0; i < items.length; i += concurrency) { const chunk = await Promise.all( items.slice(i, i + concurrency).map(item => runInWorker(item.json))
);
results.push(...chunk);
}
return this.prepareOutputData(results.map(r => ({ json: r })));
}
Why it works:
Each item runs in its own thread, freeing the main event loop.
Concurrency is capped (here at 8) to avoid CPU thrashing—critical on shared‑CPU containers.
EEFA warning: Worker threads increase memory usage per thread. Monitor the container’s memory (`docker stats` or `kubectl top pod`) and adjust concurrency accordingly. If you encounter any cpu profiling resolve them before continuing with the setup.
3. Reduce In‑Memory Payload Size
3.1 Trim Unnecessary Fields
Keep only the data you actually need before passing it downstream.
function prune(item: any) {
const { id, name, status } = item; // keep only what you need
return { id, name, status };
}
3.2 Use Chunked Processing
| Batch size | Avg. latency (ms) | Memory (MiB) |
|---|---|---|
| 500 items | 120 | 45 |
| 2 000 items | 460 | 180 |
| 5 000 items | 1 200 | 420 |
Recommendation: Keep batch size ≤ 2 000 for typical 2 GiB containers. For larger volumes, split the workflow with a “SplitInBatches” node before your custom node. If you encounter any resource limiting with cgroups resolve them before continuing with the setup.
4. Leverage n8n’s Built‑In Profiling Tools
4.1 Enable Debug Logging
Add this to ~/.n8n/config to capture detailed execution timestamps.
execution: logLevel: debug
4.2 Log Node Duration
Place a “Code” node after your custom node to output timing.
const start = $node["CustomNode"].getExecutionData().startTime;
const end = new Date();
console.log(`CustomNode duration: ${end - start} ms`);
4.3 Generate a Flame Graph
Run node --prof <your‑worker>.js and feed the output to node --prof-process. The resulting flame graph pinpoints hot functions inside your worker script.
5. Cache Reusable Results – When and How
| Situation | Cache type | Implementation |
|---|---|---|
| Repeated API calls with identical parameters | In‑memory LRU | npm i lru-cache and store results for 5 min. |
| Large static lookup tables (e.g., country codes) | Read‑only Map loaded at startup | Load once in module.exports = new Map([...]). |
| Expensive transformations that rarely change | External Redis | Use ioredis and set TTL = 1 h. |
5.1 LRU Cache Setup (4 lines)
import LRU from 'lru-cache';
const cache = new LRU<string, any>({ max: 500, ttl: 300_000 }); // 5 min TTL
5.2 Cached fetch helper (5 lines)
async function cachedFetch(key: string, fetcher: () => Promise) {
if (cache.has(key)) return cache.get(key);
const result = await fetcher();
cache.set(key, result);
return result;
}
5.3 Use inside execute() (4 lines)
const data = await cachedFetch(item.id, () => heavySyncFunction(item));
results.push({ json: data });
EEFA note: In Docker‑orchestrated environments each container has its own memory space; a per‑container LRU cache does not share state across replicas. For horizontal scaling, prefer an external Redis cache.
6. Test Performance Before Deployment
| Test | Tool | What it measures |
|---|---|---|
| Unit benchmark | benchmark.js (npm) |
Function‑level latency, GC pauses |
| End‑to‑end workflow run | n8n exec --run |
Full‑pipeline duration, memory peak |
| Load simulation | k6 script calling the REST API |
Concurrency impact, throttling behavior |
6.1 Benchmark script – Suite definition (4 lines)
const { Suite } = require('benchmark');
const { runInWorker } = require('./customNode');
6.2 Benchmark test & execution (5 lines)
new Suite()
.add('process 100 items', async () => {
const items = Array.from({ length: 100 }, (_, i) => ({ id: i }));
await Promise.all(items.map(i => runInWorker(i)));
})
.on('cycle', e => console.log(String(e.target)))
.run({ async: true });
Run node bench.js and aim for ≤ 5 ms per item on a 2 CPU container.
7. Production‑Ready Deployment Checklist
- Limit concurrency in the node (use env var
CUSTOM_NODE_MAX_CONCURRENCY). - Set V8 memory flag in Dockerfile if needed:
ENV NODE_OPTIONS="--max-old-space-size=1024"(adjust to container limit). - Expose health endpoint that runs a lightweight version of the node to verify it starts without OOM.
- Log duration and error rates to a centralized observability platform (e.g., Grafana Loki).
- Run the Docker‑performance‑tuning child page for container‑level tweaks: Docker performance tuning for n8n.
8. Common Errors & Quick Fixes
| Symptom | Root cause | One‑line fix |
|---|---|---|
| RangeError: Maximum call stack size exceeded | Recursive async calls without await |
Add await before each runInWorker or use Promise.allSettled. |
| Error: Worker stopped with code 1 | Worker script throws uncaught error | Wrap worker code in try/catch and parentPort.postMessage({ error }). |
| High memory usage after long runs | Cache never expires | Set appropriate TTL (cache.ttl = 300_000). |
| Workflow stalls at 100 % CPU | Concurrency > CPU cores | Set CUSTOM_NODE_MAX_CONCURRENCY = os.cpus().length. |
Conclusion
By converting blocking loops to capped‑concurrency worker threads, trimming payloads, caching repeatable results, and profiling with n8n’s built‑in tools, your custom node will stay responsive even under heavy load. Apply the checklist before shipping, monitor memory and CPU, and you’ll have a production‑ready node that scales reliably across containers and clusters.
