How Do You Optimize Execution History Storage Costs in n8n?

n8n Cost & Infrastructure Economics / Leave a Comment
Step by Step Guide to solve storage cost optimization execution history 
Step by Step Guide to solve storage cost optimization execution history

Who this is for: Ops engineers, DevOps leads, or SREs managing server‑less workflows (Step Functions, Durable Functions, Cloud Run jobs, etc.) who see execution‑history storage driving up their monthly bill. We cover this in detail in the n8n Cost, Scaling & Infrastructure Economics Guide.

Quick diagnosis

Server‑less workflows automatically write execution‑history records to a log service (CloudWatch Logs, Azure Log Analytics, GCP Logging). If storage costs climb, the usual suspects are:

  • Retention longer than required.
  • Uncompressed logs staying in a hot tier.
  • No archiving strategy for older data.

Pulling those three levers usually cuts storage spend by 40 %–80 %, and you still keep auditability.
In production, the creep shows up after the first billing cycle.

1. Where execution history lives?

If you encounter any n8n architecture regulated environments resolve them before continuing with the setup.

Platform Default store Typical size per execution Default retention
AWS Step Functions CloudWatch Logs (/aws/vendedlogs/states/<state‑machine‑name>) 0.5 KB – 5 KB ∞ (no auto‑expire)
Azure Durable Functions Log Analytics workspace (or Application Insights) 1 KB – 10 KB 30 days (configurable)
Google Cloud Workflows Cloud Logging (log bucket) 0.3 KB – 4 KB 30 days (default)
AWS Lambda (invocation logs) CloudWatch Logs (/aws/lambda/<function>) 0.2 KB – 2 KB ∞
Platform Cost per GB‑Month*
AWS CloudWatch Logs (standard) $0.50
Azure Log Analytics $2.76
GCP Cloud Logging (standard) $0.01

*Prices as of 2024‑06; regional variations apply.

EEFA note: If you query logs with CloudWatch Insights for compliance, deleting logs before the audit window will break compliance and may incur penalties.

Action: Create a simple spreadsheet listing every log group, workspace, or bucket your workflows use; this inventory becomes the baseline for the steps that follow. Don’t assume the default retention is infinite—it’s just that the service doesn’t auto‑expire.

2. Trim retention policies to business need

2.1 AWS – set a short retention for CloudWatch log groups

# 7‑day retention for a Step Functions log group
aws logs put-retention-policy \
  --log-group-name "/aws/vendedlogs/states/MyStateMachine" \
  --retention-in-days 7
Retention (days) Approx. monthly savings*
7 55 %
30 30 %
90 15 %

*Based on a 10 GB log volume at $0.50/GB‑mo. If you encounter any choosing instance types for n8n workloads resolve them before continuing with the setup.

2.2 Azure – shorten Log Analytics retention

# 14‑day retention for a Log Analytics workspace
Set-AzOperationalInsightsWorkspace `
  -ResourceGroupName "rg-prod" `
  -Name "log-workspace" `
  -RetentionInDays 14

2.3 GCP – lower bucket retention

# 10‑day retention for a Cloud Logging bucket
gcloud logging buckets update my-bucket \
  --retention-days=10

EEFA warning: Regulations such as PCI‑DSS or HIPAA often require ≥ 90 days of retention, so verify your compliance window before tightening policies—most teams forget to adjust retention until the bill arrives, making a double‑check worthwhile.

Most teams forget to adjust retention until the bill arrives, so it’s worth double‑checking now.

3. Export & archive historical execution logs

3.1 AWS – export to S3 and move to Glacier

# Export the last 90 days of logs to S3
aws logs create-export-task \
  --log-group-name "/aws/vendedlogs/states/MyStateMachine" \
  --from $(date -d '-90 days' +%s)000 \
  --to $(date +%s)000 \
  --destination "my-archive-bucket" \
  --destination-prefix "step-functions/$(date +%Y-%m-%d)"
# Transition the exported objects to Glacier
aws s3 cp s3://my-archive-bucket/step-functions/ \
  s3://my-archive-bucket/step-functions-glacier/ \
  --recursive --storage-class GLACIER
Tier Retrieval time Cost/GB‑Month
S3 Standard Immediate $0.023
S3 Intelligent‑Tiering 1‑5 min $0.0125
S3 Glacier 3‑5 hrs (expedited 1‑5 min) $0.004

*At this point, moving to Glacier is usually cheaper than trying to prune logs forever.

3.2 Azure – archive to Blob Cool

# Run a Log Analytics query and save CSV locally
az monitor log-analytics query \
  --workspace "log-workspace" \
  --analytics-query "AzureDiagnostics | where TimeGenerated < ago(90d)" \ --output csv > logs_$(date +%Y%m%d).csv
# Upload compressed CSV to Cool tier
az storage blob upload \
  --account-name mystorage \
  --container-name archive \
  --name "execution-history/$(date +%Y)/$(date +%m)/logs_$(date +%Y%m%d).csv.gz" \
  --file <(gzip -c logs_$(date +%Y%m%d).csv) \
  --tier Cool

3.3 GCP – sink to Cloud Storage Coldline

# Create a sink that writes old workflow logs to a bucket
gcloud logging sinks create execution-history-archive \
  storage.googleapis.com/my-archive-bucket \
  --log-filter='resource.type="cloud_workflows" AND timestamp<"$(date -d "-90 days" --iso-8601)"' \
  --include-children
# Lifecycle: move to Coldline after 30 days, delete after 365 days
cat > lifecycle.json <<'EOF'
{
  "rule": [
    {
      "action": { "type": "SetStorageClass", "storageClass": "COLDLINE" },
      "condition": { "age": 30 }
    },
    {
      "action": { "type": "Delete" },
      "condition": { "age": 365 }
    }
  ]
}
EOF
gsutil lifecycle set lifecycle.json gs://my-archive-bucket

EEFA tip: Compress (gzip) the payload before upload. JSON logs shrink by 70 %‑80 %, directly reducing storage fees.

4. Automate ongoing cleanup

4.1 AWS – Lambda that enforces retention

import os, boto3

logs = boto3.client('logs')
RETENTION = int(os.getenv('RETENTION_DAYS', '7'))

def lambda_handler(event, context):
    for grp in logs.describe_log_groups()['logGroups']:
        name = grp['logGroupName']
        if grp.get('retentionInDays') != RETENTION:
            logs.put_retention_policy(
                logGroupName=name,
                retentionInDays=RETENTION
            )
    return {"status": "ok"}

Schedule: daily CloudWatch Event (rate(1 day))—a daily Lambda is usually enough; sub‑hourly runs are rarely needed.

4.2 Azure – PowerShell runbook

workflow Set-LogRetention {
    param([int]$RetentionDays = 14)

    $ws = Get-AzOperationalInsightsWorkspace -ResourceGroupName "rg-prod"
    foreach -parallel ($w in $ws) {
        Set-AzOperationalInsightsWorkspace `
          -ResourceGroupName $w.ResourceGroupName `
          -Name $w.Name `
          -RetentionInDays $RetentionDays
    }
}

Schedule: Azure Automation Scheduler (once per day).

4.3 GCP – Cloud Function triggered by Cloud Scheduler

# main.py
import os
from datetime import datetime, timedelta
from google.cloud import logging_v2

client = logging_v2.LoggingServiceV2Client()
RETENTION = int(os.getenv('RETENTION_DAYS', '10'))

def prune_logs(request):
    cutoff = datetime.utcnow() - timedelta(days=RETENTION)
    filter_str = (
        f'timestamp < "{cutoff.isoformat()}Z" '
        f'AND resource.type="cloud_workflows"'
    )
    client.delete_log_entries(filter_=filter_str)
    return 'pruned'

Schedule: Cloud Scheduler (every 24 hours).

EEFA caution: Deletion APIs are irreversible. Test on a non‑production log group first and keep a 30‑day backup in the archive bucket. If you encounter any how logging levels impact cloud bills resolve them before continuing with the setup.

5. Continuous monitoring & alerting

Metric Threshold Alert channel
CloudWatch Logs stored bytes (per state machine) > 5 GB SNS → Ops Slack
Log Analytics daily ingestion > 1 GB Azure Monitor Action Group
GCS bucket size (Coldline) > 500 GB Pub/Sub → PagerDuty

Sample CloudWatch alarm – trigger when stored bytes exceed 5 GB:

aws cloudwatch put-metric-alarm \
  --alarm-name "StepFunctionsLogCostSpike" \
  --metric-name "IncomingBytes" \
  --namespace "AWS/Logs" \
  --statistic Sum \
  --period 86400 \
  --evaluation-periods 1 \
  --threshold 5368709120 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:OpsAlerts

*We usually see the 5 GB threshold crossed after a few weeks of heavy traffic.

6. Best‑practice checklist

Steps Action
1 Inventory every execution‑history destination (CloudWatch, Log Analytics, Cloud Logging).
2 Align retention with compliance – keep logs only as long as required.
3 Export logs older than the retention window to a cheap archive tier (Glacier, Blob Cool/Archive, Coldline).
4 Compress exported files (gzip, zstd) before archiving.
5 Deploy an automated retention‑enforcer (Lambda, Automation Runbook, Cloud Function).
6 Add bucket lifecycle rules: hot → cold → delete.
7 Enable cost‑monitoring dashboards and threshold alerts per platform.
8 Perform a quarterly audit – verify archive integrity, confirm compliance windows, adjust thresholds.
9 Document the workflow in your runbook and link to the Serverless Cost Optimization Pillar.
10 Review provider price‑change announcements (e.g., AWS Log Data‑Ingestion) and update policies.

Conclusion

By tightening retention, off‑loading stale execution history to ultra‑cheap archive tiers, and automating the pipeline, you can slash storage‑related spend on server‑less logs by up to 80 %. The approach keeps you audit‑ready, avoids surprise cost spikes, and only a few scheduled scripts run themselves. Apply the checklist, monitor the metrics, and you’ll keep execution‑history costs predictable and low in production.

Must Read

Leave a Comment

Your email address will not be published. Required fields are marked *