Who this is for: Ops engineers, DevOps leads, or SREs managing server‑less workflows (Step Functions, Durable Functions, Cloud Run jobs, etc.) who see execution‑history storage driving up their monthly bill. We cover this in detail in the n8n Cost, Scaling & Infrastructure Economics Guide.
Quick diagnosis
Server‑less workflows automatically write execution‑history records to a log service (CloudWatch Logs, Azure Log Analytics, GCP Logging). If storage costs climb, the usual suspects are:
- Retention longer than required.
- Uncompressed logs staying in a hot tier.
- No archiving strategy for older data.
Pulling those three levers usually cuts storage spend by 40 %–80 %, and you still keep auditability.
In production, the creep shows up after the first billing cycle.
1. Where execution history lives?
If you encounter any n8n architecture regulated environments resolve them before continuing with the setup.
| Platform | Default store | Typical size per execution | Default retention |
|---|---|---|---|
| AWS Step Functions | CloudWatch Logs (/aws/vendedlogs/states/<state‑machine‑name>) |
0.5 KB – 5 KB | ∞ (no auto‑expire) |
| Azure Durable Functions | Log Analytics workspace (or Application Insights) | 1 KB – 10 KB | 30 days (configurable) |
| Google Cloud Workflows | Cloud Logging (log bucket) | 0.3 KB – 4 KB | 30 days (default) |
| AWS Lambda (invocation logs) | CloudWatch Logs (/aws/lambda/<function>) |
0.2 KB – 2 KB | ∞ |
| Platform | Cost per GB‑Month* |
|---|---|
| AWS CloudWatch Logs (standard) | $0.50 |
| Azure Log Analytics | $2.76 |
| GCP Cloud Logging (standard) | $0.01 |
*Prices as of 2024‑06; regional variations apply.
EEFA note: If you query logs with CloudWatch Insights for compliance, deleting logs before the audit window will break compliance and may incur penalties.
Action: Create a simple spreadsheet listing every log group, workspace, or bucket your workflows use; this inventory becomes the baseline for the steps that follow. Don’t assume the default retention is infinite—it’s just that the service doesn’t auto‑expire.
2. Trim retention policies to business need
2.1 AWS – set a short retention for CloudWatch log groups
# 7‑day retention for a Step Functions log group aws logs put-retention-policy \ --log-group-name "/aws/vendedlogs/states/MyStateMachine" \ --retention-in-days 7
| Retention (days) | Approx. monthly savings* |
|---|---|
| 7 | 55 % |
| 30 | 30 % |
| 90 | 15 % |
*Based on a 10 GB log volume at $0.50/GB‑mo. If you encounter any choosing instance types for n8n workloads resolve them before continuing with the setup.
2.2 Azure – shorten Log Analytics retention
# 14‑day retention for a Log Analytics workspace Set-AzOperationalInsightsWorkspace ` -ResourceGroupName "rg-prod" ` -Name "log-workspace" ` -RetentionInDays 14
2.3 GCP – lower bucket retention
# 10‑day retention for a Cloud Logging bucket gcloud logging buckets update my-bucket \ --retention-days=10
EEFA warning: Regulations such as PCI‑DSS or HIPAA often require ≥ 90 days of retention, so verify your compliance window before tightening policies—most teams forget to adjust retention until the bill arrives, making a double‑check worthwhile.
Most teams forget to adjust retention until the bill arrives, so it’s worth double‑checking now.
3. Export & archive historical execution logs
3.1 AWS – export to S3 and move to Glacier
# Export the last 90 days of logs to S3 aws logs create-export-task \ --log-group-name "/aws/vendedlogs/states/MyStateMachine" \ --from $(date -d '-90 days' +%s)000 \ --to $(date +%s)000 \ --destination "my-archive-bucket" \ --destination-prefix "step-functions/$(date +%Y-%m-%d)"
# Transition the exported objects to Glacier aws s3 cp s3://my-archive-bucket/step-functions/ \ s3://my-archive-bucket/step-functions-glacier/ \ --recursive --storage-class GLACIER
| Tier | Retrieval time | Cost/GB‑Month |
|---|---|---|
| S3 Standard | Immediate | $0.023 |
| S3 Intelligent‑Tiering | 1‑5 min | $0.0125 |
| S3 Glacier | 3‑5 hrs (expedited 1‑5 min) | $0.004 |
*At this point, moving to Glacier is usually cheaper than trying to prune logs forever.
3.2 Azure – archive to Blob Cool
# Run a Log Analytics query and save CSV locally az monitor log-analytics query \ --workspace "log-workspace" \ --analytics-query "AzureDiagnostics | where TimeGenerated < ago(90d)" \ --output csv > logs_$(date +%Y%m%d).csv
# Upload compressed CSV to Cool tier az storage blob upload \ --account-name mystorage \ --container-name archive \ --name "execution-history/$(date +%Y)/$(date +%m)/logs_$(date +%Y%m%d).csv.gz" \ --file <(gzip -c logs_$(date +%Y%m%d).csv) \ --tier Cool
3.3 GCP – sink to Cloud Storage Coldline
# Create a sink that writes old workflow logs to a bucket gcloud logging sinks create execution-history-archive \ storage.googleapis.com/my-archive-bucket \ --log-filter='resource.type="cloud_workflows" AND timestamp<"$(date -d "-90 days" --iso-8601)"' \ --include-children
# Lifecycle: move to Coldline after 30 days, delete after 365 days
cat > lifecycle.json <<'EOF'
{
"rule": [
{
"action": { "type": "SetStorageClass", "storageClass": "COLDLINE" },
"condition": { "age": 30 }
},
{
"action": { "type": "Delete" },
"condition": { "age": 365 }
}
]
}
EOF
gsutil lifecycle set lifecycle.json gs://my-archive-bucket
EEFA tip: Compress (gzip) the payload before upload. JSON logs shrink by 70 %‑80 %, directly reducing storage fees.
4. Automate ongoing cleanup
4.1 AWS – Lambda that enforces retention
import os, boto3
logs = boto3.client('logs')
RETENTION = int(os.getenv('RETENTION_DAYS', '7'))
def lambda_handler(event, context):
for grp in logs.describe_log_groups()['logGroups']:
name = grp['logGroupName']
if grp.get('retentionInDays') != RETENTION:
logs.put_retention_policy(
logGroupName=name,
retentionInDays=RETENTION
)
return {"status": "ok"}
Schedule: daily CloudWatch Event (rate(1 day))—a daily Lambda is usually enough; sub‑hourly runs are rarely needed.
4.2 Azure – PowerShell runbook
workflow Set-LogRetention {
param([int]$RetentionDays = 14)
$ws = Get-AzOperationalInsightsWorkspace -ResourceGroupName "rg-prod"
foreach -parallel ($w in $ws) {
Set-AzOperationalInsightsWorkspace `
-ResourceGroupName $w.ResourceGroupName `
-Name $w.Name `
-RetentionInDays $RetentionDays
}
}
Schedule: Azure Automation Scheduler (once per day).
4.3 GCP – Cloud Function triggered by Cloud Scheduler
# main.py
import os
from datetime import datetime, timedelta
from google.cloud import logging_v2
client = logging_v2.LoggingServiceV2Client()
RETENTION = int(os.getenv('RETENTION_DAYS', '10'))
def prune_logs(request):
cutoff = datetime.utcnow() - timedelta(days=RETENTION)
filter_str = (
f'timestamp < "{cutoff.isoformat()}Z" '
f'AND resource.type="cloud_workflows"'
)
client.delete_log_entries(filter_=filter_str)
return 'pruned'
Schedule: Cloud Scheduler (every 24 hours).
EEFA caution: Deletion APIs are irreversible. Test on a non‑production log group first and keep a 30‑day backup in the archive bucket. If you encounter any how logging levels impact cloud bills resolve them before continuing with the setup.
5. Continuous monitoring & alerting
| Metric | Threshold | Alert channel |
|---|---|---|
| CloudWatch Logs stored bytes (per state machine) | > 5 GB | SNS → Ops Slack |
| Log Analytics daily ingestion | > 1 GB | Azure Monitor Action Group |
| GCS bucket size (Coldline) | > 500 GB | Pub/Sub → PagerDuty |
Sample CloudWatch alarm – trigger when stored bytes exceed 5 GB:
aws cloudwatch put-metric-alarm \ --alarm-name "StepFunctionsLogCostSpike" \ --metric-name "IncomingBytes" \ --namespace "AWS/Logs" \ --statistic Sum \ --period 86400 \ --evaluation-periods 1 \ --threshold 5368709120 \ --comparison-operator GreaterThanThreshold \ --alarm-actions arn:aws:sns:us-east-1:123456789012:OpsAlerts
*We usually see the 5 GB threshold crossed after a few weeks of heavy traffic.
6. Best‑practice checklist
| Steps | Action |
|---|---|
| 1 | Inventory every execution‑history destination (CloudWatch, Log Analytics, Cloud Logging). |
| 2 | Align retention with compliance – keep logs only as long as required. |
| 3 | Export logs older than the retention window to a cheap archive tier (Glacier, Blob Cool/Archive, Coldline). |
| 4 | Compress exported files (gzip, zstd) before archiving. |
| 5 | Deploy an automated retention‑enforcer (Lambda, Automation Runbook, Cloud Function). |
| 6 | Add bucket lifecycle rules: hot → cold → delete. |
| 7 | Enable cost‑monitoring dashboards and threshold alerts per platform. |
| 8 | Perform a quarterly audit – verify archive integrity, confirm compliance windows, adjust thresholds. |
| 9 | Document the workflow in your runbook and link to the Serverless Cost Optimization Pillar. |
| 10 | Review provider price‑change announcements (e.g., AWS Log Data‑Ingestion) and update policies. |
Conclusion
By tightening retention, off‑loading stale execution history to ultra‑cheap archive tiers, and automating the pipeline, you can slash storage‑related spend on server‑less logs by up to 80 %. The approach keeps you audit‑ready, avoids surprise cost spikes, and only a few scheduled scripts run themselves. Apply the checklist, monitor the metrics, and you’ll keep execution‑history costs predictable and low in production.
