Repository navigation
feat(cost): tune budgets, Config recording and log retention for cost control - #159
Alexanderamiri wants to merge 2 commits into
Conversation
… control - Account budget $500 -> $600 (steady state ~$550 incl. VAT); notify on forecast >100%, actual >100%, actual >150% instead of a monthly 80% alert. - Remove the inert EstimatedCharges billing alarm and its unsubscribed us-east-1 SNS topic (metric not enabled, INSUFFICIENT_DATA since 2026-03). - AWS Config: stop recording EC2 network interfaces (Fargate churn), record VPC/Subnet daily (~$27/mo of configuration items). - 730-day expiry on the Config delivery bucket. - Terraform-managed log groups with 365-day retention for all platform Lambdas and auth-claims; existing groups adopted with import blocks. - Test that every platform Lambda has a declared log group. - docs/cost-management.md. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Terraform Plan🚧 Changes detected — Plan: 15 to import, 2 to add, 17 to change, 2 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan imports 15 existing CloudWatch log groups, creates 2 resources, updates 17 (budget limit/notifications, Config recorder scope, log retention), and destroys a billing alarm and its SNS topic, with no database or security-group changes.
|
Excluding AWS::EC2::NetworkInterface would leave Security Hub EC2.180 (ENI source/destination check) without input and add a reason to Config.1. Record ENI, VPC and Subnet with a DAILY override instead, and document the trade-off, the realistic saving and the 7-day check. Also widen the Lambda log-group guard test to every platform module, and add the budget line to CLAUDE.md without rewriting the Cost Anomaly line so it no longer conflicts with the ECS impaired-alert PR. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Terraform Plan🚧 Changes detected — Plan: 15 to import, 2 to add, 17 to change, 2 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan imports 15 existing CloudWatch log groups and sets 365-day retention on them, creates 2 resources, updates 17 (budget limit/notifications, Config recorder frequency, log retention), and destroys the billing alarm and its SNS topic in us-east-1; no database, security group, or IAM changes, and nothing is internet-exposed.
|
Why
A cost audit (2026-10, live Cost Explorer/CloudWatch/Config data) found that the cost guardrails we already run have gaps:
javabin-billing-alarm(us-east-1) does nothing. It has beenINSUFFICIENT_DATAsince 2026-03-26 because the AWS/Billing EstimatedCharges metric is not enabled for the account. Its SNS topic has 0 subscribers, and its $200 threshold is below monthly spend anyway.testteam-platform-test-appcrash-looping (~80 starts/day), so fixing that service is the main saving; this PR only trims what remains./aws/lambda/javabin-*log groups and auth-claims never expire. Lambda auto-created them, and Terraform doesn't manage them.What
cost-analytics:javabin-alerts, which the slack-alert Lambda posts to #javabin-cost-alerts.aws_cloudwatch_metric_alarm.billing,aws_sns_topic.billing_alarmand thebilling_alarm_threshold_usdvariable. The us-east-1 provider alias stays because the CUR export uses it.monitoring:AWS::EC2::NetworkInterface,AWS::EC2::VPCandAWS::EC2::Subnetuse a DAILY override. Security groups stay CONTINUOUS. ENIs are deliberately not excluded: Security Hub EC2.180 (ENI source/destination check, enabled) evaluatesAWS::EC2::NetworkInterface, and Config.1 fails when an enabled control's resource type is not recorded. Trade-off: EC2.180 evaluates daily instead of on every change, and an ENI created and deleted within one day may never be recorded. Expected saving ~$15–20/mo while the crash loop lasts, a few dollars a month after it is fixed.javabin-config-<acct>bucket now expires objects after 730 days. The bucket is not versioned.lambdas/log-groups.tfandidentity/log-groups.tf:aws_cloudwatch_log_groupwith 365-day retention for every platform Lambda.imports.tf:for_eachimport blocks (TF 1.7) adopt the 14 existing groups plus auth-claims. budget-enforcer has never logged, so its group is created fresh.scripts/tests/test_lambda_log_groups.py: scans everyterraform/platform/**/*.tfand fails if a${var.project}-*Lambda has no log group (sharedlocal.platform_lambda_namesor an explicit/aws/lambda/...group in its module).docs/cost-management.mdcovering alerts, tags, CUR, built-in controls and why Config records ENIs daily, and the post-apply checks.docs/platform-modules.mdanddocs/lambda-functions.md.Not changed, on purpose:
Expected plan
module.cost_analytics.aws_budgets_budget.account: updated in place.module.cost_analytics.aws_cloudwatch_metric_alarm.billingandaws_sns_topic.billing_alarm: destroyed. Both are inert, as explained above.module.monitoring.aws_config_configuration_recorder.main: updated in place.module.monitoring.aws_s3_bucket_lifecycle_configuration.config_logs: created.How verified
terraform fmt -recursive -check, thenterraform init -backend=falseandterraform validateinterraform/platform. All pass.python3 -m unittest discover -s scripts/tests(46 OK) and-s terraform/lambda-src/tests(OK). Negative check: renaming the auth-claims log group makes the guard test fail.git merge-treeagainstfeat/ecs-impaired-alert(feat(monitoring): alert on ECS services that cannot start tasks #160): no conflict.AWS::EC2::NetworkInterface.javabin-config-*bucket is unversioned with no lifecycle, the recorder isALL_SUPPORTED_RESOURCE_TYPES+ CONTINUOUS, and the list of existing/aws/lambda/javabin-*log groups matches the import list.Follow-ups
AWS/Config ConfigurationItemsRecordedper ResourceType. If SecurityGroup still dominates, add it to the DAILY override.AWS::EC2::NetworkInterface.testteam-platform-test-appto 0, or re-push its image. It is the root cause of the Config churn.account_budget_usdas the legacy cleanup lands. The cleanup is in the cost audit report: idle EB environments, Default-ALB, Coolify, NAT.🤖 Generated with Claude Code