[Alexanderamiri]

Why

testteam-platform-test-app has been crash-looping for weeks with desired=1 and running=0, because of a CannotPullContainerError (its ECR repo is empty). That is about 80 failed task starts a day, and nobody noticed. Each attempt creates and deletes a Fargate ENI, which AWS Config records. This was the main driver of about $29/mo in Config charges found in the 2026-10 cost audit. PR #159 reduces the Config side. This PR makes the crash loop itself visible.

What

• monitoring/main.tf: new EventBridge rule javabin-ecs-task-start-impaired. It matches source=aws.ecs, detail-type=ECS Service Action and eventName=SERVICE_TASK_START_IMPAIRED, for the javabin-platform cluster only (legacy CDK clusters are excluded). The target is the javabin-security SNS topic, whose policy already allows events.amazonaws.com. slack-alert sends non-finding events from that topic to #javabin-infra-alerts. I chose that channel over the cost channel because this is an operational problem.
• slack_alert/handler.py:
  • New format_ecs_service_impaired formatter, routed before the generic EventBridge branch. Without it, the event would fall through to the IAM-change formatter.
  • The message shows the service, the cluster and a link to the service's events page in the ECS console.
  • Dedup uses the existing persistent javabin-alert-dedup table, keyed per service ARN per UTC day. ECS keeps emitting the event while the service is impaired.
• New test terraform/lambda-src/tests/test_slack_alert_ecs.py checks the formatting, once-per-day dedup per service, and that other ECS service actions are ignored.
• Docs: added a line to the CLAUDE.md alert routing section and to docs/lambda-functions.md.

Expected plan

2 to add: aws_cloudwatch_event_rule.ecs_task_start_impaired and aws_cloudwatch_event_target.ecs_task_start_impaired_sns. 1 in-place update: the slack-alert Lambda code. No IAM changes, because slack-alert already has GetItem/PutItem on the dedup table.

How verified

• python3 -m unittest discover -s terraform/lambda-src/tests (67 OK, 3 new) and -s scripts/tests pass.
• terraform fmt -recursive -check, then terraform init -backend=false and terraform validate pass.

Follow-ups

• Once this is applied, the first alert should be for testteam-platform-test-app. Scale it to 0 or push its image.
• It is independent of feat(cost): tune budgets, Config recording and log retention for cost control #159. Both touch monitoring/main.tf, but in different blocks.

🤖 Generated with Claude Code

[github-actions]

Terraform Plan

🚧 Changes detected — Plan: 2 to add, 2 to change, 0 to destroy.

Plan output

aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # module.lambdas.aws_lambda_function.securityhub_summary will be updated in-place
  ~ resource "aws_lambda_function" "securityhub_summary" {
        id                             = "javabin-securityhub-summary"
      ~ last_modified                  = "2026-10-23T09:00:34.000+0000" -> (known after apply)
      ~ source_code_hash               = "hrcvTHhG+KIWoQVwbPKmZHx4p1RGl3VZPkTdSdhyfKk=" -> "ci2qbjUrBx0+yEslzlkKpQdZ0eQltqrBbHoakTuR6Ik="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.lambdas.aws_lambda_function.slack_alert will be updated in-place
  ~ resource "aws_lambda_function" "slack_alert" {
        id                             = "javabin-slack-alert"
      ~ last_modified                  = "2026-10-23T09:00:40.000+0000" -> (known after apply)
      ~ source_code_hash               = "hrcvTHhG+KIWoQVwbPKmZHx4p1RGl3VZPkTdSdhyfKk=" -> "ci2qbjUrBx0+yEslzlkKpQdZ0eQltqrBbHoakTuR6Ik="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.monitoring.aws_cloudwatch_event_rule.ecs_task_start_impaired will be created
  + resource "aws_cloudwatch_event_rule" "ecs_task_start_impaired" {
      + arn            = (known after apply)
      + description    = "ECS services in the platform cluster that cannot start tasks"
      + event_bus_name = "default"
      + event_pattern  = jsonencode(
            {
              + detail      = {
                  + clusterArn = [
                      + "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform",
                    ]
                  + eventName  = [
                      + "SERVICE_TASK_START_IMPAIRED",
                    ]
                }
              + detail-type = [
                  + "ECS Service Action",
                ]
              + source      = [
                  + "aws.ecs",
                ]
            }
        )
      + force_destroy  = false
      + id             = (known after apply)
      + name           = "javabin-ecs-task-start-impaired"
      + name_prefix    = (known after apply)
      + tags_all       = {
          + "environment" = "production"
          + "managed-by"  = "terraform"
          + "repo"        = "javaBin/platform"
          + "service"     = "platform"
          + "team"        = "platform"
        }
    }

  # module.monitoring.aws_cloudwatch_event_target.ecs_task_start_impaired_sns will be created
  + resource "aws_cloudwatch_event_target" "ecs_task_start_impaired_sns" {
      + arn            = "arn:aws:sns:eu-central-1:553637109631:javabin-security"
      + event_bus_name = "default"
      + force_destroy  = false
      + id             = (known after apply)
      + rule           = "javabin-ecs-task-start-impaired"
      + target_id      = "send-to-security-sns"
    }

Plan: 2 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

---

LLM Review

Risk: 🟢 LOW

Plan updates two Lambda functions' code (source hash change only) and adds a CloudWatch Event rule and SNS target to alert on ECS task start impairment in the platform cluster; 0 destroys.

• ✅ [routine] aws_lambda_function.securityhub_summary is updated in-place: only source_code_hash changes (new deployment package). Configuration, role and other attributes are unchanged; no replacement or destroy.
• ✅ [routine] aws_lambda_function.slack_alert is updated in-place with the same source_code_hash change as securityhub_summary (identical hash, likely a shared package). No configuration changes visible.
• 🔒 [security] New EventBridge rule matches SERVICE_TASK_START_IMPAIRED events for the javabin-platform ECS cluster and routes them to the javabin-security SNS topic. This is an alerting addition, not an exposure: it adds no public access, IAM grants or security group changes. Confirm the SNS topic's access policy permits EventBridge to publish.
• 💰 [cost] Minimal cost: one EventBridge rule and one SNS target. Event volume should be low, and no new billable infrastructure such as NAT gateways, RDS or load balancers is created.
• ✅ [routine] Target uses rule name 'javabin-ecs-task-start-impaired' as a literal string rather than a reference to the rule resource. This works if the name matches, but it is fragile if the rule is renamed.

[github-actions]

Terraform Plan

🚧 Changes detected — Plan: 2 to add, 2 to change, 0 to destroy.

Plan output

Acquiring state lock. This may take a few moments...
aws_ssm_parameter.atlassian_org_id: Refreshing state... [id=/javabin/platform/atlassian-org-id]
aws_ssm_parameter.atlassian_api_key: Refreshing state... [id=/javabin/platform/atlassian-api-key]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # module.lambdas.aws_lambda_function.securityhub_summary will be updated in-place
  ~ resource "aws_lambda_function" "securityhub_summary" {
        id                             = "javabin-securityhub-summary"
      ~ last_modified                  = "2026-10-23T09:00:34.000+0000" -> (known after apply)
      ~ source_code_hash               = "hrcvTHhG+KIWoQVwbPKmZHx4p1RGl3VZPkTdSdhyfKk=" -> "ci2qbjUrBx0+yEslzlkKpQdZ0eQltqrBbHoakTuR6Ik="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.lambdas.aws_lambda_function.slack_alert will be updated in-place
  ~ resource "aws_lambda_function" "slack_alert" {
        id                             = "javabin-slack-alert"
      ~ last_modified                  = "2026-10-23T09:00:40.000+0000" -> (known after apply)
      ~ source_code_hash               = "hrcvTHhG+KIWoQVwbPKmZHx4p1RGl3VZPkTdSdhyfKk=" -> "ci2qbjUrBx0+yEslzlkKpQdZ0eQltqrBbHoakTuR6Ik="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.monitoring.aws_cloudwatch_event_rule.ecs_task_start_impaired will be created
  + resource "aws_cloudwatch_event_rule" "ecs_task_start_impaired" {
      + arn            = (known after apply)
      + description    = "ECS services in the platform cluster that cannot start tasks"
      + event_bus_name = "default"
      + event_pattern  = jsonencode(
            {
              + detail      = {
                  + clusterArn = [
                      + "arn:aws:ecs:eu-central-1:553637109631:cluster/javabin-platform",
                    ]
                  + eventName  = [
                      + "SERVICE_TASK_START_IMPAIRED",
                    ]
                }
              + detail-type = [
                  + "ECS Service Action",
                ]
              + source      = [
                  + "aws.ecs",
                ]
            }
        )
      + force_destroy  = false
      + id             = (known after apply)
      + name           = "javabin-ecs-task-start-impaired"
      + name_prefix    = (known after apply)
      + tags_all       = {
          + "environment" = "production"
          + "managed-by"  = "terraform"
          + "repo"        = "javaBin/platform"
          + "service"     = "platform"
          + "team"        = "platform"
        }
    }

  # module.monitoring.aws_cloudwatch_event_target.ecs_task_start_impaired_sns will be created
  + resource "aws_cloudwatch_event_target" "ecs_task_start_impaired_sns" {
      + arn            = "arn:aws:sns:eu-central-1:553637109631:javabin-security"
      + event_bus_name = "default"
      + force_destroy  = false
      + id             = (known after apply)
      + rule           = "javabin-ecs-task-start-impaired"
      + target_id      = "send-to-security-sns"
    }

Plan: 2 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

---

LLM Review

Risk: 🟢 LOW

Plan updates two existing Lambda functions (code-only source_code_hash change) and adds an EventBridge rule and target that forward ECS task-start-impaired events from the javabin-platform cluster to the javabin-security SNS topic; nothing is destroyed.

• ✅ [routine] Two in-place updates to aws_lambda_function.securityhub_summary and aws_lambda_function.slack_alert. Only source_code_hash and last_modified change, so this is a code redeploy with no configuration, IAM role, or environment changes visible in the plan.
• ✅ [routine] New EventBridge rule javabin-ecs-task-start-impaired matches aws.ecs SERVICE_TASK_START_IMPAIRED events for the javabin-platform cluster only. It is scoped to a single cluster ARN and has no public exposure.
• ✅ [routine] New EventBridge target sends matched events to the existing SNS topic javabin-security. Confirm the topic's access policy allows events.amazonaws.com to publish; this is normally already handled by the topic policy or the target setup.
• 💰 [cost] Negligible cost impact. EventBridge rule matches and SNS publishes are billed per event and only occur when ECS services fail to start tasks.
• 💥 [destruction] No resources are destroyed (0 to destroy). The Lambda updates replace code in place, not the function resources.