ci: reap the AWS resources leaked by failed Terraform test runs

[bobbyiliev]

The scratch-account reaper cleans Kinesis streams, S3 buckets, SQS queues and IAM roles named testdrive*, plus EC2 instances tagged scratch-delete-after. None of that covers the AWS Terraform tests: their resources are prefixed aws-test-dev/aws-up-dev, and the harness tags them deleteAfter (which the RequireTagsScratch SCP requires) rather than scratch-delete-after, so nothing ever read it. A run that dies before its teardown therefore leaks its EKS cluster, VPC, NAT gateway, interface endpoints, addresses and log groups permanently, and since the roots use fixed name prefixes the leaked cluster and log group then fail every later run on "already exists". clean_up_terraform_tests sweeps resources carrying the harness's reason tag whose deleteAfter has passed, deleting in dependency order: node groups before their cluster, the log group after it, NAT gateways before the addresses they hold, and the VPC last.

The sweep is best effort and never raises. Some of these deletions are order-sensitive in ways one pass cannot always satisfy, and a leftover that the next run collects is much better than a cleanup step that goes red. Expiry is what keeps it safe: the aws-persistent root sets deleteAfter to 2099 so it is never a candidate, and a temporary run in flight is 24 hours from expiring.

Test plan

Exercising this needs an expired leak in the scratch account, which is exactly the state we just cleared by hand, so it is verified by the next leak rather than on demand. The discovery half was validated against the real account before the cleanup: the same reason tag filter is what identified the two leaked environments, and their deleteAfter values were months in the past. expired_terraform_test_resources is separated out from the deletions so the ARN parsing and expiry filtering can be exercised without touching AWS.

🤖 Generated with Claude Code