ci: move the AWS terraform test roots to AWS provider 6.x

[ggevay]

Status

Testing is blocked on leftover AWS resources. Nightly 18263 ran the first commit; its apply failed at the managed node group (see below) and the teardown could not delete the EKS clusters while the failed node groups were attached (ResourceInUseException: Cluster has nodegroups attached, three attempts). Both test prefixes now have half-torn-down environments in account 400121260767, us-east-1: aws-test-dev-eks and aws-up-dev-eks with their CREATE_FAILED node groups, plus each one's VPC (10 interface endpoints, NAT gateway), KMS key and log group. The harness starts every run from empty state and has no way to adopt or remove leftovers, and the cluster and log-group names are fixed, so a rerun fails within minutes on "already exists". The cloud team has been asked to clean these up; until then the second commit is validated but unexercised on AWS.

Motivation

The Nightly Terraform + Helm Chart E2E on AWS and Upgrade jobs have failed at terraform init on every run since 2026-08-25:

hashicorp/aws: no available releases match the given constraints ~> 5.0, >= 5.92.0, >= 6.0.0, ~> 6.0, >= 6.28.0, >= 6.59.0

The three test roots pin hashicorp/aws ~> 5.0, while every module they pull from materialize-terraform-self-managed at main requires ~> 6.0 since DEP-200 (MaterializeInc/materialize-terraform-self-managed#390). Terraform resolves one provider version per configuration, so the intersection is empty.

Description

Two commits.

1. Bump the aws constraint to ~> 6.0 in aws-temporary, aws-upgrade, and aws-persistent. The other provider pins already satisfy the modules' constraints.
2. Install the VPC CNI and CoreDNS in the roots. Clusters created with EKS module v21 no longer bootstrap either, which the modules repository's examples handle with the aws/modules/vpc-cni and kubernetes/modules/coredns modules. Without them nodes never become Ready and the managed node group fails with NodeCreationFailure: Unhealthy nodes in the kubernetes cluster after 33 minutes, which is how Nightly 18263 (first commit only) failed. The roots now mirror the examples: VPC CNI before the base node group, CoreDNS after it, the partition and account_id passthrough the node-group module asks for when called with a depends_on, and the operator and load-balancer controller depending on CoreDNS.

Verification

terraform init and terraform validate pass for all three roots inside ci-builder. Nightly 18263 exercised the first commit on a real EKS cluster: init, plan and the apply up to the node group all worked, so the provider bump itself is fine. The two AWS nightly jobs on a Nightly of this branch are the check for the second commit, once the account is clean; the branch name matches their *terraform* / *aws* filter, so a trigger-ci Nightly on it runs both.

🤖 Generated with Claude Code

Tracking issue: DEP-245.

[ggevay]

(There is also #38584, which tried to do the same thing.)

[bobbyiliev]

Had a look and this looks right to me. Checked it against the modules repo, the additions match aws/examples/simple and init/validate pass against upstream main.

I cleaned up the leftover AWS resources, so this should be testable now. Also opened #38721, since the teardown couldn't recover from a failed apply and that's what kept every later run broken.

[ggevay]

(Entirely Claude comment)

Nightly 18272 (E2E job only, after the cleanup): terraform apply now completes end to end in 23 minutes, including the managed node group and CoreDNS, so both commits do what they should. The job still failed, for a different reason: environmentd and both clusterd replicas came up, but no balancerd pod ever appeared and the harness gave up after 7.5 minutes.

Cause: with the vpc-cni module the aws-vpc-cni chart runs with enableNetworkPolicy=true (module default), so NetworkPolicies are enforced for the first time in these roots; the EKS-bootstrapped CNI before DEP-200 had enforcement off and the same policies were inert. orchestratord probes environmentd's leader status on the internal HTTP port 6878 when authenticatorKind is None (6876 only for Password/SASL/OIDC), and the operator module's allow-environmentd-egress policy in materialize-terraform-self-managed allows only 6876. The environment is therefore never promoted, and balancerd is only created after promotion. The example passes because it uses authenticator_kind = "Password"; all three roots here use None, which is also the instance module's default. The environment-side allow-orchestratord ingress policy already allows both ports, so the fix is a second ports block (6878) in aws/modules/operator/main.tf of the modules repo.

Teardown: everything was destroyed except what waits on the RDS instance, whose deletion was still running after 55 minutes when the job hit its 90-minute budget (1m51s on 2026-08-24). The account has leftovers again under aws-test-dev (EKS cluster, VPC with endpoints and NAT, KMS key, log group, RDS instance); the job's terraform.tfstate artifact holds all of them, so terraform destroy from that state should clean up.

[bobbyiliev]

Pushed a commit to your branch (hope that's ok, shout if you'd rather I'd opened a separate PR): flipped enable_network_policy and enable_policy_event_logs to false in the three roots.

Your Claude's diagnosis of 18272 was right, I verified each link: apply completes (157 added), the traceback lands on the loop waiting for app=balancerd, the final pod list has environmentd and both clusterd replicas Running with no balancerd at all, and at generation.rs:1504 authenticator_kind = None falls through to environmentd_internal_http_port (6878) while the operator module's allow-environmentd-egress permits only 6876.

Fixing it in the roots rather than upstream because those NetworkPolicies were always there, just inert: before EKS module v21 the bootstrapped CNI had enforcement off, so this is a restore of the previous behaviour rather than a coverage cut. terraform validate and terraform fmt pass.

The 6876/6878 gap is still a genuine thing that we will need to look into in CI here but it's beyond this PR.