feat(terraform): harden data stores with CMK encryption, PITR, and deletion protection

[jeromevdl]

Summary

In-place data-store hardening: customer-managed encryption support, point-in-time recovery, deletion protection, and safer teardown. Safe for existing deployments with zero resource re-creation and zero data loss; new installs get the hardened posture at first deploy.

A terraform plan on an existing (unencrypted-Neptune, AWS-owned-DynamoDB) state shows only ~ update / + create (the optional KMS key) — never -/+ replace.

What's included

• KMS — new terraform/modules/kms with three modes:

  • default (backward-compatible AWS-owned key, no KMS admin needed) — module and example default
  • create (Terraform provisions one shared CMK; rotation on, 30-day deletion window)
  • existing (BYOK via kms_key_arn)

  The key policy authorizes the DynamoDB and RDS (Neptune) service principals through kms:ViaService, scoped to this account — one shared key covers both stores.

• DynamoDB — SSE (CMK-capable) on all tables; PITR + deletion protection on all durable tables (sessions, notifications, agent_questions, agent_outputs, blocks, environment_registry, discussion_read_state, yjs_documents, the git/ tables, and v2_executions).

• Neptune — deletion protection, backup_retention_period (default 7), and a unique final_snapshot_identifier (random suffix) so repeated deploy/destroy cycles never collide on the snapshot name. Prod cannot skip the final snapshot.

• S3 — force_destroy gated on non-prod for application and frontend buckets.

• Teardown — destroy.sh and install.sh refuse an effective prod environment (including a differently named tfvars whose environment value is prod), and clear deletion protection via a two-phase apply whose target list is cross-checked by tests to prevent drift. Break-glass procedure documented.

• Docs — prerequisites (KMS permissions, kms_key_arn for existing keys), setup (prod must be named exactly prod; break-glass teardown ordering), README.

• Tests — new scripts/test/terraform-hardening.test.mjs; new prod-refusal cases in release-tools.test.mjs.

Not included

• Neptune encryption-at-rest — the only replacement-forcing change; needs an opt-in variable plus a snapshot→restore migration, handled separately.

Test plan

• terraform validate clean; terraform fmt clean
• terraform-hardening.test.mjs — 4/4 pass
• release-tools.test.mjs prod-refusal + plan-guard cases pass
• terraform plan against a representative existing state confirms zero -/+ replace

[JWThewes]

Requesting changes for the three resource-lifecycle and teardown issues noted inline: KMS mode changes can delete a key still protecting data, valid HCL syntax can bypass the production teardown refusal, and retrying teardown can recreate resources that were already removed.

The optional encryption defaults, module boundaries, and unchanged resource identities are otherwise sensible and consistent with existing patterns.

Validation: all 51 relevant script tests passed; Terraform formatting, validation, and shell syntax checks passed. A mocked base-to-PR upgrade of the DynamoDB and Neptune modules showed no replacements. The findings were reproduced with mocked Terraform plans or a safe Terraform command stub; no live AWS deployment was exercised.

[JWThewes]

[P1] Keep KMS mode changes from deleting a key still protecting data

Changing from kms_mode = "create" to "existing" using the same key ARN sets this module's count to zero. In a mocked plan, Terraform deletes the key and alias while leaving the tables using that key unchanged, and scripts/inspect-terraform-plan.mjs accepts the plan. Scheduling key deletion makes the key unusable during the waiting period, so this can break table access. Switching to "default" also deletes the old key, which existing backups may still require for restoration. Please separate key retirement from mode selection and include KMS key deletion in the existing persistent-resource plan safeguards.

[JWThewes]

[P1] Resolve the production environment without partially parsing HCL

For an environment alias such as live, valid HCL like environment = "prod" // production evaluates to prod in Terraform but passes this guard because only # comments are stripped. A commented-out environment = "dev" assignment inside a block comment before the real production assignment also bypasses the check. I reproduced both cases with Terraform console and a safe Terraform command stub: the script invokes both the protection-disabling apply and destroy. Please resolve the effective environment through Terraform or a proper HCL parser before any mutation, and cover these valid syntax cases in the refusal tests.

[JWThewes]

[P2] Avoid recreating missing resources while preparing teardown

This targeted apply reconciles complete resources, not just their deletion-protection attributes. After a partial deployment or partially successful destroy, retrying the script creates missing tables, Neptune resources, and their dependencies before trying to delete them again. A mocked plan with these targets and missing resources confirmed creation actions. This adds provisioning requirements and failure points to a cleanup path that previously only destroyed resources; any failed creation also prevents the subsequent destroy because the script uses set -e. Please constrain preparation to protection updates on existing resources and reject creates, replacements, and unrelated changes before applying.

[jeromevdl]

@JWThewes I addressed the three lifecycle/teardown findings in 968232d.

• I actually removed the creation of a CMK. Users can provide their own key or use the default encryption. Users generally use their own KMS key anyway, handled by a dedicated terraform. It simplifies the lifecycle.
• I replaced the partial awk parsing with Terraform evaluation of the effective environment value before any AWS mutation. Regression coverage includes // comments and block-commented assignments.
• Teardown now intersects deletion-protection targets with terraform state list, creates a saved preparation plan, and inspects its JSON before applying it. Only in-place true → false deletion-protection updates are accepted; creates, replacements, deletions, and unrelated updates are rejected. Resources already removed by a partial teardown are skipped, while stale/inconsistent state still fails safely.

Targeted regression tests, formatting/syntax checks, and terraform validate pass. No live AWS deployment.