Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,8 @@ terraform/state/
| `terraform/modules/ecs-service/` | ECS Fargate service |
| `terraform/modules/service-bucket/` | S3 bucket with IAM policy output |
| `terraform/modules/service-database/` | DynamoDB table with IAM policy output |
| `terraform/modules/service-rds/` | RDS PostgreSQL instance with generated master password |
| `terraform/modules/service-rds-adopted/` | Existing RDS instance imported as-is (`databases[].adopt`); manages tags and deletion protection only |
| `terraform/modules/service-secret/` | SSM Parameter Store SecureString with IAM policy output |
| `terraform/modules/service-queue/` | SQS queue + DLQ with IAM policy output |
| `terraform/modules/service-alarm/` | CloudWatch alarms for ECS service |
Expand Down
31 changes: 30 additions & 1 deletion docs/app-yaml-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,11 +107,13 @@ routing:
host: moresleep.javazone.no # required if routing is specified; also java.no / javabin.no
priority: 100 # required, unique 1-50000
deregistration_delay: 30 # default: 30 (seconds)
adopt_dns: false # default: false — take over an existing record for host
```

- `host` must be exactly one label under `javazone.no`, `java.no` or `javabin.no` (e.g. `moresleep.javazone.no`, `medlem.java.no`, `styret.javabin.no`). Those are the domains with a wildcard cert on the platform ALB; deeper names like `a.b.java.no` are rejected because the wildcard cert does not cover them.
- `host` creates a Route53 A record (in the zone matching the host's domain) pointing to the platform ALB and an ALB listener rule matching the Host header.
- `priority` determines listener rule evaluation order. Must be unique across all apps. Coordinate via the registry.
- `adopt_dns: true` takes over an A record for `host` that already exists (for example a hand-made record of an app moving onto the platform). It must be approved first: the host has to be listed for your team in `ADOPTIONS` in `scripts/registry.py` (a javaBin/platform PR), otherwise generation fails. Route53 records are not protected by team permissions, so this is what stops one team from taking over a hostname another app serves. Without `adopt_dns`, the apply fails because the record exists. With it, Route53 overwrites the record (UPSERT) to point at the platform ALB, and from then on Terraform manages it. The plan shows the record as a *create*. Changing `host` later replaces the record, and removing routing deletes it, as for any managed record. See [Adopting existing resources](#adopting-existing-resources).

#### routing.login — sign-in at the load balancer

Expand Down Expand Up @@ -239,13 +241,40 @@ resources:
backup_retention_period: 7 # days, RDS only, default: 7
multi_az: false # RDS only, default: false
deletion_protection: true # RDS only, default: true
env: DATABASE_URL
env: DATABASE_URL # host:port
host_env: DB_HOST # optional, postgres only: hostname alone
port_env: DB_PORT # optional, postgres only: port alone
```

DynamoDB and PostgreSQL entries can coexist in the same `databases` list. Entries without `engine` (or with `engine: dynamodb`) use the DynamoDB module. Entries with `engine: postgres` or `engine: postgresql` use the RDS module.

RDS instances generate a master password at creation time, stored in SSM Parameter Store at `/{project}/apps/{name}/db-master-password`. The ECS task role automatically receives IAM policies for `rds-db:connect` and `ssm:GetParameter` on the password parameter.

`env` gets the endpoint as `host:port`. Apps that read host and port separately use `host_env` and `port_env` (PostgreSQL only, created or adopted).

##### Adopting an existing PostgreSQL instance

An app that moves onto the platform can keep its existing RDS instance instead of getting a new one:

```yaml
resources:
databases:
- name: main
engine: postgres
adopt: dbmoresleep # DB instance identifier of the existing instance
host_env: DBHOST
port_env: DBPORT
deletion_protection: true # default: true
backup_retention_period: 14 # optional; unset = keep the instance's current value
```

- The generated `main.tf` gets an `import {}` block, and the instance is imported into the app's Terraform state on the first apply. Nothing is created or moved. Once the instance is in state, Terraform skips the block, so it stays in the generated file.
- The `service-rds-adopted` module manages only tags, `deletion_protection` and (when set) `backup_retention_period`. Engine, version, class, storage, network placement, security groups, parameter group, password and monitoring stay as they are in AWS; `instance_class`, `allocated_storage`, `engine_version` and `multi_az` are rejected with `adopt`. The first plan shows `1 to import`, an in-place update of tags and deletion protection, one `random_id` (the suffix of the final snapshot name), and nothing replaced.
- The app keeps connecting the way it did, with its own credentials (put them in `secrets:`). No IAM policy or platform-managed password is added.
- The instance has `prevent_destroy`. To stop managing it without deleting it, add a hand-written file next to the generated ones, e.g. `terraform/release-db.tf`, with `removed { from = module.rds_main.aws_db_instance.this lifecycle { destroy = false } }`, and remove the entry from `app.yaml` in the same change.
- Approval: each adoption is approved in a javaBin/platform PR that lists the identifier for your team in `ADOPTIONS` in `scripts/registry.py`; without it generation fails. Legacy instances are often untagged, and the team permission boundary only protects resources tagged with another team, so an unapproved `adopt` could take over (and tag away) another team's database. After the first apply the instance carries `team=<team>`.
- Network: platform tasks run in the platform VPC's private subnets and reach the internet through its NAT gateway (all database traffic then pays NAT data processing). An adopted instance in another VPC must be reachable from there. If that means a public endpoint, its security group should allow only the NAT address (and any legacy clients that still need it), never `0.0.0.0/0`. The platform does not change its security group, so tighten it yourself.

#### secrets

SSM Parameter Store SecureString parameters. The platform creates each one with the value `PLACEHOLDER`; set the real value with the CLI from the app repo, signed in with your team's SSO permission set (`javabin-team-<team>`):
Expand Down
24 changes: 22 additions & 2 deletions docs/reusable-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ ECR repository with 4-rule lifecycle policy.
ALB target group + listener rule (host-header routing) + Route53 A record (alias to ALB).

**Required inputs:** `name`, `team`, `vpc_id`, `https_listener_arn`, `listener_rule_priority`, `host_header`, `route53_zone_id`, `alb_dns_name`, `alb_zone_id`
**Optional inputs:** `project`, `login_rules` (app.yaml `routing.login`), `login_session_timeout` (default 8 h), `login_client_wait` (default `60s`)
**Optional inputs:** `project`, `login_rules` (app.yaml `routing.login`), `login_session_timeout` (default 8 h), `login_client_wait` (default `60s`), `allow_dns_overwrite` (default false; take over an existing A record for `host_header` with a Route53 UPSERT, set by `routing.adopt_dns` in app.yaml)
**Outputs:** `target_group_arn`, `target_group_arn_suffix`, `dns_name`

`login_rules` (map keyed `p1`, `p2`, ...) adds one listener rule per entry, matching host + paths at the entry's priority (below `listener_rule_priority`). Protected entries get an `authenticate-oidc` action against the chosen Cognito pool, with a per-rule client `{team}-{name}.login-N` requested through a nested `cognito-app-client` (slot `login-N`); group restrictions are enforced by Cognito (auth-claims). `public` entries forward without sign-in. The host rule `depends_on` them, so a new app serves nothing until they exist. `authenticate-cognito` is not used: it needs `cognito-idp:DescribeUserPoolClient` on the platform pools, which the developer boundary denies. Requires the `hashicorp/time` provider (for the wait on the async client creation).
Expand Down Expand Up @@ -119,7 +119,7 @@ RDS PostgreSQL instance in private subnets.

**Password:** Generated once via `random_password`, stored in SSM at `/{project}/apps/{name}/db-master-password`.

**Outputs:** `endpoint`, `port`, `db_name`, `password_parameter_arn`, `access_policy_json`, `security_group_id`
**Outputs:** `endpoint`, `address`, `port`, `db_name`, `password_parameter_arn`, `access_policy_json`, `security_group_id`

**Auto-wiring:** `access_policy_json` grants `rds-db:connect` + `ssm:GetParameter` on the password parameter. Auto-attached to task role via `collect:access_policy_json`.

Expand All @@ -133,6 +133,26 @@ databases:
engine_version: "16"
```


## service-rds-adopted

An existing RDS instance brought under Terraform without changing it (`databases[].adopt` in app.yaml). The generated root module imports it with an `import {}` block.

**Inputs:**

| Input | Default |
|-------|---------|
| `identifier` | required — DB instance identifier |
| `deletion_protection` | true |
| `backup_retention_period` | null (keep the live value) |
| `tags` | `{}` |

**Manages:** tags, `deletion_protection`, `backup_retention_period` (when set), final-snapshot settings (`<identifier>-final-<random hex>`, so a later delete/re-adopt cycle never collides with an old snapshot). Needs AWS provider 5.40 or later. Every other argument is read from the live instance or listed in `ignore_changes`, so importing never plans a replacement. `prevent_destroy` is set.

**Outputs:** `endpoint`, `address`, `port` (string), `arn`, `resource_id`

**Auto-wiring:** none. The app keeps its own credentials (`secrets:`); `env`, `host_env` and `port_env` map to `endpoint`, `address` and `port`.

## service-alarm

CloudWatch alarms for ECS services: CPU high, memory high, unhealthy targets, 5xx errors.
Expand Down
88 changes: 82 additions & 6 deletions scripts/generate-modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from registry import (
PROJECT, DOMAIN, APP_DOMAINS, MODULE_SOURCES,
PROJECT, DOMAIN, APP_DOMAINS, MODULE_SOURCES, ADOPTIONS,
BACKEND_TEMPLATE, PROVIDERS_TEMPLATE, OUTPUTS_TEMPLATE, GITIGNORE_TEMPLATE,
)

Expand Down Expand Up @@ -61,7 +61,7 @@ class AppYamlError(ValueError):
"team": None,
"compute": {"cpu", "memory", "port", "desired_count", "health_check",
"health_check_matcher", "user", "readonly_root_filesystem"},
"routing": {"host", "priority", "deregistration_delay", "login"},
"routing": {"host", "priority", "deregistration_delay", "login", "adopt_dns"},
"domain": None,
"resources": {"buckets", "databases", "secrets", "queues"},
"auth": {"pool", "client", "callback_urls", "logout_urls", "scopes", "groups", "env", "extra_hosts"},
Expand All @@ -73,7 +73,8 @@ class AppYamlError(ValueError):
"buckets": {"name", "env", "versioning", "expire_days"},
"databases": {"name", "env", "engine", "hash_key", "hash_key_type", "range_key",
"range_key_type", "ttl_attribute", "instance_class", "allocated_storage",
"engine_version", "backup_retention_period", "multi_az", "deletion_protection"},
"engine_version", "backup_retention_period", "multi_az", "deletion_protection",
"adopt", "host_env", "port_env"},
"secrets": {"name", "env", "description"},
"queues": {"name", "env", "visibility_timeout", "retention_seconds", "max_receive_count"},
}
Expand Down Expand Up @@ -104,6 +105,12 @@ class AppYamlError(ValueError):
HOSTNAME_RE = re.compile(r"^(?=.{1,253}$)([a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$")
LOCAL_HOSTS = ("localhost", "127.0.0.1")

RDS_IDENTIFIER_RE = re.compile(r"^[a-zA-Z][a-zA-Z0-9-]{0,62}$")

POSTGRES_ENGINES = ("postgres", "postgresql")
# An adopted database keeps its live shape; these would suggest otherwise.
ADOPT_REJECTED = ("instance_class", "allocated_storage", "engine_version", "multi_az")


def host_zone(host):
"""The app domain a host lives under, or None if it is not one label under one."""
Expand Down Expand Up @@ -219,6 +226,38 @@ def login_warnings(rules):
return []


def _check_adoption(team, kind, identifier, where):
if identifier not in ADOPTIONS.get(team, {}).get(kind, set()):
raise AppYamlError(
f"{where}: team {team!r} is not approved to adopt {identifier!r}. Adopting takes over a "
"resource the platform did not create, so each one is approved in a javaBin/platform PR "
f"that adds it to ADOPTIONS[{team!r}][{kind!r}] in scripts/registry.py"
)


def _validate_database(where, db, team):
postgres = str(db.get("engine") or "dynamodb").lower() in POSTGRES_ENGINES
for key in ("host_env", "port_env"):
if key in db:
if not postgres:
raise AppYamlError(f"{where}.{key}: only supported for engine: postgres")
if not ENV_RE.match(str(db[key])):
raise AppYamlError(f"{where}.{key}: {db[key]!r} is not a valid variable name")
if "adopt" not in db:
return
if not postgres:
raise AppYamlError(f"{where}.adopt: only supported for engine: postgres")
if not RDS_IDENTIFIER_RE.match(str(db["adopt"])):
raise AppYamlError(f"{where}.adopt: {db['adopt']!r} is not an RDS DB instance identifier")
_check_adoption(team, "rds", db["adopt"], f"{where}.adopt")
rejected = [k for k in ADOPT_REJECTED if k in db]
if rejected:
raise AppYamlError(
f"{where}: {', '.join(rejected)} cannot be set with adopt — an adopted database keeps "
"its live engine, class, storage and placement; change those outside app.yaml"
)


def validate(app):
"""Raise AppYamlError if app.yaml uses unknown fields or invalid values."""
if not isinstance(app, dict):
Expand Down Expand Up @@ -250,6 +289,10 @@ def validate(app):
if not isinstance(routing.get("priority"), int) or not 1 <= routing["priority"] <= 50000:
raise AppYamlError("routing.priority is required: a unique integer 1-50000")
normalize_login(routing)
if "adopt_dns" in routing and not isinstance(routing["adopt_dns"], bool):
raise AppYamlError("routing.adopt_dns must be true or false")
if routing.get("adopt_dns"):
_check_adoption(app["team"], "dns", host, "routing.adopt_dns")

for rtype, items in (app.get("resources") or {}).items():
if not isinstance(items, list):
Expand All @@ -260,6 +303,8 @@ def validate(app):
raise AppYamlError(f"resources.{rtype}[{i}].name: {item.get('name')!r} is not a valid name")
if item.get("env") and not ENV_RE.match(item["env"]):
raise AppYamlError(f"resources.{rtype}[{i}].env: {item['env']!r} is not a valid variable name")
if rtype == "databases":
_validate_database(f"resources.databases[{i}]", item, app["team"])

env = app.get("environment")
if env is not None:
Expand Down Expand Up @@ -346,6 +391,18 @@ def module_block(name, source, attrs):
return "\n".join(lines)


def import_block(to, import_id):
"""An `import {}` block adopting an existing resource. Once the resource is
in state Terraform skips the block, so it can stay in the generated file."""
return f"import {{\n to = {to}\n id = {hcl_string(import_id)}\n}}"


def _wire_postgres_env(env_map, mod_name, db):
for key, output in (("env", "endpoint"), ("host_env", "address"), ("port_env", "port")):
if db.get(key):
env_map[db[key]] = Ref(f"module.{mod_name}.{output}")


# ---------------------------------------------------------------------------
# Auth
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -461,6 +518,8 @@ def generate(app, account_id, region, platform_ref):
raise AppYamlError("domain: requires routing.priority (unique 1-50000) to create the ALB rule")
routing["host"] = app["domain"]

imports = [] # (resource address, import id) for existing resources to adopt

def src(mod):
path = MODULE_SOURCES[mod]
return f"git::https://github.com/javaBin/platform//{path}?ref={platform_ref}"
Expand Down Expand Up @@ -513,6 +572,9 @@ def src(mod):
"alb_dns_name": Ref("module.platform.alb_dns_name"),
"alb_zone_id": Ref("module.platform.alb_zone_id"),
"deregistration_delay": routing.get("deregistration_delay", 30),
# adopt_dns: take over an existing record (Route53 UPSERT). Only
# emitted when set, so other apps' generated files don't change.
**({"allow_dns_overwrite": True} if routing.get("adopt_dns") else {}),
}
# routing.login: one ALB rule per entry, at the priorities directly
# below routing.priority so they are evaluated before the host rule.
Expand Down Expand Up @@ -556,7 +618,20 @@ def src(mod):
for db in resources.get("databases", []):
dname = db["name"]
engine = (db.get("engine") or "dynamodb").lower()
if engine in ("postgres", "postgresql"):
if engine in POSTGRES_ENGINES and db.get("adopt"):
# Existing instance: imported, never created. Password auth with
# the app's own credentials, so no task-role policy.
mod_name = f"rds_{dname}"
attrs = {
"identifier": db["adopt"],
"deletion_protection": db.get("deletion_protection", True),
}
if db.get("backup_retention_period") is not None:
attrs["backup_retention_period"] = db["backup_retention_period"]
blocks.append(module_block(mod_name, src("service-rds-adopted"), attrs))
imports.append((f"module.{mod_name}.aws_db_instance.this", db["adopt"]))
_wire_postgres_env(env_map, mod_name, db)
elif engine in POSTGRES_ENGINES:
mod_name = f"rds_{dname}"
blocks.append(module_block(mod_name, src("service-rds"), {
"name": dname,
Expand All @@ -573,8 +648,7 @@ def src(mod):
"deletion_protection": db.get("deletion_protection", True),
}))
policy_map[mod_name] = Ref(f"module.{mod_name}.access_policy_json")
if db.get("env"):
env_map[db["env"]] = Ref(f"module.{mod_name}.endpoint")
_wire_postgres_env(env_map, mod_name, db)
elif engine == "dynamodb":
mod_name = f"database_{dname}"
attrs = {
Expand Down Expand Up @@ -739,6 +813,8 @@ def src(mod):
name = "{PROJECT}-alerts"
}}""")

blocks.extend(import_block(to, id_) for to, id_ in imports)

files["main.tf"] = GENERATED + "\n\n".join(blocks) + "\n"

# -- outputs.tf --
Expand Down
15 changes: 15 additions & 0 deletions scripts/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,20 @@
# cert on the platform ALB, so a host must be exactly one label under one.
APP_DOMAINS = ("javazone.no", "java.no", "javabin.no")

# Existing resources a team may adopt with app.yaml (databases[].adopt and
# routing.adopt_dns). Adopting takes over a resource the platform did not
# create: the team's Terraform imports or overwrites it and then owns it.
# Neither Route53 records nor untagged legacy resources are protected by the
# team permission boundary, so each adoption is approved here, in a platform
# PR, instead of being a one-line app.yaml change. Remove an entry once the
# adoption is applied if the resource should never be adopted again.
ADOPTIONS = {
"moresleep": {
"rds": {"dbmoresleep"},
"dns": {"sleepingpill.javazone.no"},
},
}

# ---------------------------------------------------------------------------
# Module source paths (relative to platform repo root)
# ---------------------------------------------------------------------------
Expand All @@ -29,6 +43,7 @@
"service-bucket": "terraform/modules/service-bucket",
"service-database": "terraform/modules/service-database",
"service-rds": "terraform/modules/service-rds",
"service-rds-adopted": "terraform/modules/service-rds-adopted",
"service-secret": "terraform/modules/service-secret",
"service-queue": "terraform/modules/service-queue",
"cognito-app-client": "terraform/modules/cognito-app-client",
Expand Down
Loading
Loading