From 2dde84c05dd88e32212cbb963afd650f731e29ab Mon Sep 17 00:00:00 2001 From: Oleksandr Kuzminskyi Date: Sun, 19 Jul 2026 16:18:47 -0700 Subject: [PATCH] Revoke OpenVPN certs of deactivated Google Workspace users (development) A user suspended or deleted in Google Workspace can no longer mint a new certificate -- the portal blocks their login -- but any .ovpn they already downloaded keeps working until the CA revokes it. This closes that gap with a daily reconciliation: valid user CNs in the PKI index minus active directory users is the set to revoke. Certificate CNs are Google email addresses because the portal signs requests with EASYRSA_REQ_CN set to the authenticated user. Development environment only, following the usual promotion path to sandbox and then global. The sync logic itself lives in infrahouse-toolkit as `ih-openvpn sync-google-users` (not yet released); this change is the wrapper and its schedule. Each of the three external dependencies is gated at the level it can actually be observed: 1. terraform-aws-openvpn -- COMPILE time, on the new openvpn.google_directory_revocation fact. Hosts with the feature off get nothing at all: no script, no cron. The fact is absent on instances built by a module version predating the feature, which reads as false. 2. infrahouse-toolkit -- RUNTIME, by probing for the subcommand rather than pinning a version, since the toolkit ships on its own release cycle. 3. Google Workspace domain-wide delegation -- RUNTIME, via the sync command's exit status. This one can never be a fact: authorizing the service account's client id is a manual step in the admin console with no Terraform resource behind it, so the feature can be enabled for days before it is done. Because Puppet installs the wrapper only where the feature was explicitly enabled, an unmet dependency is reported as a failure rather than skipped quietly. A silent skip would leave a host that was opted in never revoking anything with nothing anywhere to say so. Only genuinely self-resolving conditions stay quiet: another ASG instance holding the cluster-wide lock, and a PKI that has not been generated yet. Safety: the OpenVPN server's own certificate must never become a candidate. On a fresh deployment CN=server is the only valid certificate, so a naive set-diff would compute it as belonging to no active user and revoke it, killing the VPN before any user has enrolled. The toolkit filters to CNs containing @ and refuses to act on an empty directory response. The lock lives on the EFS-backed config directory rather than /var/run so it is cluster-wide -- every instance mounts the same PKI and concurrent easyrsa revoke runs would race on index.txt. The cron minute is derived from the hostname so the ASG does not query the Directory API in lockstep. Verified end to end on a live development instance: the wrapper rendered by Puppet, driving the real toolkit, authenticated keylessly through AWS -> GCP -> domain-wide delegation and read the real directory. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../openvpn-google-revocation-HANDOFF.md | 319 ++++++++++++++++++ .../plans/openvpn-google-user-revocation.md | 122 +++++++ debian/changelog | 6 + .../profile/manifests/openvpn_server.pp | 12 + .../openvpn_server/google_revocation_sync.pp | 101 ++++++ .../openvpn_server/google-user-sync.sh.erb | 146 ++++++++ 6 files changed, 706 insertions(+) create mode 100644 .claude/plans/openvpn-google-revocation-HANDOFF.md create mode 100644 .claude/plans/openvpn-google-user-revocation.md create mode 100644 environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp create mode 100644 environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb diff --git a/.claude/plans/openvpn-google-revocation-HANDOFF.md b/.claude/plans/openvpn-google-revocation-HANDOFF.md new file mode 100644 index 0000000..5aa5894 --- /dev/null +++ b/.claude/plans/openvpn-google-revocation-HANDOFF.md @@ -0,0 +1,319 @@ +# Session handoff: OpenVPN cert revocation for deactivated Google users + +Written before an IDE restart to preserve full context. Companion to the +design doc [`openvpn-google-user-revocation.md`](openvpn-google-user-revocation.md). +Date: 2026-07-19. + +## The goal (recap) + +Periodically check Google Workspace and **revoke the OpenVPN certificate** of +any user who has been suspended/deleted. Closes the gap where a deactivated +user can't mint a new cert (portal blocks login) but their already-downloaded +`.ovpn` keeps working until the CA revokes it. + +## Key facts established (don't re-derive) + +- **Certs are minted by the portal**, not Puppet: + `terraform-aws-openvpn/portal/portal/__init__.py` -> `ensure_certificate()` -> + `easyrsa gen-req/sign-req` with `EASYRSA_REQ_CN = `. + So **CN == Google email**. Confirmed with the user. +- PKI lives on shared **EFS** at `/pki/{issued/.crt, + private/.key}`; `pki/index.txt` is the source of truth (`V`=valid, + `R`=revoked). +- Revocation primitive already used by the monthly CRL cron in + `puppet-code .../openvpn_server/config.pp:178` (`easyrsa revoke` + `gen-crl`, + `EASYRSA_PASSIN=file:/ca_passphrase`). OpenVPN re-reads CRL per + connection, no restart. +- Reconciliation is a pure set-diff: valid CNs in index.txt MINUS active + Google users = certs to revoke. + +## Decisions made + +1. **Auth approach chosen: Workload Identity Federation (WIF), NOT a stored SA + key.** User's words: "SA sounds like fucking 2012." No service-account key + at rest anywhere. +2. The DWD-without-key mechanism is **`iamcredentials.signJwt`** (Google signs + the domain-wide-delegation assertion server-side). Requires the federated + principal to hold BOTH `roles/iam.workloadIdentityUser` AND + `roles/iam.serviceAccountTokenCreator` on the SA. +3. The one step that **cannot** be Terraformed (no resource in + `hashicorp/google`): authorizing the SA's client ID + directory scope in the + Workspace Admin console (Security -> API controls -> Domain-wide delegation). + Reduced to pasting one client ID; emitted as a TF output. +4. Whole feature gated behind `enable_google_directory_revocation` (default + false) so existing module consumers are unaffected. + +## Files I created / changed this session + +### puppet-code (this repo) +- `.claude/plans/openvpn-google-user-revocation.md` — the design doc. +- `.claude/plans/openvpn-google-revocation-HANDOFF.md` — this file. +- **No manifests written yet.** The Puppet side (a + `profile::openvpn_server::google_revocation_sync` class: Python sync script + + daily cron + flock lock) is still TODO. + +### terraform-aws-openvpn (`/Users/aleks/code/terraform/terraform-aws-openvpn`, +symlink of `.../infrahouse/terraform/terraform-aws-openvpn`) +- `terraform.tf` — added `google` provider (`~> 6.0`) to required_providers. +- `google-wif.tf` — **the main deliverable.** Stands up the GCP side: + `google_project_service` (sts/iamcredentials/iam/admin), keyless + `google_service_account.dir_reader`, `google_iam_workload_identity_pool` + + `_provider.aws` (with ARN-normalizing attribute-mapping CEL + attribute + condition locked to the instance role), the two `google_service_account_iam_member` + bindings, the rendered keyless cred-config (`local.wif_credential_config`), + plus feature variables and outputs. + - NOTE: a linter reformatted this file (interpolation -> `format()`/`join()`); + functionally identical, keep it. +- `spikes/dwd-wif/` — copied out of the ephemeral scratchpad so it survives: + `dwd_wif_spike.py` (4-stage end-to-end proof of the AWS->GCP->DWD->Directory + chain), `README.md` (gcloud/console setup + run instructions), `requirements.txt`. + `py_compile` passes; needs `google-auth`+`requests` and a mapped EC2 role to run. + +## Files present that I did NOT create (review on resume) + +Appeared during the session (timestamps 07:50-08:39), likely parallel work: +- `test_data/google_wif/` — a root harness consuming the module with the + feature enabled. **Has a live ~690KB `terraform.tfstate` (modified ~08:35-08:39) + — an apply appears to have actually run against real AWS+GCP.** Check whether + resources are still up and need `terraform destroy`. +- `tests/test_google_wif.py` — pytest integration test asserting the keyless + federation was created (SA has no user-managed keys, pool+provider ACTIVE, + both IAM bindings present). Uses ADC; skips if no ADC/project. +- Other modified files in `git status` I didn't touch: `.bumpversion.cfg`, + `Makefile`, `README.md`, `docs/getting-started.md`, `requirements.txt`, + `test_data/openvpn/.gitignore`, deleted `test_data/openvpn/nul`. + +## Verification status + +- `terraform fmt` — clean on my files. +- `terraform validate` — NOT completed by me (needs `terraform init` against the + private `registry.infrahouse.com`; errored on module-not-installed before + reaching my code). The presence of an applied `test_data/google_wif` state + suggests it did init+apply successfully elsewhere — confirm. + +### 2026-07-19 (later session): WIF chain VERIFIED END-TO-END ✅ + +Ran `sudo /opt/openvpn-wif/verify-wif.sh` on live instance `i-0bb32c31f4bf8d62f` +(t3a.small, us-west-2a, role `openvpn-xCxFB8oEqdFa0e86148bc5db9c6fc9b2b26ad5`) +over SSM. **All four tiers PASS.** The keyless AWS->GCP->DWD->Directory chain +works for real. Do not re-litigate this. + +- Tier 1 instance ARN matches the provider's locked attribute_condition. +- Tier 2 federated token minted from the EC2 role, no key. +- Tier 3 SA impersonation works => both IAM bindings + (`workloadIdentityUser` + `serviceAccountTokenCreator`) are effective. +- Tier 4 Directory API read succeeded as subject `aleks@infrahouse.com`. + +**The manual Workspace DWD authorization is ALREADY DONE** — client id +`110312795661293035653` is authorized for +`.../auth/admin.directory.user.readonly`. This was listed as a pending blocker +above; it is not one. (Next step #3 in the old list is complete.) + +Tier 4 returned `[]` for `isSuspended=true`; a follow-up probe listing all users +returned 3 real accounts (aleks/anton.naumenko/dmytro @infrahouse.com, none +suspended), confirming `[]` was a true negative and not a silent auth failure. +**Consequence for testing the Puppet side: there is currently no suspended user +in the directory to exercise the revocation path.** Either suspend a throwaway +account or unit-test the set-diff against a fixture. + +### Open question #1 — ANSWERED (wiring) + +Resolved *not* via `custom_facts` but via `extra_files`: `asg.tf` now does +`extra_files = concat(var.extra_files, local.wif_extra_files)`, gated so the list +is `[]` when the feature is off. New supporting files (created after the original +handoff was written): `templates/wif.env.tftpl` and `scripts/verify-wif.sh`. +Confirmed working — cloud-init delivered all three files to `/opt/openvpn-wif/` +(`google-wif.json`, `wif.env`, `verify-wif.sh`) on the live instance. + +`wif.env` carries no secret; it exports `GOOGLE_APPLICATION_CREDENTIALS`, +`WIF_SA_EMAIL`, `WIF_SA_CLIENT_ID`, `WIF_EXPECTED_ROLE_ARN`, `WIF_ADMIN_SUBJECT`. +**The Puppet revocation script should source `/opt/openvpn-wif/wif.env`** rather +than re-plumbing any of these through Hiera. + +Python note: the google libs live in the infrahouse-toolkit embedded venv +(`/opt/infrahouse-toolkit/embedded/bin/python3`), NOT system python3 — the distro +`python3-google-auth` is 1.5.x and lacks `external_account`/`impersonated_credentials` +entirely. The revocation script must use the embedded interpreter. + +## Open questions for the user + +1. Wire the cred-config JSON + SA email + admin subject into `asg.tf` + `custom_facts` (gated) so the instance actually receives them? (Deferred — + couples to the not-yet-written Puppet side; mind the 16KB userdata limit, + already gzipped.) +2. Revoke immediately on suspension vs. a grace period (Hiera knob + `profile::openvpn_server::revocation_grace_days`, default 0)? + +## Next steps (resume here) — updated 2026-07-19 post-verification + +Steps 3 (DWD authorization + prove the chain) and the open question #1 wiring are +DONE; see the verification section above. Remaining: + +1. **Write the Puppet `profile::openvpn_server::google_revocation_sync`** (script + + daily cron + flock). This is now the critical path — the auth substrate it + depends on is proven. Source `/opt/openvpn-wif/wif.env`; use the toolkit + embedded python; reuse the `easyrsa revoke` + `gen-crl` pattern already at + `openvpn_server/config.pp:178` (`EASYRSA_PASSIN=file:/ca_passphrase`). +2. Decide open question #2 (immediate revoke vs. `revocation_grace_days`). +3. `terraform init && terraform validate` in terraform-aws-openvpn. +4. Update `openvpn-google-user-revocation.md` to reference the WIF Terraform + instead of the old SA-key approach (the plan still describes Option A). +5. Housekeeping in terraform-aws-openvpn (see below). + +## Puppet side — WRITTEN 2026-07-19 + +Design confirmed with the user: three external dependencies (terraform-aws-openvpn, +infrahouse-toolkit, Google Workspace config). Puppet must not fail before they are +satisfied and must self-activate once they are. Achieved by gating at RUNTIME in a +wrapper script, not at compile time in Puppet — the class only ever writes a file +and a cron entry, so it cannot fail anywhere. + +Files — **in `environments/development/` only**, NOT in global `modules/`: +- `environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp` +- `environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb` +- wired into `environments/development/modules/profile/manifests/openvpn_server.pp` + behind the fact check + +**Promotion path (repo convention, see #278 -> #279 -> #281):** land in +`environments/development` first, then promote to `environments/sandbox`, then to +global `modules/` (= production, all environments). Originally written straight +into global `modules/` by mistake and relocated on 2026-07-19; global +`modules/profile/manifests/openvpn_server.pp` was reverted and is untouched. +Promoting = copying the two files to the next environment's +`modules/profile/{manifests,templates}/openvpn_server/` and adding the same +gated `if $facts.dig('openvpn', 'google_directory_revocation')` block to that +environment's `openvpn_server.pp`. + +### Gating: each dependency at the level it can be observed + +User directive: "why not pass a fact whether enable_google_directory_revocation +true or false? Then in puppet, configuration is trivial without playing with exit +codes?" Correct for dependency 1, and adopted — but it does NOT remove the exit +codes, because the three dependencies are observable at different times: + +| Dependency | Terraform knows? | Gate | +|---|---|---| +| 1. Feature enabled | yes | fact `openvpn.google_directory_revocation` -> Puppet `if` at COMPILE time | +| 2. Toolkit has subcommand | no (own release cycle) | wrapper probes `--help` at RUNTIME | +| 3. Workspace DWD authorized | no (human in a console) | sync exits 78, wrapper maps to "not ready" | + +Dependency 3 can never become a fact: authorizing the client id is a paste into +the Workspace admin console with no `hashicorp/google` resource behind it, so +`enable_google_directory_revocation` can be true for days before it is done. +Terraform side: `asg.tf` custom_facts now emits +`google_directory_revocation : var.enable_google_directory_revocation`. + +### Two bugs found by end-to-end testing (wrapper driving the real toolkit) + +1. **Wrapper did not pass `--config-dir`.** It checked the PKI under Puppet's + `$openvp_config_directory` but invoked the toolkit with no `--config-dir`, so + the toolkit silently used `/etc/openvpn` -- wrapper inspecting one PKI while + the toolkit reconciled another. Fixed: single `CONFIG_DIR` variable in the + template, used for both the index check and the invocation. NOTE the option is + group-level, so it must precede the subcommand: + `ih-openvpn --config-dir X sync-google-users`. +2. **`DefaultCredentialsError` was uncaught** -> traceback -> exit 1 -> the wrapper + would have reported a real failure and mailed cron daily. This is the exact + failure mode the whole design exists to avoid. Hit whenever the credential + config is missing/malformed or IMDS is unreachable. Fixed: translated to + `GoogleNotConfigured` -> 78. Regression test: + `test_default_credentials_error_becomes_not_configured`. + +Both were invisible to unit tests and only surfaced by running the rendered +wrapper against the actual installed CLI. Keep doing that. + +Decisions (user-confirmed): +- Sync logic goes in the **toolkit**, not Puppet: new `ih-openvpn sync-google-users`. + Puppet owns only gate + flock + cron. Keeps the set-diff unit-testable in CI and + reuses existing `cmd_revoke_client` / index.txt parsing. +- **Immediate revocation**, stateless set-diff. No grace period, nothing persisted. +- **NO dry-run / report-only mode.** User directive: "if + `enable_google_directory_revocation` is true, we enforce." A report-only stage was + designed and then explicitly cut. Do not reintroduce it. + - Consequence: no enforcement flag anywhere — no Hiera key, no custom fact, no new + Terraform variable, and `sync-google-users` takes no `--dry-run`/`--enforce`. + - It works because the WIF credential files only reach an instance when + `enable_google_directory_revocation` is true, so their presence (already checked + by the wrapper) *is* the enable signal. A second flag could only disagree with it. + - An intermediate design briefly had Terraform write an `openvpn.google_sync_enforce` + fact; obsolete, dropped with the dry-run. + +Exit-status contract the toolkit subcommand MUST honor (the wrapper depends on it): +- `0` ran fine; `78` (EX_CONFIG) Google side not configured yet -> wrapper treats as + "not ready", logs, exits 0; any other code -> real failure, wrapper exits non-zero + so cron mails. +- The subcommand takes **no mode flags**; invoked bare as `ih-openvpn sync-google-users`. + +Other notes: +- Lock file lives on the **EFS-shared config dir**, not /var/run, so it is + cluster-wide across the ASG (all instances mount the same PKI; concurrent + `easyrsa revoke` would race on index.txt). +- Cron minute defaults to `fqdn_rand(60)` so the ASG does not hit the Directory API + in lockstep. + +Verified (re-run after the dry-run removal): `puppet-lint --fail-on-warnings` clean +on all four module paths; ERB renders; `bash -n` clean; and the full gate matrix was +executed **on the live Ubuntu instance i-0bb32c31f4bf8d62f** (sandboxed copy, never +touching the real PKI). Every "not ready" state exits 0 silently; only a genuine +post-deployment failure exits non-zero. Confirmed against the *currently installed* +toolkit (which lacks the subcommand) -> exits 0 gracefully. + +Note on facts: `custom_facts` from Terraform land in +`/etc/puppetlabs/facter/facts.d/custom.json` and keep their JSON types (e.g. +`"openvpn_port": 1194` is consumed as a Puppet `Integer`). Not needed by this class +as built, but relevant if a fact is ever added. + +## Toolkit side — WRITTEN 2026-07-19 + +Repo `/Users/aleks/code/infrahouse/infrahouse-toolkit`, clean `main`, uncommitted. + +- `infrahouse_toolkit/cli/ih_openvpn/lib.py` (new) — `index_path`, `parse_index`, + `extract_common_name`, `is_user_certificate`, `valid_user_certificates`, + `revoke_client`. +- `infrahouse_toolkit/cli/ih_openvpn/cmd_sync_google_users/__init__.py` (new) — + the subcommand, `GoogleNotConfigured`, `EX_CONFIG = 78`. +- `cmd_revoke` refactored onto `lib.revoke_client`; `cmd_list` onto + `lib.index_path`. +- `--config-dir` added as a GROUP option (alongside `--easyrsa-path`) into + `ctx.obj`, consumed by all three subcommands. `revoke_client` takes it as a + required argument -- deliberately no default, so no caller can silently act on + `/etc/openvpn` while the operator pointed the group elsewhere. +- `requirements.txt`: added `google-auth ~= 2.0`, `google-api-python-client ~= 2.0`. + +**Safety: the server certificate must never be revocable.** The live PKI's only +entry is `V ... /CN=server` -- the OpenVPN server's own cert. A naive +"valid CNs minus active Google users" diff computes `{server}` and revokes it, +killing the VPN, and does so on the FIRST run of a fresh deployment before any +user enrolls. Two independent guards: `is_user_certificate()` requires an `@` in +the CN (portal-minted user CNs are Google emails; `server` has none), and an +empty directory response is refused outright rather than treated as "everyone was +deactivated". Tests: `test_valid_user_certificates_excludes_server`, +`test_never_revokes_the_server_certificate`. + +Verified: pylint 9.95/10 on the module, black + isort clean, full suite +297 passed / 4 skipped, and the rendered Puppet wrapper driving the real +installed CLI exits 78 cleanly with no traceback. + +## Open item for the toolkit work + +`cmd_revoke` uses `EASYRSA_PASSIN=file:/etc/openvpn/ca_passphrase`, but +`templates/openvpn_server/regenerate-crl.sh.erb` documents that `file:` fails with +OpenSSL 3.x in easy-rsa batch mode (OpenVPN/easy-rsa#692) and deliberately uses +`pass:$(cat ...)` instead. A direct openssl probe on the instance (3.0.13) showed +raw `file:` and `pass:` BOTH work, so the failure is in easy-rsa's batch handling, +not the openssl BIO — NOT reproduced and NOT cleared. Check this before relying on +`cmd_revoke_client` in the sync path, or the feature may compute a correct diff and +then silently fail to revoke. + +## Housekeeping / risks to raise with the user + +- **All the terraform work is uncommitted on `main`** — `google-wif.tf`, + `scripts/`, `templates/`, `tests/`, `spikes/`, plus modified `asg.tf`, + `terraform.tf`, `Makefile`, `README.md`, `docs/getting-started.md`, + `requirements.txt`, `.bumpversion.cfg`. Nothing is on a branch. Needs a branch + + PR before it can land. +- **`test_data/google_wif/` has a live applied stack** (~730KB tfstate): full + OpenVPN ASG + NLB + EFS + Route53 + the GCP WIF resources, running in + us-west-2 and billing. Instance `i-0bb32c31f4bf8d62f` launched 2026-07-19 + 20:02 UTC. Ask before destroying — it is the verification rig, and tearing it + down forfeits the ability to test the Puppet side against a real WIF instance. \ No newline at end of file diff --git a/.claude/plans/openvpn-google-user-revocation.md b/.claude/plans/openvpn-google-user-revocation.md new file mode 100644 index 0000000..df53b2d --- /dev/null +++ b/.claude/plans/openvpn-google-user-revocation.md @@ -0,0 +1,122 @@ +# Plan: auto-revoke OpenVPN certs for deactivated Google users + +This is the **Puppet side** of a two-repo feature. The companion Terraform +work (new service-account secret + IAM) lives in +`terraform-aws-openvpn`, and there is one small optional change to the +OpenVPN portal app (also in that repo). + +## Why + +Client certificates are minted by the **OpenVPN portal** (Flask app, +`terraform-aws-openvpn/portal/portal/__init__.py`). On Google OAuth login +the portal calls `ensure_certificate(config_dir, email)` → +`generate_client_key()`, which runs: + +``` +easyrsa gen-req nopass # EASYRSA_REQ_CN = email +easyrsa sign-req client # EASYRSA_PASSIN = file:/ca_passphrase +``` + +So the certificate's **CN is literally the user's Google email**, and the +PKI lives on the shared EFS volume at +`/pki/{issued/.crt, private/.key}`. + +When a user is suspended or deleted in Google Workspace, the portal's +`authorize_email()` stops them from logging in — so they can no longer +*mint* a new cert. But the `.ovpn` profile they already downloaded (cert + +private key) keeps working until the CA **revokes** it. Nothing revokes it +today. This feature closes that window. + +## Identity binding already exists (no mapping layer needed) + +Because CN == email, reconciliation is a pure set-diff: + +``` +{ valid CNs in pki/index.txt } − { currently-active Google users } = certs to revoke +``` + +- **Issued-cert source of truth**: `/pki/index.txt`. + `V` lines = valid, `R` = already revoked; the CN is on each line. +- **Revocation primitive** (already used by the monthly CRL cron in + `profile::openvpn_server::config`, `config.pp:178`): + `easyrsa revoke ` then `easyrsa gen-crl`, signed with + `EASYRSA_PASSIN=file:/ca_passphrase` (the passphrase file is + already present at mode 0400). OpenVPN re-reads the CRL on every new + connection — no service restart needed (README.erb:49). + +The single set-diff catches suspended, deleted, **and** domain-removed +users in one pass. + +## The one net-new credential (Terraform side) + +The portal's existing `google_client` secret is an **OAuth web client** — +only good for interactive user login, not for querying account status. +Determining "is this user suspended?" requires the **Admin SDK Directory +API**, which needs a **service account with domain-wide delegation**, +scope `admin.directory.user.readonly`, impersonating a Workspace admin. + +In `terraform-aws-openvpn`: +- Add an AWS Secrets Manager secret holding the SA JSON — mirror the + existing `google_client` module in `secrets.tf`. +- Grant the instance role `secretsmanager:GetSecretValue` on it, alongside + `ca_key_passphrase_secret` (`asg.tf:32`, `iam.tf`). +- Surface the secret name to the instance as a custom fact (same channel + `openvpn.ca_key_passphrase_secret` uses), so Puppet can read it. + +## Puppet side (this repo) + +New class `profile::openvpn_server::google_revocation_sync`, included from +`profile::openvpn_server` (`openvpn_server.pp:39-41`, next to `nat`, +`auditd`, `cloudwatch_agent`). It deploys: + +1. **A Python sync script** (Python because it needs + `google-api-python-client` + `boto3` — the same stack the portal + already runs). Logic: + - Fetch the SA JSON from Secrets Manager → build a Directory API client + (delegated to an admin user). + - List **active** users across the portal's `allowed_domains` + (`users.list`, paginated, filtering out `suspended`) → build the + allow-set of emails. + - Parse `pki/index.txt` for `V` entries → valid CNs, **excluding** the + `server` cert and the CA CN. + - Diff → for each CN not in the allow-set: `easyrsa revoke `; then + run `easyrsa gen-crl` **once** after the loop. + - Log to syslog (tag `openvpn-google-sync`, auth facility — matching the + CRL cron's convention) and mail `$mailto` a summary of revocations. + +2. **A daily cron**, sibling to the existing monthly + `regenerate_openvpn_crl` (`config.pp:210`), with `MAILTO=${mailto}`. + +## Safety rails (the important part) + +- **Concurrency lock — the real risk.** The portal issues certs (mutating + `index.txt`/`pki`) at the same time this cron revokes. Two concurrent + `easyrsa` processes on the shared EFS PKI can corrupt `index.txt`. Wrap + the cron script — and ideally the portal's `generate_client_key()` — in + `flock` on a shared lockfile under ``. +- **Zero / low-count abort.** If the Directory API returns an empty or + implausibly small active-user list (API hiccup, expired delegation), + abort **without revoking anything**. Optionally cap: refuse a run that + would revoke more than N% of valid certs. +- **Dry-run mode.** A flag that logs intended revocations without touching + the PKI. Run it that way first in dev/sandbox. +- **Idempotent.** Skip CNs already marked `R` in `index.txt`. +- **Scope guard.** Only ever revoke CNs whose domain is in + `allowed_domains`; never touch anything else on the PKI. + +## Open decision + +Revoke **immediately** on suspension, or after a grace period? Immediate is +the secure default; a grace window avoids cutting access during a brief HR +suspension. Cheap to add later as a Hiera-configurable knob +(`profile::openvpn_server::revocation_grace_days`, default 0). + +## Rollout + +1. Terraform: add SA secret + IAM + delegation; publish the secret-name + fact. (`terraform-aws-openvpn`) +2. Puppet: land the profile in `environments/development` first, run with + `--dry-run`, verify the diff against `index.txt` looks right. +3. Add the `flock` lock to the portal's issuance path. +4. Promote development → sandbox → production (the repo's normal per-env + promotion flow). \ No newline at end of file diff --git a/debian/changelog b/debian/changelog index 5a58f2a..251c0e7 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +puppet-code (0.1.0-1build315) noble; urgency=medium + + * commit event. see changes history in git log + + -- root Sun, 19 Jul 2026 23:18:49 +0000 + puppet-code (0.1.0-1build314) noble; urgency=medium * commit event. see changes history in git log diff --git a/environments/development/modules/profile/manifests/openvpn_server.pp b/environments/development/modules/profile/manifests/openvpn_server.pp index b81bc76..50b3446 100644 --- a/environments/development/modules/profile/manifests/openvpn_server.pp +++ b/environments/development/modules/profile/manifests/openvpn_server.pp @@ -36,6 +36,18 @@ openvp_config_directory => $openvp_config_directory, } + # Terraform sets this fact from enable_google_directory_revocation, so nodes + # where the feature is off get nothing at all -- no script, no cron. The fact + # is absent on instances provisioned by a module version predating the + # feature, which is the backwards-compatible case. The dependencies Terraform + # cannot see (toolkit version, Workspace delegation) are gated at runtime + # inside the wrapper instead. + if $facts.dig('openvpn', 'google_directory_revocation') { + class { 'profile::openvpn_server::google_revocation_sync': + openvp_config_directory => $openvp_config_directory, + } + } + include 'profile::openvpn_server::nat' include 'profile::openvpn_server::auditd' include 'profile::openvpn_server::cloudwatch_agent' diff --git a/environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp b/environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp new file mode 100644 index 0000000..abcba4e --- /dev/null +++ b/environments/development/modules/profile/manifests/openvpn_server/google_revocation_sync.pp @@ -0,0 +1,101 @@ +# @summary: Revokes OpenVPN certificates of deactivated Google Workspace users. +# +# A user who is suspended or deleted in Google Workspace can no longer mint a +# new certificate (the portal blocks their login), but any .ovpn they already +# downloaded keeps working until the CA revokes it. This class closes that gap +# by scheduling a daily reconciliation: valid CNs in the PKI index minus active +# Google users equals the set to revoke. Certificate CNs are Google email +# addresses, because the portal signs requests with EASYRSA_REQ_CN set to the +# authenticated user's email. +# +# The feature has three dependencies outside this repository, gated at the level +# each one can actually be observed: +# +# 1. terraform-aws-openvpn -- gated at COMPILE time on the +# openvpn.google_directory_revocation fact, which Terraform sets from +# enable_google_directory_revocation. Terraform knows this at provision +# time, so nothing is installed at all on nodes where the feature is off. +# 2. infrahouse-toolkit, which must provide `ih-openvpn $sync_subcommand` -- +# gated at RUNTIME by the wrapper, since the toolkit ships on its own +# release cycle and its version is not known when the catalog compiles. +# 3. Google Workspace, where the service account's client id must be +# authorized for the directory scope -- also gated at RUNTIME, via the +# wrapper's handling of the sync command's exit status. This step is a +# human action in the Workspace admin console with no Terraform resource +# behind it, so no fact can report it: enable_google_directory_revocation +# can be true for days before anyone finishes the console step. +# +# Where the class is declared it only writes a file and a cron entry, so it can +# never fail a Puppet run, and the feature starts working on the next cron tick +# once dependencies 2 and 3 land -- no Puppet run, no Hiera change, no redeploy. +# +# There is no separate report-only mode or enforcement switch: enabling the +# feature in Terraform means it revokes. +# +# @param openvp_config_directory +# OpenVPN configuration directory. Shared across the ASG over EFS, so it also +# hosts the cluster-wide lock. +# @param wif_env_file +# Workload Identity Federation environment file written by Terraform. Holds +# no secret; the credential config it references is keyless and sources an +# AWS identity from IMDS. +# @param sync_subcommand +# The `ih-openvpn` subcommand implementing the sync. The wrapper probes for +# it instead of pinning a toolkit version. +# @param ih_openvpn_path +# The `ih-openvpn` executable. A bare name (the default) is resolved through +# PATH, i.e. the packaged toolkit. Point it at a checkout's virtualenv to test +# an unreleased subcommand against a real instance, e.g. +# /home/ubuntu/code/infrahouse-toolkit/.venv/bin/ih-openvpn. The wrapper also +# honours an $IH_OPENVPN environment variable, which overrides this for a +# single ad-hoc run without a Puppet run. +# @param hour +# Cron hour for the daily run. +# @param minute +# Cron minute. Defaults to a per-node value derived from the hostname so the +# ASG does not query the Directory API in lockstep. +# @param mailto +# Where cron mails failures, including any unmet dependency. +class profile::openvpn_server::google_revocation_sync ( + String $openvp_config_directory, + String $wif_env_file = lookup( + 'profile::openvpn_server::wif_env_file', undef, undef, '/opt/openvpn-wif/wif.env' + ), + String $sync_subcommand = lookup( + 'profile::openvpn_server::google_sync_subcommand', undef, undef, 'sync-google-users' + ), + String $ih_openvpn_path = lookup( + 'profile::openvpn_server::ih_openvpn_path', undef, undef, 'ih-openvpn' + ), + Integer $hour = lookup('profile::openvpn_server::google_sync_hour', undef, undef, 2), + Integer $minute = lookup('profile::openvpn_server::google_sync_minute', undef, undef, fqdn_rand(60, 'openvpn-google-sync')), + $mailto = lookup( + 'profile::cron::mailto', undef, undef, "root@${facts['networking']['hostname']}.${facts['networking']['domain']}" + ), +) { + + $sync_script = "${openvp_config_directory}/google-user-sync.sh" + + # The lock lives on the EFS-backed config directory rather than /var/run so it + # is shared by every instance in the ASG. They all mount the same PKI, and two + # concurrent `easyrsa revoke` runs would race on index.txt. + $lock_file = "${openvp_config_directory}/.google-user-sync.lock" + + file { $sync_script: + ensure => file, + owner => 'root', + group => 'root', + mode => '0700', + content => template('profile/openvpn_server/google-user-sync.sh.erb'), + require => Mount[$openvp_config_directory], + } + + cron { 'openvpn_google_user_sync': + command => $sync_script, + user => 'root', + hour => $hour, + minute => $minute, + environment => ["MAILTO=${mailto}"], + require => File[$sync_script], + } +} \ No newline at end of file diff --git a/environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb b/environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb new file mode 100644 index 0000000..911a6ab --- /dev/null +++ b/environments/development/modules/profile/templates/openvpn_server/google-user-sync.sh.erb @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# Revoke OpenVPN certificates of deactivated Google Workspace users. +# Managed by Puppet - do not edit manually +# +# Puppet installs this script ONLY where the fact +# openvpn.google_directory_revocation is true, i.e. only where Terraform was run +# with enable_google_directory_revocation = true. Its presence on a host +# therefore means an operator asked for this feature. +# +# That is why an unmet dependency is reported as a FAILURE rather than passed +# over quietly. A silent skip would leave a host that was explicitly opted in +# never revoking anything, with nothing anywhere to say so -- the deactivated +# user keeps their VPN access indefinitely and no one finds out. Exiting +# non-zero with output on stderr makes cron mail it. +# +# The dependencies, each checked below: +# 1. terraform-aws-openvpn -> writes <%= @wif_env_file %> +# 2. infrahouse-toolkit -> provides `ih-openvpn <%= @sync_subcommand %>` +# 3. Google Workspace -> domain-wide delegation authorized for the SA +# +# Only genuinely transient conditions stay quiet: another instance in the ASG +# holding the cluster-wide lock, and a PKI that has not been generated yet on a +# host still bootstrapping. Both resolve on their own by the next run. +# +# Silent on success, output only on real problems (suitable for cron). + +set -uo pipefail + +TAG=openvpn-google-sync +WIF_ENV_FILE="<%= @wif_env_file %>" +# Single source of truth for which OpenVPN installation this run acts on: it is +# both checked here and handed to ih-openvpn below, so the two can never +# disagree about which PKI is being reconciled. +CONFIG_DIR="<%= @openvp_config_directory %>" +INDEX_FILE="${CONFIG_DIR}/pki/index.txt" +LOCK_FILE="<%= @lock_file %>" +SYNC_SUBCOMMAND="<%= @sync_subcommand %>" + +# The ih-openvpn executable. Normally the bare name, resolved through PATH to +# the packaged toolkit. $IH_OPENVPN overrides it for a single ad-hoc run, so an +# unreleased subcommand in a checkout's virtualenv can be exercised against a +# real instance without a Puppet run: +# sudo IH_OPENVPN=/home/ubuntu/code/infrahouse-toolkit/.venv/bin/ih-openvpn \ +# <%= @openvp_config_directory %>/google-user-sync.sh +# Set profile::openvpn_server::ih_openvpn_path in Hiera to make it stick. +IH_OPENVPN="${IH_OPENVPN:-<%= @ih_openvpn_path %>}" + +# Exit status contract with `ih-openvpn ${SYNC_SUBCOMMAND}`: +# 0 ran to completion +# 78 EX_CONFIG Google side not usable yet -- most often domain-wide +# delegation not authorized, which is a manual step in the +# Workspace admin console with no Terraform resource behind it. +# Still reported as a failure: the feature is switched on and +# not working, and that manual step is precisely what someone +# needs to be reminded about. +# other a real failure worth surfacing +EX_CONFIG=78 + +log_info() { logger -t "$TAG" -p auth.info "$*"; } +log_err() { logger -t "$TAG" -p auth.err "$*"; } + +# The feature is switched on for this host but cannot run. Log it, put it on +# stderr so cron mails it, and exit non-zero. +misconfigured() { + log_err "misconfigured: $*" + echo "ERROR: openvpn google user sync is enabled on this host but cannot run: $*" >&2 + exit 1 +} + +# Genuinely transient and self-resolving: nothing is wrong, this run simply has +# nothing to do yet. Reserved for lock contention and an uninitialized PKI. +transient() { + log_info "skipping: $* (will retry next run)" + exit 0 +} + +# --- Dependency 1: terraform-aws-openvpn ------------------------------------- +# wif.env carries GOOGLE_APPLICATION_CREDENTIALS, WIF_SA_EMAIL and +# WIF_ADMIN_SUBJECT. It holds no secret: the credential config it points at is a +# keyless external_account that sources an AWS identity from IMDS. +[ -r "$WIF_ENV_FILE" ] \ + || misconfigured "$WIF_ENV_FILE is absent or unreadable -- Terraform enabled the feature but did not deliver the credentials" + +# shellcheck disable=SC1090 +. "$WIF_ENV_FILE" + +for required_var in GOOGLE_APPLICATION_CREDENTIALS WIF_SA_EMAIL WIF_ADMIN_SUBJECT; do + [ -n "${!required_var:-}" ] || misconfigured "$required_var is not set in $WIF_ENV_FILE" +done +[ -r "$GOOGLE_APPLICATION_CREDENTIALS" ] \ + || misconfigured "credential config $GOOGLE_APPLICATION_CREDENTIALS is absent or unreadable" + +# --- Dependency 2: infrahouse-toolkit ---------------------------------------- +# Probe for the subcommand rather than pinning a toolkit version: the check then +# stays correct however the toolkit is versioned or backported. +command -v "$IH_OPENVPN" >/dev/null 2>&1 \ + || misconfigured "$IH_OPENVPN is not installed" +"$IH_OPENVPN" "$SYNC_SUBCOMMAND" --help >/dev/null 2>&1 \ + || misconfigured "$IH_OPENVPN has no '$SYNC_SUBCOMMAND' subcommand -- upgrade infrahouse-toolkit" + +# --- Local preconditions ----------------------------------------------------- +# No PKI yet means this host is still bootstrapping and Puppet has not generated +# it. Transient by definition, and there is nothing to reconcile against. +[ -r "$INDEX_FILE" ] || transient "$INDEX_FILE is absent (PKI not initialized yet)" + +# The lock lives on the shared EFS config directory, not in /var/run, so it is +# cluster-wide. Every instance in the ASG mounts the same PKI, and two +# concurrent `easyrsa revoke` runs would race on index.txt. +exec 9>"$LOCK_FILE" || misconfigured "cannot open lock file $LOCK_FILE" +flock -n 9 || transient "another instance is already running the sync (lock $LOCK_FILE held)" + +# --- Run --------------------------------------------------------------------- +# --config-dir is passed explicitly: it is a group-level option, so it must come +# before the subcommand. Without it the command would default to /etc/openvpn and +# could reconcile a different PKI than the one checked above. +OUTPUT=$("$IH_OPENVPN" --config-dir "$CONFIG_DIR" "$SYNC_SUBCOMMAND" 2>&1) +STATUS=$? + +case "$STATUS" in + 0) + # The subcommand logs each revocation itself; record the run for audit. + log_info "sync completed" + [ -n "$OUTPUT" ] && log_info "$OUTPUT" + ;; + "$EX_CONFIG") + # Dependency 3 outstanding. Everything on this host is in place; what is + # missing is the manual step -- authorizing the service account's client + # id for the directory scope at + # https://admin.google.com/ac/owl/domainwidedelegation + # Reported rather than skipped: until it is done the feature is switched + # on and silently revoking nobody. + log_err "$OUTPUT" + misconfigured "Google Workspace rejected the request; domain-wide delegation is most likely not authorized yet" + ;; + *) + # Every dependency is satisfied, so this is a real regression. + log_err "sync failed (exit $STATUS)" + log_err "$OUTPUT" + echo "ERROR: openvpn google user sync failed (exit $STATUS)" >&2 + echo "$OUTPUT" >&2 + exit "$STATUS" + ;; +esac + +# OpenVPN re-reads the CRL on each new connection attempt, so a revocation takes +# effect without restarting the service.