From 07f29f7672fce581c954007a588a064e20a21147 Mon Sep 17 00:00:00 2001 From: Marcel Hild Date: Thu, 10 Sep 2026 12:30:38 +0200 Subject: [PATCH 1/6] docs: update gcp-infrastructure for dual WIF pools, Owner role, and IAM incident Reflects the current state after the 2026-09-09 IAM incident and subsequent restoration: documents both WIF pools (fullsend-pool + fullsend-inference), Owner role, org-level IAM bindings for both orgs, and the incident root cause. Co-Authored-By: Claude Opus 4.6 --- docs/gcp-infrastructure.md | 67 ++++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 24 deletions(-) diff --git a/docs/gcp-infrastructure.md b/docs/gcp-infrastructure.md index 815b6a9..bd3860c 100644 --- a/docs/gcp-infrastructure.md +++ b/docs/gcp-infrastructure.md @@ -10,32 +10,39 @@ for the RHDH fullsend setup. | GCP project ID | `rhdh-sidekick-167988` | | GCP project number | `189673402608` | | Vertex AI region | `us-east5` | -| WIF pool | `fullsend-pool` (ACTIVE) | +| WIF pools | `fullsend-pool`, `fullsend-inference` (both ACTIVE) | | IAM admin group | `rhdh-sidekick@redhat.com` | +| Project role | **Owner** (via `group:rhdh-sidekick@redhat.com`) | The project lives under `IT Public Cloud > Sandbox > Customers` in the GCP -org hierarchy. The admin group has `iam.workloadIdentityPoolAdmin`, -`iam.serviceAccountAdmin`, and `iam.serviceAccountKeyAdmin` — sufficient to -self-provision WIF providers and service accounts without fullsend team -involvement. +org hierarchy. The admin group has Owner, which includes all permissions +needed to manage WIF, IAM, and Model Garden. -**Conditional IAM restriction:** The `projectIamAdmin` role on this project -is restricted to granting only `roles/aiplatform.user`: +### IAM incident 2026-09-09 -``` -expression: api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', - []).hasOnly(['roles/aiplatform.user']) -``` +An Ansible/AAP automation (`aap-access-sa@utility-project-468319`) ran +`SetIamPolicy` (full replace, not merge) on `rhdh-sidekick-167988` at +12:59 UTC, overwriting the project policy with only the Owner binding. +This removed `roles/aiplatform.user` from all Fullsend WIF principals. + +Bindings were restored on 2026-09-10 using `add-iam-policy-binding` +(which merges, not replaces). To prevent recurrence, IT/AAP must be told +to use `add-iam-policy-binding` or read-modify-write with etag checks +instead of `set-iam-policy`. -This means you cannot grant yourself additional roles or enable APIs. All -changes beyond `aiplatform.user` must go through IT (ServiceNow ticket). +## WIF pools and providers -## WIF providers +Two pools exist. Each repo gets its own OIDC provider scoped via +`attribute-condition`. -Each repo gets its own WIF provider, scoped via `attribute-condition` to -that specific repository. +### Pools -### Current providers +| Pool | Providers | Created by | +|------|-----------|------------| +| `fullsend-pool` | 3 (manually provisioned) | Team | +| `fullsend-inference` | ~49 (auto-provisioned) | `fullsend admin install` | + +### fullsend-pool providers | Provider | Repo scope | State | |----------|-----------|-------| @@ -43,6 +50,15 @@ that specific repository. | `gh-redhat-developer-rhdh-plugins` | `redhat-developer/rhdh-plugins` | ACTIVE | | `gh-rhdeveloper-plugin-export` | `redhat-developer/rhdh-plugin-export-overlays` | ACTIVE | +The `fullsend-inference` pool has providers for most `redhat-developer/*` +and `rhdh-parasol/*` repos. List them with: + +```bash +gcloud iam workload-identity-pools providers list \ + --location=global --workload-identity-pool=fullsend-inference \ + --project=rhdh-sidekick-167988 --format="table(name.basename(), attributeCondition)" +``` + ### Creating a new WIF provider ```bash @@ -72,16 +88,19 @@ Omitting the second audience causes an `audience mismatch` error at the "Setup GCP" step in the workflow. The `fullsend admin install` CLI sets both automatically; manual provider creation must include both. -### IAM binding +### IAM bindings -The existing `aiplatform.user` binding covers all `redhat-developer` repos -via the `attribute.repository_owner` principal set: +Both pools have org-level `aiplatform.user` bindings for both orgs: -``` -principalSet://iam.googleapis.com/projects/189673402608/locations/global/workloadIdentityPools/fullsend-pool/attribute.repository_owner/redhat-developer -``` +| Pool | Org | Principal set | +|------|-----|---------------| +| `fullsend-pool` | `redhat-developer` | `principalSet://.../fullsend-pool/attribute.repository_owner/redhat-developer` | +| `fullsend-pool` | `rhdh-parasol` | `principalSet://.../fullsend-pool/attribute.repository_owner/rhdh-parasol` | +| `fullsend-inference` | `redhat-developer` | `principalSet://.../fullsend-inference/attribute.repository_owner/redhat-developer` | +| `fullsend-inference` | `rhdh-parasol` | `principalSet://.../fullsend-inference/attribute.repository_owner/rhdh-parasol` | -No per-repo IAM binding is needed after the initial setup. +No per-repo IAM binding is needed — the org-level principal sets cover all +repos automatically. ## Service accounts From 07282bd530372b804318c1dab9a53be3d425d8b9 Mon Sep 17 00:00:00 2001 From: Marcel Hild Date: Thu, 10 Sep 2026 12:35:18 +0200 Subject: [PATCH 2/6] docs: add mint configuration section Documents the public mint migration (2026-09-10) and how mint URLs are managed via GitHub Actions variables, not workflow file changes. Co-Authored-By: Claude Opus 4.6 --- docs/gcp-infrastructure.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/gcp-infrastructure.md b/docs/gcp-infrastructure.md index bd3860c..a586e9f 100644 --- a/docs/gcp-infrastructure.md +++ b/docs/gcp-infrastructure.md @@ -102,6 +102,32 @@ Both pools have org-level `aiplatform.user` bindings for both orgs: No per-repo IAM binding is needed — the org-level principal sets cover all repos automatically. +## Mint configuration + +The mint URL is stored as a GitHub Actions variable (`FULLSEND_MINT_URL`) +on each repo, not hardcoded in the workflow. The workflow references it via +`${{ vars.FULLSEND_MINT_URL }}`. + +| Repo | Mint URL | +|------|----------| +| `redhat-developer/rhdh-agentic` | `https://mint.fullsend.sh` | +| `redhat-developer/rhdh-plugins` | `https://mint.fullsend.sh` | +| `redhat-developer/rhdh-plugin-export-overlays` | `https://mint.fullsend.sh` | + +Switched from private mint (`https://fullsend-mint-gljhbkcloq-uc.a.run.app`) +to public mint on 2026-09-10. + +To change the mint URL on a repo: + +```bash +gh variable set FULLSEND_MINT_URL --body "https://mint.fullsend.sh" --repo / +``` + +The `repos.yaml` fleet manifest also stores `mint_url`. When running +`fullsend repos install -f repos.yaml`, it sets this variable on each repo +automatically. Direct `gh variable set` is equivalent and takes effect +immediately. + ## Service accounts For local agent runs (not CI). Run `/fullsend help setup` for the full local setup guide. From c6d1eee5a3ea71ec272e7081eb493a466c032495 Mon Sep 17 00:00:00 2001 From: Marcel Hild Date: Thu, 10 Sep 2026 12:54:15 +0200 Subject: [PATCH 3/6] =?UTF-8?q?docs:=20consolidate=20WIF=20pools=20?= =?UTF-8?q?=E2=80=94=20fullsend-pool=20deleted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All 3 providers migrated to fullsend-inference, old pool and its IAM bindings removed. Single pool simplifies IAM management. Co-Authored-By: Claude Opus 4.6 --- docs/gcp-infrastructure.md | 49 +++++++++++++------------------------- 1 file changed, 16 insertions(+), 33 deletions(-) diff --git a/docs/gcp-infrastructure.md b/docs/gcp-infrastructure.md index a586e9f..7c691fd 100644 --- a/docs/gcp-infrastructure.md +++ b/docs/gcp-infrastructure.md @@ -10,7 +10,7 @@ for the RHDH fullsend setup. | GCP project ID | `rhdh-sidekick-167988` | | GCP project number | `189673402608` | | Vertex AI region | `us-east5` | -| WIF pools | `fullsend-pool`, `fullsend-inference` (both ACTIVE) | +| WIF pool | `fullsend-inference` (ACTIVE) | | IAM admin group | `rhdh-sidekick@redhat.com` | | Project role | **Owner** (via `group:rhdh-sidekick@redhat.com`) | @@ -30,28 +30,10 @@ Bindings were restored on 2026-09-10 using `add-iam-policy-binding` to use `add-iam-policy-binding` or read-modify-write with etag checks instead of `set-iam-policy`. -## WIF pools and providers +## WIF pool and providers -Two pools exist. Each repo gets its own OIDC provider scoped via -`attribute-condition`. - -### Pools - -| Pool | Providers | Created by | -|------|-----------|------------| -| `fullsend-pool` | 3 (manually provisioned) | Team | -| `fullsend-inference` | ~49 (auto-provisioned) | `fullsend admin install` | - -### fullsend-pool providers - -| Provider | Repo scope | State | -|----------|-----------|-------| -| `gh-redhat-developer-rhdh-agentic` | `redhat-developer/rhdh-agentic` | ACTIVE | -| `gh-redhat-developer-rhdh-plugins` | `redhat-developer/rhdh-plugins` | ACTIVE | -| `gh-rhdeveloper-plugin-export` | `redhat-developer/rhdh-plugin-export-overlays` | ACTIVE | - -The `fullsend-inference` pool has providers for most `redhat-developer/*` -and `rhdh-parasol/*` repos. List them with: +All repos use a single pool (`fullsend-inference`). Each repo gets its own +OIDC provider scoped via `attribute-condition`. List all providers: ```bash gcloud iam workload-identity-pools providers list \ @@ -59,15 +41,18 @@ gcloud iam workload-identity-pools providers list \ --project=rhdh-sidekick-167988 --format="table(name.basename(), attributeCondition)" ``` +A previous `fullsend-pool` (3 manually provisioned providers) was +consolidated into `fullsend-inference` on 2026-09-10 and deleted. + ### Creating a new WIF provider ```bash PROVIDER_NAME="gh-redhat-developer-" # max 32 chars -PROVIDER_PATH="projects/189673402608/locations/global/workloadIdentityPools/fullsend-pool/providers/${PROVIDER_NAME}" +PROVIDER_PATH="projects/189673402608/locations/global/workloadIdentityPools/fullsend-inference/providers/${PROVIDER_NAME}" gcloud iam workload-identity-pools providers create-oidc "$PROVIDER_NAME" \ --location=global \ - --workload-identity-pool=fullsend-pool \ + --workload-identity-pool=fullsend-inference \ --project=rhdh-sidekick-167988 \ --issuer-uri=https://token.actions.githubusercontent.com \ --allowed-audiences="fullsend-mint,https://iam.googleapis.com/${PROVIDER_PATH}" \ @@ -90,14 +75,12 @@ both automatically; manual provider creation must include both. ### IAM bindings -Both pools have org-level `aiplatform.user` bindings for both orgs: +Org-level `aiplatform.user` bindings cover all repos in both orgs: -| Pool | Org | Principal set | -|------|-----|---------------| -| `fullsend-pool` | `redhat-developer` | `principalSet://.../fullsend-pool/attribute.repository_owner/redhat-developer` | -| `fullsend-pool` | `rhdh-parasol` | `principalSet://.../fullsend-pool/attribute.repository_owner/rhdh-parasol` | -| `fullsend-inference` | `redhat-developer` | `principalSet://.../fullsend-inference/attribute.repository_owner/redhat-developer` | -| `fullsend-inference` | `rhdh-parasol` | `principalSet://.../fullsend-inference/attribute.repository_owner/rhdh-parasol` | +| Org | Principal set | +|-----|---------------| +| `redhat-developer` | `principalSet://.../fullsend-inference/attribute.repository_owner/redhat-developer` | +| `rhdh-parasol` | `principalSet://.../fullsend-inference/attribute.repository_owner/rhdh-parasol` | No per-repo IAM binding is needed — the org-level principal sets cover all repos automatically. @@ -236,7 +219,7 @@ all repos under `redhat-developer`): ```bash gcloud projects add-iam-policy-binding rhdh-sidekick-167988 \ --role="roles/aiplatform.user" \ - --member="principalSet://iam.googleapis.com/projects/189673402608/locations/global/workloadIdentityPools/fullsend-pool/attribute.repository_owner/redhat-developer" \ + --member="principalSet://iam.googleapis.com/projects/189673402608/locations/global/workloadIdentityPools/fullsend-inference/attribute.repository_owner/redhat-developer" \ --condition=None ``` @@ -255,7 +238,7 @@ include both: ```bash gcloud iam workload-identity-pools providers update-oidc \ --location=global \ - --workload-identity-pool=fullsend-pool \ + --workload-identity-pool=fullsend-inference \ --project=rhdh-sidekick-167988 \ --allowed-audiences="fullsend-mint,https://iam.googleapis.com/" ``` From 94729be82e28cac482314bdb3c2ca655f74edc6a Mon Sep 17 00:00:00 2001 From: Marcel Hild Date: Thu, 10 Sep 2026 13:01:56 +0200 Subject: [PATCH 4/6] feat: add /fullsend onboard skill and update docs - Add fleet vs. WIF-enabled distinction to gcp-infrastructure.md - Add /fullsend onboard command for easy repo onboarding (WIF, vars, secrets, scaffold) - Update upgrade skill to reflect public mint (no longer private) - Register onboard in SKILL.md command table and routing Co-Authored-By: Claude Opus 4.6 --- .claude/skills/fullsend/SKILL.md | 4 + .claude/skills/fullsend/references/onboard.md | 146 ++++++++++++++++++ .claude/skills/fullsend/references/upgrade.md | 8 +- docs/gcp-infrastructure.md | 17 ++ 4 files changed, 171 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fullsend/references/onboard.md diff --git a/.claude/skills/fullsend/SKILL.md b/.claude/skills/fullsend/SKILL.md index 5a47675..98473e2 100644 --- a/.claude/skills/fullsend/SKILL.md +++ b/.claude/skills/fullsend/SKILL.md @@ -10,6 +10,8 @@ description: | check if yarn/corepack/openspec work, or build a custom fullsend agent. Also use when asked to upgrade fullsend, sync scaffold files with a new version, update the CLI, bump fullsend, or check what version we're on. + Also use when asked to onboard a new repo, add a repo to the fleet, + set up fullsend on a repo, or configure WIF for a new repo. Also use when asked about local fullsend setup, RHDH GCP project config, or the custom sandbox image. --- @@ -91,6 +93,7 @@ To add a variable, create an env file and wire it via `host_files` in the harnes | `debug <#issue> [--repo]` | Run sandbox diagnostics (shortcut for `trigger debug`) | | `comment <#issue> [--repo]` | Post a comment on an issue or PR | | `label <#issue>