feat: scaffold upjet-based Crossplane provider for LiteLLM

[Copilot]

Bootstraps a complete upjet v2-based Crossplane provider for LiteLLM, embedding the terraform-provider-litellm SDK directly (no external Terraform CLI). Follows the crossplane-contrib/provider-keycloak pattern end-to-end: Go scaffold, CI/CD, e2e test suite, and release automation. Running make generate will produce the actual managed-resource CRDs and controllers.

Provider scaffold

• cmd/provider/main.go — controller-manager entry point; wires WorkspaceStore, OperationTrackerStore, and the TerraformSetupBuilder
• cmd/generator/main.go — upjet code-gen entry point; cluster-scoped only (pipeline.Run(provider, nil, absRootDir))
• config/provider.go — wraps BerriAI/terraform-provider-litellm via GetProvider(generationProvider bool); all 11 resources use IdentifierFromProvider external names
• internal/clients/litellm.go — reads api_base + api_key from a Kubernetes Secret; type-asserts resource.Managed → resource.ModernManaged (crossplane-runtime v2 moved GetProviderConfigReference off the base interface)
• apis/v1beta1/ — hand-written ProviderConfig / ProviderConfigUsage types; ProviderConfigUsage embeds xpv1.ProviderConfigUsage inline (no Spec wrapper — the ProviderConfigUsageSpec type does not exist in crossplane-runtime v2)

Build & CI

• Makefile — full build system: generate, build, test, e2e, download-tf-provider-platforms
• .github/workflows/ci.yml — detect-noop → lint → check-diff → unit-tests → build → e2e
• .github/workflows/auto-release.yml + tag.yml — semver tagging and OCI image release
• .github/renovate.json, CODEOWNERS, Dockerfile, .golangci.yml
• cluster/test/ — Chainsaw e2e setup (ProviderConfig secret, chainsaw config)

Dependency notes

• go 1.26.5 required (upjet v2.4.2 floor); toolchain go1.26.5 directive added
• terraform-plugin-framework pinned to v1.19.0 — terraform-provider-litellm pulls terraform-plugin-go@v0.31.0 which added GenerateResourceConfig to ProviderServer; framework ≤v1.15.x predates that interface change and fails to compile
• go-openapi/swag pinned to v0.25.5 to fix broken test-dep import in v0.25.4

Original prompt

Follow the approach of https://github.com/crossplane-contrib/provider-keycloak ENTIRELY to build an upjet based provider for litellm based on https://github.com/BerriAI/terraform-provider-litellm. From CI to e2e suites, automatic workflows etc. Other good projects that do everything right are https://github.com/corewire/drop.

[Breee]

@copilot not yet like provider-keycloak. We need everything. (Provider schema diff etc)

[Breee]

@copilot also create a skill for the "perfect upjet Provider" which defines everything we do in Provider-keycloak so I can point an AI agent to it to implement a new Provider for a given terraform Provider.

[Copilot]

@copilot not yet like provider-keycloak. We need everything. (Provider schema diff etc)

Added the missing provider-keycloak pattern components in commit feat: add provider-keycloak pattern scaffold:

Generation pipeline (generate/generate.go)
Full //go:generate chain: cleanup → scraper (provider-metadata.yaml from TF docs) → upjet generator → controller-gen (deepcopy + CRDs) → angryjet (crossplane-runtime methodsets) → upjet resolver (import-cycle fix).

Schema diff tooling

• scripts/version_diff.py — compare two schema.json versions; reports new/removed resources, schema version bumps, attribute changes for generated resources
• scripts/schema_diff_issues.py — reads generated.lst vs schema.json and files a GitHub issue per un-generated resource
• scripts/check_provider_release.py — weekly check for new terraform-provider-litellm releases; files a tracking issue (--mode issue) or auto-opens a bump PR with schema diff (--mode bump)

Workflows

• .github/workflows/schema-diff-issues.yml — dry-run on PRs, creates issues on main/schedule
• .github/workflows/provider-release-check.yml — dry-run on PRs, issue + draft bump PR on schedule

Makefile targets
generated-lst, generated-lst-check, schema-diff OLD_PROVIDER_VERSION=x.y.z, schema-version-diff; generate.done now includes generated-lst.

Supporting files
hack/boilerplate.go.txt, config/generated.lst, cmd/generatedlist/main.go, crossplane-tools dep in go.mod.

[Copilot]

@copilot also create a skill for the "perfect upjet Provider" which defines everything we do in Provider-keycloak so I can point an AI agent to it to implement a new Provider for a given...

Added in 4f24c33: .github/skills/upjet-provider/, a reusable agent skill describing the "perfect upjet provider" as implemented in provider-keycloak, written so it can be pointed at any Terraform provider.

SKILL.md covers the inputs to collect up front (TF repo/version, SDK vs plugin-framework, provider constructor, docs path, auth model, release asset naming), the build order, the non-negotiable rules (Terraform < 1.6 for BSL, no-fork only, committed schema.json/provider-metadata.yaml, external_name.go as source of truth, never hand-edit generated output, CI must fail on a dirty tree after make generate), and the command surface every provider must expose.

references/ holds the detail:

• repository-layout.md — full tree, hand-written vs generated, cluster vs namespaced scoping decision
• build-system.md — Makefile variables, schema acquisition via terraform filesystem mirror, pull-docs sparse clone, generate.init/generate.done, schema-diff/schema-version-diff/crddiff
• config-patterns.md — GetProvider shape, include lists, external-name strategy selection, per-group Configure, KnownReferencers, references/extractors/sensitive/late-init knobs
• clients.md — TerraformSetupBuilder, dual credential secret shapes, config-hash session caching + client pooling, required test cases
• provider-main.md — flags, manager options, tjcontroller.Options, poll jitter, safe-start gating, conversion webhooks
• generation-pipeline.md — every //go:generate stage and what it emits, generated.lst
• testing.md — examples, uptest/chainsaw, cases files, coverage gates, DAG-based targeted e2e
• automation.md — CI jobs, release workflows, schema-diff issues, upstream release check, Renovate handling of the TF provider dep
• troubleshooting.md — the concrete upjet v2 / crossplane-runtime v2 pitfalls (ModernManaged, missing ProviderConfigUsageSpec, terraform-plugin-go v0.31.0 / framework ≥ v1.19.0, resolver import cycles, wrong external-name symptoms)
• checklist.md — completeness checklist to audit a provider against

Also linked from CONTRIBUTING.md.

[Breee]

@copilot get rid of all the old garbage. https://github.com/BerriAI/terraform-provider-litellm/releases/tag/v1.98.0 latest release. also use the latest go versions and tool versions. I dont know where you took them from, but its wrong. also get this ready for a releaseable state

[Copilot]

@copilot get rid of all the old garbage. https://github.com/BerriAI/terraform-provider-litellm/releases/tag/v1.98.0 latest release. also use the latest go versions and tool versions. I dont know wher...

Done in 6bce775, 059fb2e and ea32f65.

Garbage removed

• 68 MB generator binary that was committed to the repo
• stub config/schema.json ({}) and config/provider-metadata.yaml (resources: {})
• placeholder comment-only examples and the stub internal/controller/setup.go that collided with the generated zz_setup.go
• duplicated Makefile recipes (Crossplane CLI, controlplane.up, Crossplane chart download) that shadowed the ones from the build submodule — the submodule gitlink was also never committed, that's fixed too

terraform-provider-litellm v1.98.0
Schema regenerated against v1.98.0 and the full upjet pipeline actually run, so real artifacts are committed now: 11 resources in config/externalname.go (incl. the new litellm_organization_member_add), generated APIs in apis/{litellm,mcp,organization,team,vector}/v1alpha1, controllers, package/crds (13 CRDs) and examples-generated/.

Versions
Go 1.27.0, golangci-lint 2.13.1, controller-tools 0.21.0, kubectl 1.36.4, Crossplane + CLI 2.4.0, kind 0.32.0, chainsaw 0.2.15, uptest 2.2.0, k8s.io/* 0.36.4, controller-runtime 0.24.1. crossplane-runtime stays at v2.2.4 on purpose — upjet v2.4.2 imports apis/common/v1, which was removed in v2.3+. The terraform-plugin-framework pin from the original PR description is gone; v1.98.0 is pure terraform-plugin-sdk/v2.

Releasable state

• runtime image moved to cluster/images/provider-litellm/ on a digest-pinned gcr.io/distroless/static (USER 65532), wired into the standard crossplane build machinery, so make build now produces binaries → image → .xpkg
• registry switched to ghcr.io/corewire (the old xpkg.upbound.io/corewire needed credentials nobody has); make publish pushes the package
• new Release workflow: tag vX.Y.Z → build → push ghcr.io/corewire/provider-litellm:vX.Y.Z → attach the .xpkg to the GitHub release; CI publishes from main/release-*
• found and fixed a real bug while adding unit tests for internal/clients: the setup function asserted resource.ModernManaged, which cluster-scoped MRs never satisfy (they return an untyped *xpv1.Reference), so every reconcile would have failed with "no providerConfigRef provided". Both shapes are handled now and covered by tests.
• README/CONTRIBUTING updated to the actual kinds and API groups (mcp…/Server, vector…/Store, team…/Member, …), Go 1.27 and the real build/release commands.

Locally verified: make generate produces no diff, make lint 0 issues, make test and make build green. The CI runs on this PR are all sitting in action_required (workflow approval for a first-time contributor), so they need your approval to actually execute.

[Breee]

@copilot i want that all resources live under api group litellm.crossplane.io

[Copilot]

@copilot i want that all resources live under api group litellm.crossplane.io

Done in 6584c19. config.RootGroupConfiguration() is now a default resource option that clears ShortGroup and derives the kind from the full Terraform resource name, so everything is generated into litellm.crossplane.io/v1alpha1:

Model, Team, TeamMember, TeamMemberAdd, Organization, OrganizationMember, OrganizationMemberAdd, Key, Credential, McpServer, VectorStore

The previous mcp./team./organization./vector./litellm. subgroups are gone. APIs, controllers (internal/controller/litellm/*), package/crds (all litellm.crossplane.io_*.yaml) and examples-generated/ are regenerated; the hand-written examples, the chainsaw cases list and the README kind table were updated too (examples/mcp/server.yaml → examples/litellm/mcpserver.yaml).

Verified locally: go build ./..., unit tests, and golangci-lint run (0 issues) are green, and the generation pipeline reproduces the committed tree. CI on this PR is still sitting in action_required and needs your approval to run.

[Breee]

@copilot e2e tests? Dev/ci Environment? Deploy a litellm and the Provider and configure it. We need strong e2e tests

[Copilot]

@copilot e2e tests? Dev/ci Environment? Deploy a litellm and the Provider and configure it. We need strong e2e tests

Done. The e2e suite is now fully hermetic — no external LiteLLM endpoint or secrets required.

What was added:

• cluster/test/litellm/deploy.yaml — Deploys a real LiteLLM instance inside the kind cluster (Namespace + ConfigMap with master-key config + Deployment with readiness probe + ClusterIP Service on port 4000).

• cluster/test/setup.sh — Replaced: now deploys LiteLLM in-cluster, waits for it to become ready (up to 5 min), then creates the litellm-credentials Secret and ProviderConfig pointing at http://litellm.litellm.svc.cluster.local:4000.

• cluster/test/chainsaw/ — 11 Chainsaw test suites, one per MR kind:
  model, team, key, mcpserver, credential, organization, teammember, teammemberadd, organizationmember, organizationmemberadd, vectorstore.
  Each suite exercises the full CRUD lifecycle: create → assert Ready=True+Synced=True → update a field → assert the change reconciled → delete → assert gone.
  Relational resources (TeamMember, TeamMemberAdd, OrganizationMember, OrganizationMemberAdd) use a script step to poll status.atProvider.id from the parent resource and inject it into the child manifest before applying.

• Makefile — New chainsaw target (make chainsaw = setup.sh + chainsaw test cluster/test/chainsaw/); make e2e now calls local-deploy chainsaw instead of local-deploy uptest.

• CI — Removed the RUN_E2E == 'true' gate and LITELLM_API_BASE/LITELLM_API_KEY secrets. The e2e-tests job now always runs after a successful build, with Go caching and a failure-diagnostics step (provider pod logs, LiteLLM logs, managed resource status, events).

[Breee]

@copilot and how do we do local development? Use tilt for that.

[Copilot]

@copilot and how do we do local development? Use tilt for that.

Added in the latest commit. Here's what you get with make tilt-up:

• Tiltfile — full local dev loop:

  • Rebuilds the provider binary (go build) on every Go source change under cmd/, internal/, apis/, config/
  • Builds and loads the provider image into the local kind cluster (no registry push needed)
  • Deploys Crossplane (Helm), the in-cluster LiteLLM instance (cluster/test/litellm/deploy.yaml), the litellm-credentials Secret, and the ProviderConfig
  • Port-forwards LiteLLM to localhost:4000 so you can hit the API directly from your machine
  • Streams provider pod logs in the Tilt UI

• tilt-settings.yaml.example — optional overrides (image repo, LiteLLM master key, provider tag); copy to tilt-settings.yaml (git-ignored) to customise

• Makefile targets: make tilt-up (create kind cluster + start Tilt) / make tilt-down (stop Tilt + delete cluster)

• README updated with the quick-start and settings instructions

[Breee]

@copilot why no uptest testcases? look at provider-keycloak.

[Copilot]

@copilot why no uptest testcases? look at provider-keycloak.

Added provider-keycloak-style uptest coverage in commit 0fe3830. cluster/test/cases.txt now points to real LiteLLM manifests for Credential, Key, McpServer, Model, Team, Organization, and VectorStore; the latter includes its required credential and secret. make e2e now runs uptest, while the stronger relational-resource Chainsaw suites remain available via make chainsaw.