diff --git a/README.md b/README.md index 70fb844..9b0944c 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,24 @@ -This repository contains the Temporal Cloud Operator: a Kubernetes -[Operator][k8s-operator] that manages the lifecycle of one or more [Temporal -Cloud][tcloud] resources. +# Temporal Cloud Operator +The Temporal Cloud Operator (TCO) is a Kubernetes [Operator][k8s-operator] for +the lifecycle management of Temporal Cloud resources. It lets you define +Temporal Cloud resources declaratively as Kubernetes [Custom Resources][cr] and +keep them in version control, alongside the applications that use them. This +makes creating, changing and retiring those resources automated, auditable and +easy to review. The operator continuously reconciles the state of Temporal Cloud +resources with the desired state specified in the custom resources, reverting +any out-of-band changes. + +[cr]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ [k8s-operator]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ -[tcloud]: https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction -# How It Works +## How It Works -After installing the Temporal Cloud Operator (TCO), a user can create, modify -or delete Temporal Cloud resources using the `kubectl` CLI. +After installing the Temporal Cloud Operator, a user can create, modify or +delete Temporal Cloud resources using the `kubectl` CLI. -For example, to create a new Temporal Cloud Namespace called -`my-org-namespace`, you might do: +For example, to create a new Temporal Cloud Namespace called `my-org-namespace`, +you might do: ```bash $ kubectl apply -f - < +TCO_REPO=https://github.com/temporalio/cloud-operator -As a best practice, use a service account key when possible, to contain the -blast radius. Creating a service account needs the Owner or Admin role. With -the Developer role, your only option is a personal API key for local runs. +# 1. Install the operator +kubectl apply -k "$TCO_REPO//deploy/kustomize/default?ref=$TCO_VERSION" +kubectl wait --for=condition=Established --timeout=60s crd/clusterconnections.cloud.temporal.io -#### 1. Check your role +# 2. Store the API key +kubectl create secret generic temporal-cloud-api-key \ + -n temporal-cloud-operator --from-literal=apikey="$TEMPORAL_CLOUD_API_KEY" -``` -make temporal-cloud-cli -bin/temporal-cloud login -bin/temporal-cloud whoami -o json | jq -r .user.spec.access.accountAccess.role -mkdir -p ~/.config/temporalio -export TCO_SA_NAME=cloud-operator-e2e -``` - -#### 2a. Owner or Admin - -If the account doesn't have this service account yet, create it. The Developer -role lets it create namespaces and makes it Admin only on the namespaces it -creates: - -``` -bin/temporal-cloud service-account create --name "$TCO_SA_NAME" \ - --account-role developer \ - --description "e2e tests for github.com/temporalio/cloud-operator" -``` - -Then create your key and save it: - -``` -TCO_SA_ID=$(bin/temporal-cloud service-account list -o json | - jq -er --arg name "$TCO_SA_NAME" '.ServiceAccounts[] | select(.spec.name == $name) | .id') - -(umask 077; bin/temporal-cloud apikey create-for-service-account \ - --service-account-id "$TCO_SA_ID" \ - --display-name "$TCO_SA_NAME-$USER" --expiry-duration 30d \ - -o json | jq -er .token > ~/.config/temporalio/cloud-operator-e2e-api-key) -``` - -#### 2b. Developer - -``` -(umask 077; bin/temporal-cloud apikey create-for-me \ - --display-name "$TCO_SA_NAME-$USER" --expiry-duration 30d \ - -o json | jq -er .token > ~/.config/temporalio/cloud-operator-e2e-api-key) -``` - -The key has your own permissions, so pre-flight warns that it isn't a service -account key. If personal keys are turned off for the account, ask an Admin for -a service account key. - -#### 3. Check it works - -``` -make test-cloud-preflight -``` - -Pre-flight checks the key and its access, and warns about extra rights or -expiry within 7 days. It never creates anything. - -- **Projects:** if the account has Projects, give the service account access to - one project and set `TEST_CLOUD_PROJECT_ID`. Otherwise leave it unset. -- **`TEMPORAL_CLOUD_API_KEY`:** set this to pass the key another way, such as - from CI or a password manager. It overrides the file. - -### Unit testing - -For unit testing, we use simple, [table-driven][table-driven-testing] tests and -the following test libraries: - -* `github.com/stretchr/testify/` -* `github.com/stretchr/testify/mock` - -Unit tests should validate the smallest testable unit of work possible. If your -unit tests cross lots of function/method boundaries, you might not be writing a -unit test but instead building a larger functional or integration test. - -To run unit tests: - -``` -make test-unit -``` - -[table-driven-testing]: https://dave.cheney.net/2019/05/07/prefer-table-driven-tests - -### Functional and end-to-end testing - -For functional and integration testing, we use a declarative testing framework -called [`gdt`][gdt] which allows test authors to focus on clearly defining the -*assertions* that their tests should verify instead of cluttering tests with -lots of setup and implementation details. - -`gdt` test cases are YAML files in the `test/` directory. - -To run all functional end-to-end tests: - -``` -make test-e2e -``` - -To run just the functional tests that verify configuration artifacts are -installed properly: - -``` -make test-e2e-install -``` - -To run functional tests that verify that the Temporal Cloud Operator properly -manages the lifecycle of Temporal Cloud resources: - -``` -make test-e2e-resource -``` - -To run functional tests that verify that the Namespace CRD accepts valid -manifests, including `examples/namespace.yaml`, and rejects invalid ones at -`kubectl apply`: +# 3. Connect the operator to Temporal Cloud +kubectl apply -k "$TCO_REPO//deploy/kustomize/overlays/default-connection?ref=$TCO_VERSION" +# 4. Check the connection +kubectl -n temporal-cloud-operator rollout status deployment/temporal-cloud-operator +kubectl get clusterconnections ``` -make test-e2e-namespace-validation -``` - -#### Tests against Temporal Cloud - -These need the [Temporal Cloud test account](#temporal-cloud-test-account). To check that the harness can -create, check and delete a namespace in Cloud, without the operator: - -``` -make test-cloud-harness -``` - -To write a Cloud test, follow `test/e2e/harness/cloud-basic.yaml`. - -An illustrative example of a test that goes through the operator looks like -the following. It applies a Namespace resource, checks it in Kubernetes, -checks the result in Temporal Cloud with `e2e-cloud`, and cleans up. - -Note: this example can run once [#8](https://github.com/temporalio/cloud-operator/issues/8) (create and update namespaces) and [#9](https://github.com/temporalio/cloud-operator/issues/9) (delete -namespaces) land. -```yaml -name: namespace-lifecycle -description: Applying a Namespace creates it in Temporal Cloud and deleting it removes it. +`kubectl get clusterconnections` should show the `default` connection with +`READY` set to `True`. -depends: - - name: e2e-cloud - - name: kubectl +### Next steps -defaults: - kube: - namespace: $TEST_USER_NAMESPACE +* To create your first cloud resource, see [Managing Temporal Cloud + resources](docs/resources.md). +* For additional installation details, including connections, upgrades and + uninstalling, see [Installing the Temporal Cloud Operator](docs/install.md). -tests: - - name: preflight - exec: e2e-cloud preflight - require: - exit-code: 0 +## Documentation - # Apply the resource. The operator creates the Cloud namespace from its name. - - name: create-namespace - kube.create: manifests/lifecycle.yaml +| Document | Description | +|---|---| +| [Installing the Temporal Cloud Operator](docs/install.md) | Installation details, connections, upgrades and uninstalling. | +| [Managing Temporal Cloud resources](docs/resources.md) | Create, change and delete Temporal Cloud resources. | +| [Development and testing](docs/development.md) | How contributors build and run the operator locally, set up the kind test cluster and Temporal Cloud test account, and run unit and end-to-end tests. | - # The operator reports the resource as ready in K8s. - - name: namespace-ready - kube.get: namespaces.cloud.temporal.io/tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle - timeout: 10m - retry: - interval: 15s - assert: - conditions: - ready: true +## Contributing - # The namespace really exist in Temporal Cloud and is active. - - name: cloud-namespace-active - exec: e2e-cloud namespace-exists --active tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle - timeout: 2m - retry: - interval: 15s +We welcome all contributions! - # The Temporal Cloud namespace has the settings the manifest asked for. - # The expected file lists only the fields to compare, as Cloud names them. - # (namespace-matches is planned, not yet in e2e-cloud.) - - name: cloud-namespace-matches-manifest - exec: e2e-cloud namespace-matches tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle expected/lifecycle-cloud.yaml +* For code contributions, please start by [opening an issue](https://github.com/temporalio/cloud-operator/issues/new) to discuss your idea. +* For bug reports and feature requests, [file an issue](https://github.com/temporalio/cloud-operator/issues). +* For feedback, join the #temporal-workers channel on [Temporal Slack](https://t.mp/slack). - - name: delete-namespace - kube.delete: namespaces.cloud.temporal.io/tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle +## Development and testing - - name: cloud-namespace-absent-after-delete - exec: e2e-cloud namespace-does-not-exist tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle - timeout: 10m - retry: - interval: 15s - - # Cleanup everything from this run in Temporal Cloud. - - name: cleanup - exec: e2e-cloud cleanup-run ${TEST_CLOUD_RUN_ID} - timeout: 10m -``` - -`manifests/lifecycle.yaml`, next to the test file. gdt substitutes environment -variables in manifest files too: - -```yaml -apiVersion: cloud.temporal.io/v1alpha1 -kind: Namespace -metadata: - name: tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle -spec: - description: manager=cloud-operator run-id=${TEST_CLOUD_RUN_ID} owner=${USER} - replicas: - - region: aws-us-west-2 - retentionDays: 1 - apiKeyAuth: - enabled: true -``` - -Tests for other resources follow the same steps, with their own manifest and -`e2e-cloud` checks. - -Cleanup only deletes your own run's namespaces, so several people can share -the test account. - -If a run is interrupted, the next pre-flight lists what it left, with the run -ID and owner. Delete your own with: - -``` -TEMPORAL_CLOUD_CLI=bin/temporal-cloud scripts/e2e/e2e-cloud cleanup-run -``` +Want to try the operator locally? Check out [Development and testing](docs/development.md) for the development setup. -Each test namespace's description in Temporal Cloud records the manager, run -ID and owner, e.g. `manager=cloud-operator run-id=hs6or-a1b2 owner=tomba`. -The owner is `$USER`; set `TEST_CLOUD_OWNER` to override it, for example in CI. +## License -[gdt]: https://github.com/gdt-dev/gdt +This project is licensed under the [Apache License 2.0](LICENSE). diff --git a/deploy/kustomize/overlays/default-connection/cluster_connection.yaml b/deploy/kustomize/overlays/default-connection/cluster_connection.yaml new file mode 100644 index 0000000..5e4a11e --- /dev/null +++ b/deploy/kustomize/overlays/default-connection/cluster_connection.yaml @@ -0,0 +1,11 @@ +apiVersion: cloud.temporal.io/v1alpha1 +kind: ClusterConnection +metadata: + name: default +spec: + hostPort: saas-api.tmprl.cloud:443 + credentials: + apiKey: + secretRef: + name: temporal-cloud-api-key + key: apikey diff --git a/deploy/kustomize/overlays/default-connection/deployment_patch.yaml b/deploy/kustomize/overlays/default-connection/deployment_patch.yaml new file mode 100644 index 0000000..733efa6 --- /dev/null +++ b/deploy/kustomize/overlays/default-connection/deployment_patch.yaml @@ -0,0 +1,13 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: temporal-cloud-operator + namespace: temporal-cloud-operator +spec: + template: + spec: + containers: + - name: cloud-operator + env: + - name: DEFAULT_CLOUD_CONNECTION_NAME + value: default diff --git a/deploy/kustomize/overlays/default-connection/kustomization.yaml b/deploy/kustomize/overlays/default-connection/kustomization.yaml new file mode 100644 index 0000000..a75fdf9 --- /dev/null +++ b/deploy/kustomize/overlays/default-connection/kustomization.yaml @@ -0,0 +1,7 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- ../../default +- cluster_connection.yaml +patches: +- path: deployment_patch.yaml diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..1eef594 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,339 @@ +# Development and testing + +This guide covers building, generating code and testing the Temporal Cloud +Operator. + +* [Developing](#developing) + * [Building](#building) + * [Generating code](#generating-code) +* [Testing](#testing) + * [Test environment](#test-environment) + * [Temporal Cloud test account](#temporal-cloud-test-account) + * [Unit testing](#unit-testing) + * [Functional and end-to-end testing](#functional-and-end-to-end-testing) + * [Tests against Temporal Cloud](#tests-against-temporal-cloud) + +## Developing + +To list every Makefile target, run `make help`. + +### Building + +`make build` builds the binary to `build/cloud-operator`, and `make build-image` +builds the container image. + +### Generating code + +After changing anything under `api/`, regenerate the code and commit the +generated files with your change: + +``` +make gen-all +``` + +`gen-all` runs three targets, which you can also run on their own: + +| Target | Generates | +|---|---| +| `gen-kube-helpers` | Deepcopy and validation Go code for the API types | +| `gen-kube-register` | Scheme and group-version registration Go code | +| `gen-kube-manifests` | The CRDs in `deploy/kustomize/crd/bases` and the ClusterRole in `deploy/kustomize/rbac` | + +## Testing + +This repository does *not* use Ginkgo or the upstream Kubernetes +controller-runtime `envtest` package for testing the binaries and configuration +artifacts for Temporal Cloud Operator. + +### Test environment + +Several Makefile targets automate common tasks for setting up, tearing down, and +using a [KinD][kind] cluster for local testing of Temporal Cloud Operator. + +``` +make test-cluster-create +``` + +Will create the KinD cluster for local testing. + +``` +make test-cluster-delete +``` + +Will tear down the KinD cluster for local testing. + +``` +make test-cluster-load-image +``` + +Will build the Temporal Cloud Operator container image with your local changes +and load the container image into the KinD cluster's Docker registry. + +``` +make test-cluster-reset +``` + +Can be used to delete any existing test cluster environment, create a new one, +build the Temporal Cloud Operator container image and load it into the KinD +cluster's Docker registry. + +``` +make test-deploy +``` + +Will install the Temporal Cloud Operator running your local code along with all +of the CRDs, RBAC resources and related manifests in the `deploy/kustomize` +directory. + +``` +make test-undeploy +``` + +Will remove all CRDs, RBAC roles and Temporal Cloud Operator deployment +resources from the test cluster environment. + +[kind]: https://kind.sigs.k8s.io/ + +### Temporal Cloud test account + +Tests that manage Temporal Cloud resources run against a real Temporal Cloud +test account. They authenticate with an API key using a dedicated service +account. + +As a best practice, use a service account key when possible, to contain the +blast radius. Creating a service account needs the Owner or Admin role. With the +Developer role, your only option is a personal API key for local runs. + +#### 1. Check your role + +``` +make temporal-cloud-cli +bin/temporal-cloud login +bin/temporal-cloud whoami -o json | jq -r .user.spec.access.accountAccess.role +mkdir -p ~/.config/temporalio +export TCO_SA_NAME=cloud-operator-e2e +``` + +#### 2a. Owner or Admin + +If the account doesn't have this service account yet, create it. The Developer +role lets it create namespaces and makes it Admin only on the namespaces it +creates: + +``` +bin/temporal-cloud service-account create --name "$TCO_SA_NAME" \ + --account-role developer \ + --description "e2e tests for github.com/temporalio/cloud-operator" +``` + +Then create your key and save it: + +``` +TCO_SA_ID=$(bin/temporal-cloud service-account list -o json | + jq -er --arg name "$TCO_SA_NAME" '.ServiceAccounts[] | select(.spec.name == $name) | .id') + +(umask 077; bin/temporal-cloud apikey create-for-service-account \ + --service-account-id "$TCO_SA_ID" \ + --display-name "$TCO_SA_NAME-$USER" --expiry-duration 30d \ + -o json | jq -er .token > ~/.config/temporalio/cloud-operator-e2e-api-key) +``` + +#### 2b. Developer + +``` +(umask 077; bin/temporal-cloud apikey create-for-me \ + --display-name "$TCO_SA_NAME-$USER" --expiry-duration 30d \ + -o json | jq -er .token > ~/.config/temporalio/cloud-operator-e2e-api-key) +``` + +The key has your own permissions, so pre-flight warns that it isn't a service +account key. If personal keys are turned off for the account, ask an Admin for a +service account key. + +#### 3. Check it works + +``` +make test-cloud-preflight +``` + +Pre-flight checks the key and its access, and warns about extra rights or expiry +within 7 days. It never creates anything. + +- **Projects:** if the account has Projects, give the service account access to + one project and set `TEST_CLOUD_PROJECT_ID`. Otherwise leave it unset. +- **`TEMPORAL_CLOUD_API_KEY`:** set this to pass the key another way, such as + from CI or a password manager. It overrides the file. + +### Unit testing + +For unit testing, we use simple, [table-driven][table-driven-testing] tests and +the following test libraries: + +* `github.com/stretchr/testify/` +* `github.com/stretchr/testify/mock` + +Unit tests should validate the smallest testable unit of work possible. If your +unit tests cross lots of function/method boundaries, you might not be writing a +unit test but instead building a larger functional or integration test. + +To run unit tests: + +``` +make test-unit +``` + +[table-driven-testing]: https://dave.cheney.net/2019/05/07/prefer-table-driven-tests + +### Functional and end-to-end testing + +For functional and integration testing, we use a declarative testing framework +called [`gdt`][gdt] which allows test authors to focus on clearly defining the +*assertions* that their tests should verify instead of cluttering tests with +lots of setup and implementation details. + +`gdt` test cases are YAML files in the `test/` directory. + +To run all functional end-to-end tests: + +``` +make test-e2e +``` + +To run just the functional tests that verify configuration artifacts are +installed properly: + +``` +make test-e2e-install +``` + +To run functional tests that verify that the Temporal Cloud Operator properly +manages the lifecycle of Temporal Cloud resources: + +``` +make test-e2e-resource +``` + +To run functional tests that verify that the Namespace CRD accepts valid +manifests, including `examples/namespace.yaml`, and rejects invalid ones at +`kubectl apply`: + +``` +make test-e2e-namespace-validation +``` + +#### Tests against Temporal Cloud + +These need the [Temporal Cloud test account](#temporal-cloud-test-account). To +check that the harness can create, check and delete a namespace in Cloud, +without the operator: + +``` +make test-cloud-harness +``` + +To write a Cloud test, follow `test/e2e/harness/cloud-basic.yaml`. + +An illustrative example of a test that goes through the operator looks like the +following. It applies a Namespace resource, checks it in Kubernetes, checks the +result in Temporal Cloud with `e2e-cloud`, and cleans up. + +Note: this example can run once +[#8](https://github.com/temporalio/cloud-operator/issues/8) (create and update +namespaces) and [#9](https://github.com/temporalio/cloud-operator/issues/9) +(delete namespaces) land. + +```yaml +name: namespace-lifecycle +description: Applying a Namespace creates it in Temporal Cloud and deleting it removes it. + +depends: + - name: e2e-cloud + - name: kubectl + +defaults: + kube: + namespace: $TEST_USER_NAMESPACE + +tests: + - name: preflight + exec: e2e-cloud preflight + require: + exit-code: 0 + + # Apply the resource. The operator creates the Cloud namespace from its name. + - name: create-namespace + kube.create: manifests/lifecycle.yaml + + # The operator reports the resource as ready in K8s. + - name: namespace-ready + kube.get: namespaces.cloud.temporal.io/tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle + timeout: 10m + retry: + interval: 15s + assert: + conditions: + ready: true + + # The namespace really exist in Temporal Cloud and is active. + - name: cloud-namespace-active + exec: e2e-cloud namespace-exists --active tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle + timeout: 2m + retry: + interval: 15s + + # The Temporal Cloud namespace has the settings the manifest asked for. + # The expected file lists only the fields to compare, as Cloud names them. + # (namespace-matches is planned, not yet in e2e-cloud.) + - name: cloud-namespace-matches-manifest + exec: e2e-cloud namespace-matches tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle expected/lifecycle-cloud.yaml + + - name: delete-namespace + kube.delete: namespaces.cloud.temporal.io/tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle + + - name: cloud-namespace-absent-after-delete + exec: e2e-cloud namespace-does-not-exist tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle + timeout: 10m + retry: + interval: 15s + + # Cleanup everything from this run in Temporal Cloud. + - name: cleanup + exec: e2e-cloud cleanup-run ${TEST_CLOUD_RUN_ID} + timeout: 10m +``` + +`manifests/lifecycle.yaml`, next to the test file. gdt substitutes environment +variables in manifest files too: + +```yaml +apiVersion: cloud.temporal.io/v1alpha1 +kind: Namespace +metadata: + name: tco-e2e-${TEST_CLOUD_RUN_ID}-lifecycle +spec: + description: manager=cloud-operator run-id=${TEST_CLOUD_RUN_ID} owner=${USER} + replicas: + - region: aws-us-west-2 + retentionDays: 1 + apiKeyAuth: + enabled: true +``` + +Tests for other resources follow the same steps, with their own manifest and +`e2e-cloud` checks. + +Cleanup only deletes your own run's namespaces, so several people can share the +test account. + +If a run is interrupted, the next pre-flight lists what it left, with the run ID +and owner. Delete your own with: + +``` +TEMPORAL_CLOUD_CLI=bin/temporal-cloud scripts/e2e/e2e-cloud cleanup-run +``` + +Each test namespace's description in Temporal Cloud records the manager, run ID +and owner, e.g. `manager=cloud-operator run-id=hs6or-a1b2 owner=tomba`. The +owner is `$USER`; set `TEST_CLOUD_OWNER` to override it, for example in CI. + +[gdt]: https://github.com/gdt-dev/gdt diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..282ea8c --- /dev/null +++ b/docs/install.md @@ -0,0 +1,161 @@ +# Installing the Temporal Cloud Operator + +This guide is for cluster admins who install and manage the Temporal Cloud +Operator. + +## Prerequisites + +You need a Kubernetes cluster, `kubectl` with cluster admin access and a +Temporal Cloud [API key](https://docs.temporal.io/cloud/api-keys). As a +best practice, use the API key of a [service account](https://docs.temporal.io/cloud/service-accounts) +with the Global Admin [role](https://docs.temporal.io/cloud/manage-access/roles-and-permissions) +(Account Owner also has the access the operator needs, but it's not +recommended). + +## Install + +The recommended way to install the Temporal Cloud Operator is with +[kustomize](https://kustomize.io/), and a Helm chart is planned for [Public +Preview](https://github.com/temporalio/cloud-operator/milestone/2). + +### 1. Install the operator + +Pick a release from the [releases +page](https://github.com/temporalio/cloud-operator/releases) and install it: + +```bash +export TCO_VERSION=v0.1.0 +TCO_REPO=https://github.com/temporalio/cloud-operator + +kubectl apply -k "$TCO_REPO//deploy/kustomize/default?ref=$TCO_VERSION" +kubectl wait --for=condition=Established --timeout=60s \ + crd/clusterconnections.cloud.temporal.io \ + crd/namespaces.cloud.temporal.io +``` + +This creates the `cloud.temporal.io` CustomResourceDefinitions and the +`temporal-cloud-operator` Kubernetes Namespace, with the operator's +ServiceAccount, RBAC rules and Deployment. At this point the operator is +running, but it isn't connected to Temporal Cloud yet. The next two steps store +the API key in a Secret and connect the operator to Temporal Cloud. + +### 2. Store the API key + +Create a Secret to store the API key, so that the connection resource in the +next step can read from it. + +```bash +export TEMPORAL_CLOUD_API_KEY= + +kubectl create secret generic temporal-cloud-api-key \ + -n temporal-cloud-operator --from-literal=apikey="$TEMPORAL_CLOUD_API_KEY" +``` + +In production, it's recommended to create this Secret from a secrets manager +instead, for example with [External Secrets](https://external-secrets.io/), or to +generate it from your own kustomization (see [Manage the install with your own +kustomization](#manage-the-install-with-your-own-kustomization)). + +### 3. Connect the operator to Temporal Cloud + +A connection tells the operator how to connect to the Temporal Cloud Ops API +and where to find the API key. The following creates a default connection +the operator uses for every resource. Support for multiple connections is +planned, to enable one cluster manage more than one Temporal Cloud account, +such as production and staging, or give each team its own service account +and API key. + +```bash +kubectl apply -k "$TCO_REPO//deploy/kustomize/overlays/default-connection?ref=$TCO_VERSION" +``` + +The `default-connection` overlay applies the same install again, plus a +ClusterConnection named `default`: + +```yaml +apiVersion: cloud.temporal.io/v1alpha1 +kind: ClusterConnection +metadata: + name: default +spec: + hostPort: saas-api.tmprl.cloud:443 + credentials: + apiKey: + secretRef: + name: temporal-cloud-api-key + key: apikey +``` + +### 4. Check the connection + +```bash +$ kubectl -n temporal-cloud-operator rollout status deployment/temporal-cloud-operator +$ kubectl get clusterconnections +NAME ENDPOINT READY AGE +default saas-api.tmprl.cloud:443 True 10s +``` + +`READY` is `True` once the operator has read the API key. + +### Manage the install with your own kustomization + +To add your own patches, write a kustomization that uses the `default-connection` +overlay. It can also generate the API key Secret with `secretGenerator`: + +```yaml +# kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- https://github.com/temporalio/cloud-operator//deploy/kustomize/overlays/default-connection?ref=v0.1.0 +secretGenerator: +- name: temporal-cloud-api-key + namespace: temporal-cloud-operator + files: + - apikey=apikey.txt # a local file holding the key and should be kept out of Git + options: + disableNameSuffixHash: true +``` + +Apply it with `kubectl apply -k .` once the CustomResourceDefinitions exist. + +## Upgrade + +Apply the new release the same way it was installed: + +```bash +export TCO_VERSION= + +kubectl apply -k "$TCO_REPO//deploy/kustomize/default?ref=$TCO_VERSION" +kubectl apply -k "$TCO_REPO//deploy/kustomize/overlays/default-connection?ref=$TCO_VERSION" +``` + +Since the Secret isn't part of the install, the upgrade will leave it unchanged. + +## Uninstall + +1. Set every custom resource to retain its corresponding Temporal Cloud resource, so + that the uninstall doesn't delete it: + + ```bash + kubectl annotate namespaces.cloud.temporal.io --all --all-namespaces \ + cloud.temporal.io/deletion-policy=retain --overwrite + ``` + + Without this step, the uninstall will attempt to delete every Temporal Cloud + resource the operator created. + +2. Delete the resources from Kubernetes while the operator is still running, so + it can remove their finalizers: + + ```bash + kubectl delete namespaces.cloud.temporal.io --all --all-namespaces + ``` + +3. Remove the operator, the default connection, the CustomResourceDefinitions + and the `temporal-cloud-operator` Kubernetes Namespace, including the API key + Secret: + + ```bash + kubectl delete -k "$TCO_REPO//deploy/kustomize/overlays/default-connection?ref=$TCO_VERSION" + ``` \ No newline at end of file diff --git a/docs/resources.md b/docs/resources.md new file mode 100644 index 0000000..5460f0a --- /dev/null +++ b/docs/resources.md @@ -0,0 +1,139 @@ +# Managing Temporal Cloud resources + +This guide covers managing the lifecycle of Temporal Cloud resources. It assumes +a cluster admin has [installed the operator](install.md) and connected it to Temporal +Cloud. + +* [How it works](#how-it-works) +* [Namespace](#namespace) + * [Create a Namespace](#create-a-namespace) + * [Modify a Namespace](#modify-a-namespace) + * [Delete a Namespace](#delete-a-namespace) + +## How it works + +Each Temporal Cloud resource is described by a Kubernetes custom resource in the +`cloud.temporal.io` API group. When you apply one, the operator creates the +resource in Temporal Cloud. When you change it, the operator updates the +resource to match. The operator manages every resource through the connection +your cluster admin set up. + +| Kind | Temporal Cloud resource | Availability | +|---|---|---| +| `Namespace` | [Namespace](https://docs.temporal.io/cloud/namespaces) | [Private Preview](https://github.com/temporalio/cloud-operator/milestone/1) | +| `Project`, `ServiceAccount`, `ApiKey` | [Project](https://docs.temporal.io/cloud/projects), [Service Account](https://docs.temporal.io/cloud/service-accounts), [API key](https://docs.temporal.io/cloud/api-keys) | [Public Preview](https://github.com/temporalio/cloud-operator/milestone/2) | + +The resource's `spec` is the source of truth. If a user changes a field directly +in Temporal Cloud, for example in the Temporal Cloud UI, the operator +changes it back on its next check, every 30 seconds by default. + +The resource's `status` shows its state in Temporal Cloud and surfaces any +errors or differences from the spec. To see why something failed, run `kubectl +describe` on the resource and read its conditions. + +## Namespace + +### Create a Namespace + +Describe the Temporal Cloud Namespace in a `Namespace` manifest, such as +[`examples/namespace.yaml`](../examples/namespace.yaml): + +```yaml +apiVersion: cloud.temporal.io/v1alpha1 +kind: Namespace +metadata: + name: payments-prod +spec: + tags: + environment: production + team: payments + replicas: + - region: aws-us-west-2 + retentionDays: 30 + apiKeyAuth: + enabled: true + capacity: + onDemand: {} + description: Payment workflows for production. +``` + +Apply it to the Kubernetes Namespace that holds your Temporal Cloud resources: + +```bash +$ kubectl apply -n temporal-cloud -f examples/namespace.yaml +namespace.cloud.temporal.io/payments-prod created +``` + +The Temporal Cloud Namespace takes its name from `metadata.name`, which must be +a valid Temporal Cloud Namespace name. To list every `spec` field with the +Temporal Cloud field it maps to, run: + +```bash +$ kubectl explain namespaces.spec --api-version=cloud.temporal.io/v1alpha1 +``` + +The Kubernetes API server checks the structure of the manifest when you apply +it: + +```bash +$ kubectl apply -n temporal-cloud -f examples/namespace.yaml # with both onDemand and provisioned capacity +The Namespace "payments-prod" is invalid: spec.capacity: Invalid value: exactly one of onDemand or provisioned must be set +``` + +The API server checks that: + +* `replicas` and `retentionDays` are set, and `replicas` is not empty. +* Enum values are valid, replica regions are unique, and exactly one capacity + mode is set. +* Updates do not add, change or remove `project`, and do not remove search + attributes or change their type. + +Temporal Cloud reports all other errors, such as invalid tags or a retention +period out of range. They appear in the resource's conditions. + +**Note**: For production Namespaces, turn on Temporal Cloud's [deletion +protection](https://docs.temporal.io/cloud/namespaces#delete-protection) so the Namespace can't be +deleted by accident: + +```yaml +spec: + lifecycle: + enableDeleteProtection: true +``` + +### Modify a Namespace + +Edit the manifest and apply it again: + +```bash +$ kubectl apply -n temporal-cloud -f examples/namespace.yaml +namespace.cloud.temporal.io/payments-prod configured +``` + +The operator updates the Temporal Cloud Namespace to match. You can't change a +Namespace's name or `project`, or remove a search attribute. If you change +`metadata.name`, Kubernetes treats it as a new resource. The operator creates a +new, empty Temporal Cloud Namespace, and the old one stays until you delete its +resource. + +### Delete a Namespace + +```bash +$ kubectl delete -n temporal-cloud namespaces.cloud.temporal.io payments-prod +``` + +When you delete the resource, the operator also deletes the Temporal Cloud +Namespace. The resource stays in Kubernetes until Temporal Cloud has finished +deleting it. + +To keep the Temporal Cloud Namespace, set the `cloud.temporal.io/deletion-policy` +annotation to `retain` before you delete the resource: + +```bash +$ kubectl annotate -n temporal-cloud namespaces.cloud.temporal.io payments-prod \ + cloud.temporal.io/deletion-policy=retain +``` + +Temporal Cloud won't delete a Namespace that has deletion protection turned on. +To delete it, set `spec.lifecycle.enableDeleteProtection` to `false` and apply +the manifest first. diff --git a/examples/cluster-connection.yaml b/examples/cluster-connection.yaml new file mode 100644 index 0000000..4fad789 --- /dev/null +++ b/examples/cluster-connection.yaml @@ -0,0 +1,23 @@ +# The default connection to the Temporal Cloud API. This is the same +# ClusterConnection that deploy/kustomize/overlays/default-connection installs. +# +# The API key is read from the `apikey` entry of the Secret +# temporal-cloud-api-key in the operator's Kubernetes Namespace. To create the +# Secret, run: +# +# kubectl create secret generic temporal-cloud-api-key \ +# -n temporal-cloud-operator --from-literal=apikey="$TEMPORAL_CLOUD_API_KEY" +# +# To make this the default connection, set DEFAULT_CLOUD_CONNECTION_NAME=default +# on the operator. See docs/install.md. +apiVersion: cloud.temporal.io/v1alpha1 +kind: ClusterConnection +metadata: + name: default +spec: + hostPort: saas-api.tmprl.cloud:443 + credentials: + apiKey: + secretRef: + name: temporal-cloud-api-key + key: apikey