Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .mk/e2e-cloud.mk
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# ==== Temporal Cloud e2e test targets ====

TEMPORAL_CLOUD_CLI ?= $(BIN_DIR)/temporal-cloud
TEMPORAL_CLOUD_CLI_VERSION ?= v0.2.0
$(TEMPORAL_CLOUD_CLI): | $(BIN_DIR)
@echo -n "installing temporal-cloud@$(TEMPORAL_CLOUD_CLI_VERSION) ... "
@$(call go-install-tool,$(TEMPORAL_CLOUD_CLI),github.com/temporalio/cloud-cli/cmd/temporal-cloud,$(TEMPORAL_CLOUD_CLI_VERSION))
@echo "ok."

# Environment for the Cloud e2e helpers: the pinned CLI, and the helpers on
# PATH so gdt exec steps can call them by name.
E2E_CLOUD_ENV = TEMPORAL_CLOUD_CLI="$(TEMPORAL_CLOUD_CLI)" PATH="$(SCRIPTS_DIR)/e2e:$$PATH"

# Runs gdt on Cloud tests with a fresh run ID, e.g. `@$(CLOUD_GDT) test/e2e/harness`.
# Cloud tests take minutes, so gdt is verbose unless TEST_GDT_ARGS says otherwise.
CLOUD_GDT = export $(E2E_CLOUD_ENV); \
export TEST_CLOUD_RUN_ID="$$(e2e-cloud new-run-id)"; \
echo "run ID: $$TEST_CLOUD_RUN_ID (namespaces tro-e2e-$$TEST_CLOUD_RUN_ID-*)"; \
$(GDT) $(or $(TEST_GDT_ARGS),-v) run

##@ Test (Temporal Cloud)

.PHONY: temporal-cloud-cli
temporal-cloud-cli: $(TEMPORAL_CLOUD_CLI) ## Install the pinned Temporal Cloud CLI into bin/.

.PHONY: test-cloud-preflight
test-cloud-preflight: $(TEMPORAL_CLOUD_CLI) ## Check the Temporal Cloud e2e API key and access.
@$(E2E_CLOUD_ENV) e2e-cloud preflight

.PHONY: test-cloud-harness
test-cloud-harness: $(GDT) $(TEMPORAL_CLOUD_CLI) ## Test that the e2e harness can create, check and clean up a namespace in Temporal Cloud.
@$(CLOUD_GDT) test/e2e/harness
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,6 @@ include .mk/build.mk
include .mk/go.mk
include .mk/gen.mk
include .mk/test.mk
include .mk/e2e-cloud.mk
include .mk/install.mk
include .mk/help.mk
182 changes: 182 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,76 @@ resources from the test cluster environment.

[kind]: https://kind.sigs.k8s.io/

### Temporal Cloud test account

Tests that manage Temporal Cloud resources run against a real Temporal Cloud
test account. They authenticate with an API key using a dedicated service
account.

As a best practice, use a service account key when possible, to contain the
blast radius. Creating a service account needs the Owner or Admin role. With
the Developer role, your only option is a personal API key for local runs.

#### 1. Check your role

```
make temporal-cloud-cli
bin/temporal-cloud login
bin/temporal-cloud whoami -o json | jq -r .user.spec.access.accountAccess.role
mkdir -p ~/.config/temporalio
export TRO_SA_NAME=resource-operator-e2e
```

#### 2a. Owner or Admin

If the account doesn't have this service account yet, create it. The Developer
role lets it create namespaces and makes it Admin only on the namespaces it
creates:

```
bin/temporal-cloud service-account create --name "$TRO_SA_NAME" \
--account-role developer \
--description "e2e tests for github.com/temporalio/resource-operator"
```

Then create your key and save it:

```
TRO_SA_ID=$(bin/temporal-cloud service-account list -o json |
jq -er --arg name "$TRO_SA_NAME" '.ServiceAccounts[] | select(.spec.name == $name) | .id')

(umask 077; bin/temporal-cloud apikey create-for-service-account \
--service-account-id "$TRO_SA_ID" \
--display-name "$TRO_SA_NAME-$USER" --expiry-duration 30d \
-o json | jq -er .token > ~/.config/temporalio/resource-operator-e2e-api-key)
```

#### 2b. Developer

```
(umask 077; bin/temporal-cloud apikey create-for-me \
--display-name "$TRO_SA_NAME-$USER" --expiry-duration 30d \
-o json | jq -er .token > ~/.config/temporalio/resource-operator-e2e-api-key)
```

The key has your own permissions, so pre-flight warns that it isn't a service
account key. If personal keys are turned off for the account, ask an Admin for
a service account key.

#### 3. Check it works

```
make test-cloud-preflight
```

Pre-flight checks the key and its access, and warns about extra rights or
expiry within 7 days. It never creates anything.

- **Projects:** if the account has Projects, give the service account access to
one project and set `TEST_CLOUD_PROJECT_ID`. Otherwise leave it unset.
- **`TEMPORAL_CLOUD_API_KEY`:** set this to pass the key another way, such as
from CI or a password manager. It overrides the file.

### Unit testing

For unit testing, we use simple, [table-driven][table-driven-testing] tests and
Expand Down Expand Up @@ -252,4 +322,116 @@ manifests, including `examples/namespace.yaml`, and rejects invalid ones at
make test-e2e-namespace-validation
```

#### Tests against Temporal Cloud

These need the [Temporal Cloud test account](#temporal-cloud-test-account). To check that the harness can
create, check and delete a namespace in Cloud, without the operator:

```
make test-cloud-harness
```

To write a Cloud test, follow `test/e2e/harness/cloud-basic.yaml`.

An illustrative example of a test that goes through the operator looks like
the following. It applies a Namespace resource, checks it in Kubernetes,
checks the result in Temporal Cloud with `e2e-cloud`, and cleans up.

Note: this example can run once [#8](https://github.com/temporalio/resource-operator/issues/8) (create and update namespaces) and [#9](https://github.com/temporalio/resource-operator/issues/9) (delete
namespaces) land.

```yaml
name: namespace-lifecycle
description: Applying a Namespace creates it in Temporal Cloud and deleting it removes it.

depends:
- name: e2e-cloud
- name: kubectl

defaults:
kube:
namespace: $TEST_USER_NAMESPACE

tests:
- name: preflight
exec: e2e-cloud preflight
require:
exit-code: 0

# Apply the resource. The operator creates the Cloud namespace from its name.
- name: create-namespace
kube.create: manifests/lifecycle.yaml

# The operator reports the resource as ready in K8s.
- name: namespace-ready
kube.get: namespaces.cloud.temporal.io/tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle
timeout: 10m
retry:
interval: 15s
assert:
conditions:
ready: true

# The namespace really exist in Temporal Cloud and is active.
- name: cloud-namespace-active
exec: e2e-cloud namespace-exists --active tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle
timeout: 2m
retry:
interval: 15s

# The Temporal Cloud namespace has the settings the manifest asked for.
# The expected file lists only the fields to compare, as Cloud names them.
# (namespace-matches is planned, not yet in e2e-cloud.)
- name: cloud-namespace-matches-manifest
exec: e2e-cloud namespace-matches tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle expected/lifecycle-cloud.yaml

- name: delete-namespace
kube.delete: namespaces.cloud.temporal.io/tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle

- name: cloud-namespace-absent-after-delete
exec: e2e-cloud namespace-does-not-exist tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle
timeout: 10m
retry:
interval: 15s

# Cleanup everything from this run in Temporal Cloud.
- name: cleanup
exec: e2e-cloud cleanup-run ${TEST_CLOUD_RUN_ID}
timeout: 10m
```

`manifests/lifecycle.yaml`, next to the test file. gdt substitutes environment
variables in manifest files too:

```yaml
apiVersion: cloud.temporal.io/v1alpha1
kind: Namespace
metadata:
name: tro-e2e-${TEST_CLOUD_RUN_ID}-lifecycle
spec:
description: manager=resource-operator run-id=${TEST_CLOUD_RUN_ID} owner=${USER}
replicas:
- region: aws-us-west-2
retentionDays: 1
apiKeyAuth:
enabled: true
```

Tests for other resources follow the same steps, with their own manifest and
`e2e-cloud` checks.

Cleanup only deletes your own run's namespaces, so several people can share
the test account.

If a run is interrupted, the next pre-flight lists what it left, with the run
ID and owner. Delete your own with:

```
TEMPORAL_CLOUD_CLI=bin/temporal-cloud scripts/e2e/e2e-cloud cleanup-run <run ID>
```

Each test namespace's description in Temporal Cloud records the manager, run
ID and owner, e.g. `manager=resource-operator run-id=hs6or-a1b2 owner=tomba`.
The owner is `$USER`; set `TEST_CLOUD_OWNER` to override it, for example in CI.

[gdt]: https://github.com/gdt-dev/gdt
Loading
Loading