From fcb3660fb43a4b502e8112ef068f6bbe2d2825aa Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 11:30:41 +0200 Subject: [PATCH] feat(provider-tck): source the spec artifacts from the open-feature/spec submodule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Gherkin, the canonical flag set and the control API document are not Java artifacts. They are language-agnostic definitions of the provider contract that every language's TCK must agree on byte for byte, and they only lived in this module because the proof of concept had to start somewhere. They now live in open-feature/spec as Appendix F, under specification/assets/provider-tck/, and are copied in from the `spec` git submodule at generate-resources — the same mechanism tools/flagd-api-testkit already uses for the flagd test harness. The copies are git-ignored and carry a do-not-edit note; changes belong in the spec repo and arrive here by bumping the submodule. Consumers are unaffected: the artifacts are still packaged into the release JAR, @SelectClasspathResource("features") still resolves, and nobody needs a submodule of their own. Verified byte-identical after the round trip. The in-memory CI job now checks out submodules, since without them there is no suite to run. DEPENDS ON open-feature/spec#423. The submodule is pinned to that PR's branch commit rather than to a commit on the spec repo's main branch. That is reachable, so CI can fetch it, but it must be re-pinned to main once #423 merges and before this lands. Signed-off-by: Simon Schrottner --- .github/workflows/ci.yml | 5 +- .gitmodules | 3 + tools/provider-tck/.gitignore | 7 + tools/provider-tck/README.md | 37 +- tools/provider-tck/pom.xml | 149 +++++-- tools/provider-tck/spec | 1 + .../main/resources/features/errors.feature | 80 ---- .../resources/features/evaluation.feature | 59 --- .../main/resources/features/events.feature | 42 -- .../main/resources/features/lifecycle.feature | 33 -- .../main/resources/flags/canonical-flags.json | 82 ---- .../main/resources/openapi/control-api.yaml | 368 ------------------ 12 files changed, 162 insertions(+), 704 deletions(-) create mode 100644 tools/provider-tck/.gitignore create mode 160000 tools/provider-tck/spec delete mode 100644 tools/provider-tck/src/main/resources/features/errors.feature delete mode 100644 tools/provider-tck/src/main/resources/features/evaluation.feature delete mode 100644 tools/provider-tck/src/main/resources/features/events.feature delete mode 100644 tools/provider-tck/src/main/resources/features/lifecycle.feature delete mode 100644 tools/provider-tck/src/main/resources/flags/canonical-flags.json delete mode 100644 tools/provider-tck/src/main/resources/openapi/control-api.yaml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index de0eaf2e4..ad34cb8c8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,7 +25,10 @@ jobs: steps: - name: Checkout Repository uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5 - # No submodules: this module's feature files, flags and control-API spec are in-repo. + with: + # The feature files, canonical flag set and control-API document are copied in from + # the open-feature/spec submodule at generate-resources; without it there is no suite. + submodules: recursive - name: Set up JDK 21 uses: actions/setup-java@dded0888837ed1f317902acf8a20df0ad188d165 # v5 diff --git a/.gitmodules b/.gitmodules index 106b0d69f..f2ef1779c 100644 --- a/.gitmodules +++ b/.gitmodules @@ -16,3 +16,6 @@ [submodule "tools/flagd-api-testkit/test-harness"] path = tools/flagd-api-testkit/test-harness url = https://github.com/open-feature/test-harness.git +[submodule "tools/provider-tck/spec"] + path = tools/provider-tck/spec + url = https://github.com/open-feature/spec.git diff --git a/tools/provider-tck/.gitignore b/tools/provider-tck/.gitignore new file mode 100644 index 000000000..4148e2b63 --- /dev/null +++ b/tools/provider-tck/.gitignore @@ -0,0 +1,7 @@ +# Copied from the `spec` submodule at build time (mvn generate-resources). +# Do not edit these files directly — they are the language-agnostic definition of the +# provider contract and live in open-feature/spec, under +# specification/assets/provider-tck/ (Appendix F). +src/main/resources/features/ +src/main/resources/flags/ +src/main/resources/openapi/ diff --git a/tools/provider-tck/README.md b/tools/provider-tck/README.md index 85e252682..e31de0491 100644 --- a/tools/provider-tck/README.md +++ b/tools/provider-tck/README.md @@ -427,19 +427,36 @@ Three steps are new: | `When the resolved value is remembered` / `Then the resolved details value should have changed` | the control API only requires that `/change` changes `changing-flag`'s value, not which value it changes to; asserting a delta keeps the scenario vendor-neutral | | `Then no exception should have been thrown` | makes the "never throws" half of the error contract explicit rather than implicit in a step failure | -## Where these artifacts should live +## Where these artifacts come from -The feature files, the control API spec and the canonical flag set are **not Java artifacts**. They -are language-agnostic definitions of the provider contract that every language's TCK must agree on -byte for byte, and that backend vendors implement in whatever language their testbed is written in. +The feature files, the canonical flag set and the control API document are **not Java artifacts**. +They are language-agnostic definitions of the provider contract that every language's TCK must agree +on byte for byte, and that backend vendors implement in whatever language their testbed is written +in. -They belong in the OpenFeature [spec repository](https://github.com/open-feature/spec), with this -module as their Java delivery vehicle. The three travel together by necessity: a feature file that -evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is -meaningless without the endpoint that produces the disconnect. +They live in the OpenFeature [spec repository](https://github.com/open-feature/spec) as +[Appendix F: Provider Conformance](https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md), +under `specification/assets/provider-tck/`. This module is their Java delivery vehicle: the `spec` +git submodule is updated at `initialize`, the three directories are copied into +`src/main/resources/` at `generate-resources`, and from there they are packaged into the release +JAR. Consumers see no difference — the features stay on the classpath and need no submodule of their +own. -They live here for now only because the PoC had to start somewhere. Moving them changes nothing for -consumers — the features stay on the classpath and stay inside the JAR. +The three travel together by necessity: a feature file that evaluates `boolean-flag` is meaningless +without the flag definition, and a disconnect scenario is meaningless without the control endpoint +that produces the disconnect. + +> **Do not edit `src/main/resources/features/`, `flags/` or `openapi/`.** They are generated and +> git-ignored. Changes belong in `open-feature/spec` and arrive here by bumping the submodule. + +Building this module therefore needs the submodule: + +```bash +git submodule update --init tools/provider-tck/spec +``` + +Maven does this itself at `initialize`, so a plain `mvn verify` works from a fresh clone; the +explicit command is only useful when working offline or inspecting the sources by hand. ## Known gaps diff --git a/tools/provider-tck/pom.xml b/tools/provider-tck/pom.xml index ee5d7ce9e..5d350dd6b 100644 --- a/tools/provider-tck/pom.xml +++ b/tools/provider-tck/pom.xml @@ -45,39 +45,32 @@ @@ -199,4 +192,102 @@ + + + + + org.codehaus.mojo + exec-maven-plugin + 3.6.3 + + + update-spec-submodule + initialize + + exec + + + git + + submodule + update + --init + spec + + + + + + + + + maven-resources-plugin + 3.5.0 + + + copy-provider-tck-gherkin + generate-resources + + copy-resources + + + ${basedir}/src/main/resources/features/ + + + ${basedir}/spec/specification/assets/provider-tck/gherkin/ + + **/*.feature + + + + + + + copy-provider-tck-flags + generate-resources + + copy-resources + + + ${basedir}/src/main/resources/flags/ + + + ${basedir}/spec/specification/assets/provider-tck/flags/ + + **/*.json + + + + + + + copy-provider-tck-openapi + generate-resources + + copy-resources + + + ${basedir}/src/main/resources/openapi/ + + + ${basedir}/spec/specification/assets/provider-tck/openapi/ + + **/*.yaml + + + + + + + + + + diff --git a/tools/provider-tck/spec b/tools/provider-tck/spec new file mode 160000 index 000000000..42b047b9f --- /dev/null +++ b/tools/provider-tck/spec @@ -0,0 +1 @@ +Subproject commit 42b047b9fb1aa37458ec4a3dff2340b188710aff diff --git a/tools/provider-tck/src/main/resources/features/errors.feature b/tools/provider-tck/src/main/resources/features/errors.feature deleted file mode 100644 index 0346df3da..000000000 --- a/tools/provider-tck/src/main/resources/features/errors.feature +++ /dev/null @@ -1,80 +0,0 @@ -Feature: Provider error handling - - # Every scenario here asserts the same three-part contract, because all three parts matter and - # providers routinely get one of them wrong: - # - # 1. the code default is returned — an application must keep working, - # 2. the correct error code is reported — an application must be able to tell what went wrong, - # 3. nothing is thrown — an unhandled exception from a flag evaluation is never acceptable. - # - # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. - - Background: - Given a stable provider - - Scenario Outline: Requesting the wrong type returns the code default - # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered - # by the @strict-numeric-typing scenarios below, because "is 0.5 an integer?" has a defensible - # wrong answer whereas "is a string a boolean?" does not. - Given a -flag with key "" and a default value "" - When the flag was evaluated with details - Then the resolved details value should be "" - And the reason should be "ERROR" - And the error-code should be "TYPE_MISMATCH" - And no exception should have been thrown - - Examples: a string flag requested as something else - | key | requested | default | - | string-flag | Boolean | false | - | string-flag | Integer | 1 | - | string-flag | Float | 0.1 | - | wrong-flag | Boolean | false | - - Examples: a boolean flag requested as something else - | key | requested | default | - | boolean-flag | String | fallback | - | boolean-flag | Integer | 1 | - | boolean-flag | Float | 0.1 | - - Examples: a numeric flag requested as a non-numeric type - | key | requested | default | - | integer-flag | Boolean | false | - | integer-flag | String | fallback | - | float-flag | Boolean | false | - | float-flag | String | fallback | - - @object - Scenario Outline: Requesting a structured flag as a scalar returns the code default - Given a -flag with key "object-flag" and a default value "" - When the flag was evaluated with details - Then the resolved details value should be "" - And the reason should be "ERROR" - And the error-code should be "TYPE_MISMATCH" - And no exception should have been thrown - - Examples: - | requested | default | - | Boolean | false | - | String | fallback | - | Integer | 1 | - | Float | 0.1 | - - @strict-numeric-typing - Scenario: A float flag is not silently narrowed to an integer - # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information - # silently, so it must be reported as a type mismatch rather than rounded. - Given a Integer-flag with key "float-flag" and a default value "1" - When the flag was evaluated with details - Then the resolved details value should be "1" - And the reason should be "ERROR" - And the error-code should be "TYPE_MISMATCH" - And no exception should have been thrown - - Scenario: An unknown flag key returns the code default - # 'missing-flag' is deliberately absent from the canonical flag set. - Given a String-flag with key "missing-flag" and a default value "fallback" - When the flag was evaluated with details - Then the resolved details value should be "fallback" - And the reason should be "ERROR" - And the error-code should be "FLAG_NOT_FOUND" - And no exception should have been thrown diff --git a/tools/provider-tck/src/main/resources/features/evaluation.feature b/tools/provider-tck/src/main/resources/features/evaluation.feature deleted file mode 100644 index e89f174a5..000000000 --- a/tools/provider-tck/src/main/resources/features/evaluation.feature +++ /dev/null @@ -1,59 +0,0 @@ -Feature: Provider flag evaluation - - # Verifies that a provider maps backend responses onto typed resolution details correctly. - # - # This does NOT test the backend's evaluation logic. Every flag in the canonical set resolves - # to its default variant with no targeting involved, so what is under test is purely the - # provider's mapping of a backend response to a value, a variant and a reason. - # - # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. - - Background: - Given a stable provider - - Scenario Outline: Resolve values with variant and reason - Given a -flag with key "" and a default value "" - When the flag was evaluated with details - Then the resolved details value should be "" - And the variant should be "" - And the reason should be "" - And the error-code should be "" - And no exception should have been thrown - - Examples: - | key | type | default | value | variant | reason | - | boolean-flag | Boolean | false | true | on | STATIC | - | string-flag | String | bye | hi | greeting | STATIC | - | integer-flag | Integer | 1 | 10 | ten | STATIC | - | float-flag | Float | 0.1 | 0.5 | half | STATIC | - - Scenario: An integer flag resolves as an integer - # Paired with the float scenario below and with the narrowing scenario in errors.feature. - # Together they pin down that the two numeric types stay distinct rather than both being - # funnelled through one numeric representation. - Given a Integer-flag with key "integer-flag" and a default value "1" - When the flag was evaluated with details - Then the resolved details value should be "10" - And the error-code should be "" - And no exception should have been thrown - - Scenario: A float flag resolves as a float - Given a Float-flag with key "float-flag" and a default value "0.1" - When the flag was evaluated with details - Then the resolved details value should be "0.5" - And the error-code should be "" - And no exception should have been thrown - - @object - Scenario: Resolve a structured value - Given a Object-flag with key "object-flag" and a default value "{}" - When the flag was evaluated with details - Then the variant should be "template" - And the reason should be "STATIC" - And the error-code should be "" - And no exception should have been thrown - And the resolved object value should contain - | key | type | value | - | showImages | Boolean | true | - | title | String | Check out these pics! | - | imagesPerPage | Integer | 100 | diff --git a/tools/provider-tck/src/main/resources/features/events.feature b/tools/provider-tck/src/main/resources/features/events.feature deleted file mode 100644 index 00e7e5ef6..000000000 --- a/tools/provider-tck/src/main/resources/features/events.feature +++ /dev/null @@ -1,42 +0,0 @@ -@events -Feature: Provider events - - # Verifies that a provider notices changes in its backend and both signals them and acts on - # them. Signalling alone is not enough: a configuration-change event that is not followed by - # a changed evaluation result is a lie, so each scenario asserts the event AND the behaviour. - # - # Outages here are simulated inside the running stack via the control API. No container is - # ever stopped or restarted — see the invariant in openapi/control-api.yaml. - - Background: - Given a stable provider - - @configuration-change - Scenario: A configuration change is signalled and applied - Given a String-flag with key "changing-flag" and a default value "unset" - And a change event handler - When the flag was evaluated with details - And the resolved value is remembered - And the flag was modified - Then the change event handler should have been executed - And the flag should be part of the event payload - When the flag was evaluated with details - Then the resolved details value should have changed - And no exception should have been thrown - - @stale - Scenario: Losing the backend makes the provider stale, regaining it makes it ready again - Given a ready event handler - And a stale event handler - When a ready event was fired - And the connection is lost - Then the stale event handler should have been executed - And the client should be in stale state - When the connection is restored - Then the ready event handler should have been executed - And the client should be in ready state - - # Deliberately NOT covered here: whether a stale provider keeps serving last-known values - # during the outage. That is caching behaviour, which depends on whether the provider holds a - # local copy of the ruleset, and it belongs behind the @caching capability once those - # scenarios are written. See the "Known gaps" section of the README. diff --git a/tools/provider-tck/src/main/resources/features/lifecycle.feature b/tools/provider-tck/src/main/resources/features/lifecycle.feature deleted file mode 100644 index 256164106..000000000 --- a/tools/provider-tck/src/main/resources/features/lifecycle.feature +++ /dev/null @@ -1,33 +0,0 @@ -@events -Feature: Provider lifecycle - - # Verifies the two terminal outcomes of provider initialisation: reaching READY against a - # healthy backend, and settling into ERROR against one that cannot be reached. - # - # The failure case matters more than it looks. A provider that blocks forever, or throws out - # of provider registration, takes the host application down with it — so the requirement is - # not merely that initialisation fails, but that it fails observably and promptly. - - Scenario: A provider reaching its backend becomes ready - Given a stable provider - And a ready event handler - Then the ready event handler should have been executed - And the client should be in ready state - - @unavailable - Scenario: A provider that cannot reach its backend reports an error - Given a unavailable provider - And a error event handler - Then the error event handler should have been executed within 10000ms - And the client should be in error state - - @unavailable - Scenario: A provider that cannot reach its backend still returns code defaults - Given a unavailable provider - And a error event handler - And a Boolean-flag with key "boolean-flag" and a default value "false" - Then the error event handler should have been executed within 10000ms - When the flag was evaluated with details - Then the resolved details value should be "false" - And the reason should be "ERROR" - And no exception should have been thrown diff --git a/tools/provider-tck/src/main/resources/flags/canonical-flags.json b/tools/provider-tck/src/main/resources/flags/canonical-flags.json deleted file mode 100644 index 343b3ae52..000000000 --- a/tools/provider-tck/src/main/resources/flags/canonical-flags.json +++ /dev/null @@ -1,82 +0,0 @@ -{ - "$comment": [ - "The canonical flag set the TCK's feature files assume. A backend under test MUST serve an", - "equivalent set under the configuration named 'default'.", - "", - "Expressed in the flagd flag-definition format because that is the only widely implemented", - "vendor-neutral format today. The format is not what matters — the keys, types, variant", - "names and resolved values are. Seed them however your backend seeds flags.", - "", - "Two things are load-bearing and easy to get wrong:", - " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", - " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", - " tests the provider's mapping of a response, not the backend's evaluation logic." - ], - "flags": { - "boolean-flag": { - "state": "ENABLED", - "variants": { - "on": true, - "off": false - }, - "defaultVariant": "on" - }, - "string-flag": { - "state": "ENABLED", - "variants": { - "greeting": "hi", - "parting": "bye" - }, - "defaultVariant": "greeting" - }, - "integer-flag": { - "state": "ENABLED", - "variants": { - "one": 1, - "ten": 10 - }, - "defaultVariant": "ten" - }, - "float-flag": { - "state": "ENABLED", - "variants": { - "tenth": 0.1, - "half": 0.5 - }, - "defaultVariant": "half" - }, - "object-flag": { - "state": "ENABLED", - "variants": { - "empty": {}, - "template": { - "showImages": true, - "title": "Check out these pics!", - "imagesPerPage": 100 - } - }, - "defaultVariant": "template" - }, - "wrong-flag": { - "$comment": "A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario.", - "state": "ENABLED", - "variants": { - "one": "uno", - "two": "dos" - }, - "defaultVariant": "one" - }, - "changing-flag": { - "$comment": [ - "The flag POST /change mutates. The TCK asserts only that its resolved value differs", - "after the change, so which of the two variants you start from does not matter." - ], - "state": "ENABLED", - "variants": { - "foo": "foo", - "bar": "bar" - }, - "defaultVariant": "foo" - } - } -} diff --git a/tools/provider-tck/src/main/resources/openapi/control-api.yaml b/tools/provider-tck/src/main/resources/openapi/control-api.yaml deleted file mode 100644 index fd9bc7000..000000000 --- a/tools/provider-tck/src/main/resources/openapi/control-api.yaml +++ /dev/null @@ -1,368 +0,0 @@ -openapi: 3.0.3 - -info: - title: OpenFeature Provider TCK — Backend Control API - version: 0.0.1 - description: | - The control API that a **backend under test** must expose so the OpenFeature - Provider TCK can drive it. - - The TCK verifies the *provider contract*: how a provider maps backend - responses to typed resolution details, lifecycle states and events. To do - that it must be able to put the backend into specific states on demand — - running, unreachable, reconfigured. This document standardises how. - - This specification is derived from the control endpoints already implemented - by [`flagd-testbed`](https://github.com/open-feature/flagd-testbed)'s - "launchpad" server, which is the reference implementation. - - ## Where this document should live - - This file currently ships inside the Java `provider-tck` artifact, but it is - not a Java artifact: it is a language-agnostic contract that every language's - TCK must implement identically, and that backend vendors implement in - whatever language their testbed is written in (Go, for flagd). - - It therefore belongs in the OpenFeature **spec** repository - (`open-feature/spec`), alongside the canonical Gherkin feature files and the - canonical flag set. Those three artifacts are a single unit — a feature file - that evaluates `boolean-flag` is meaningless without the flag definition, and - a disconnect scenario is meaningless without the endpoint that produces the - disconnect. Splitting them across repositories would let them drift. - - Each language's TCK then vendors the spec repo (git submodule or equivalent) - and packages these files into its own distribution format, so that adopting a - TCK never requires a consumer to check out a submodule of their own. - - ## Conformance language - - The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT and MAY are to be - interpreted as described in RFC 2119. - - Each operation below is tagged **REQUIRED** or **OPTIONAL**. A backend that - implements every REQUIRED operation can run the full TCK. OPTIONAL operations - have a defined fallback that the TCK applies automatically, so omitting them - costs nothing but precision. - - --- - - ## Normative requirement 1 — the no-container-restart invariant - - > **Container lifecycle operations MUST NOT be used to simulate backend - > unavailability. Backend unavailability MUST be simulated from inside the - > running stack.** - - The TCK starts the vendor's Docker Compose stack **once per test suite** and - reads the dynamically mapped host ports. Testcontainers cannot reliably - preserve mapped ports across a container stop/start in all language - bindings — a restarted container generally comes back on a *different* host - port, which silently invalidates every provider instance already pointed at - the old one. Any TCK implementation in any language hits this, so the - constraint is part of the contract rather than a Java detail. - - Therefore an implementation of `/stop`, `/restart` or any other outage - simulation MUST achieve the outage by one of: - - * killing or suspending the backend **process** inside its container - (the reference behaviour — this is what flagd-testbed does); - * a proxy in the stack refusing or blackholing connections - (e.g. a toxiproxy toxic, an envoy `direct_response`); - * an in-container firewall or socket-level block. - - An implementation MUST NOT `docker stop`, `docker kill`, `docker rm` or - recreate any container in the stack while the suite is running. The stack is - brought up before the first scenario and torn down after the last one, and - the mapped ports MUST remain stable for that entire window. - - --- - - ## Normative requirement 2 — flag state semantics across outages - - Outage simulation and flag-state seeding are orthogonal, and the TCK relies - on that separation for scenario isolation: - - * `POST /start` **MUST** (re)seed flag state to the baseline defined by the - named configuration. Any mutation previously applied by `POST /change` - MUST be discarded. This is what makes `/start` usable as a reset. - * `POST /restart` and a `POST /stop` followed by a `POST /start` **of the - same configuration** MUST leave the backend serving the same baseline - flag state it served before the outage. An outage MUST NOT be observable - as a change in flag *values* — only as a change in *availability*. - * `POST /change` mutations persist until the next `/start` or `/reset`. - - --- - - ## Normative requirement 3 — compose stack conventions - - The backend under test is delivered as a **Docker Compose stack**, not a - single image, so vendors can compose proxies, edge services or several - containers. The TCK only relies on these conventions: - - * One service — by default named `backend`, overridable by the provider - author — exposes the control API on container-internal port `8080` - (also overridable). - * The same stack exposes whatever port(s) the provider connects to. - * **All external ports are dynamically mapped.** A stack MUST NOT pin host - ports; the TCK discovers them after startup and hands them to the - provider factory. - * The stack MAY contain any number of additional services. - - --- - - ## Known gap — evaluation context passthrough - - There is currently no operation for asserting that an evaluation context sent - by the provider actually reached the backend intact. Verifying that requires - an echo mechanism (e.g. `GET /last-evaluation` returning the most recent - request the backend received). Until such an operation exists, context - passthrough is out of scope for the TCK. - - license: - name: Apache 2.0 - url: https://www.apache.org/licenses/LICENSE-2.0 - -servers: - - url: http://{host}:{port} - description: | - Resolved at runtime from the Compose stack. `host` is the Docker host and - `port` is the dynamically mapped host port for the control service's - internal port 8080. - variables: - host: - default: localhost - port: - default: "8080" - -tags: - - name: lifecycle - description: Start and stop the backend process. - - name: availability - description: Simulate outages without touching containers. - - name: flags - description: Seed and mutate flag configuration. - - name: health - description: Readiness of the control API itself. - -paths: - - /start: - post: - tags: [lifecycle] - operationId: start - summary: "[REQUIRED] Start the backend and seed flags to a named baseline" - description: | - Starts the backend process using the named configuration and seeds flag - state to that configuration's baseline. - - MUST be idempotent in the sense that calling it while the backend is - already running is not an error: the implementation restarts the process - (or otherwise ensures it is running) with the requested configuration. - - Because this operation resets flag state, the TCK uses it as its default - scenario-isolation mechanism when `/reset` is not implemented. - - The set of valid configuration names is vendor-defined. Every - implementation MUST support the name `default`, which MUST serve the - canonical flag set the TCK's feature files assume. - - Reference implementation: flagd-testbed launches the `flagd` binary with - the config file of that name from `launchpad/configs` and rewrites - `/flags/allFlags.json`. - parameters: - - name: config - in: query - required: false - description: | - Name of the configuration to start with. Defaults to `default`. - schema: - type: string - default: default - example: default - responses: - "200": - description: Backend started and flag state seeded. - "400": - description: Unknown configuration name. - content: - application/json: - schema: - $ref: "#/components/schemas/Error" - - /stop: - post: - tags: [availability] - operationId: stop - summary: "[REQUIRED] Make the backend unreachable" - description: | - Makes the backend unreachable to the provider, simulating an outage. - - **MUST NOT stop the container.** See normative requirement 1. The - reference implementation kills the flagd process while its container - keeps running. - - The backend stays unreachable until a subsequent `POST /start`. Calling - `/stop` when the backend is already stopped MUST succeed. - - The TCK uses this to drive providers into `STALE` and `ERROR` states and - to assert `PROVIDER_STALE` / `PROVIDER_ERROR` events. - responses: - "200": - description: Backend is now unreachable; container still running. - - /restart: - post: - tags: [availability] - operationId: restart - summary: "[REQUIRED] Simulate an outage of a bounded duration" - description: | - Makes the backend unreachable, waits `seconds`, then starts it again with - the configuration currently in effect. - - Flag state MUST be preserved across the outage — see normative - requirement 2. This is what distinguishes `/restart` from - `/stop` + `/start`: the former is an availability event, the latter is - also a reset. - - This operation MAY return as soon as the outage has begun rather than - blocking for the full duration; the TCK does not rely on the response - being delayed. It awaits provider events instead. - - The TCK uses this for the disconnect/reconnect scenarios: `STALE` → - `PROVIDER_STALE`, then back to `READY` → `PROVIDER_READY`. - parameters: - - name: seconds - in: query - required: false - description: | - How long the backend stays unreachable. Defaults to 5. - - Providers differ enormously in how fast they notice an outage — - a streaming provider may see it in milliseconds while a polling - provider needs up to a full poll interval. Feature files therefore - parameterise this value and provider authors tune the matching - await timeouts. - schema: - type: integer - format: int32 - minimum: 0 - default: 5 - example: 5 - responses: - "200": - description: Outage started (and, for blocking implementations, ended). - - /change: - post: - tags: [flags] - operationId: change - summary: "[REQUIRED] Mutate flag configuration so the provider observes a change" - description: | - Mutates the flag configuration such that a conforming provider observes a - configuration change and, on re-evaluation, resolves a **different value** - for the affected flag. - - The implementation MUST: - - * change the resolved value of the flag with key `changing-flag`; - * do so without restarting the backend process, so that a provider sees - a configuration-change signal rather than a reconnect; - * make the change durable until the next `/start` or `/reset`. - - The implementation SHOULD toggle between exactly two known values so that - repeated calls are meaningful and the test remains deterministic - regardless of how many times it has run against the same stack. The - reference implementation toggles `changing-flag`'s `defaultVariant` - between `foo` and `bar`. - - The TCK uses this to assert `PROVIDER_CONFIGURATION_CHANGED`, that the - changed flag key appears in the event payload, and that a subsequent - evaluation returns the new value. - responses: - "200": - description: Flag configuration mutated. - - /reset: - post: - tags: [flags] - operationId: reset - summary: "[OPTIONAL] Restore the seeded baseline without an outage" - description: | - Restores flag state to the baseline of the configuration currently in - effect, discarding any mutation applied by `/change`, **without** making - the backend unreachable at any point. - - This is the preferred scenario-isolation primitive: unlike `/start` it - causes no availability blip, so it cannot inject spurious lifecycle - events into the next scenario. - - **Scope.** This operation resets flag state only. It MUST NOT be - expected to start a backend that is currently stopped — that is what - `/start` is for. A TCK therefore uses `/reset` only when the backend is - known to be running, and `/start` otherwise. The reference client tracks - this: `/stop` and `/restart` mark the backend as possibly-unreachable, so - the scenario that follows either of them is prepared with `/start`. - - **Fallback when not implemented.** A backend that does not implement this - operation MUST respond `404` or `501`. The TCK then falls back to - `POST /start?config={defaultConfig}`, which resets flag state at the cost - of a process restart. The fallback is detected once per suite and cached. - - Implementing `/reset` is RECOMMENDED for providers whose reconnect - behaviour makes the `/start` blip hard to distinguish from a real event. - responses: - "200": - description: Flag state restored to the baseline. - "404": - description: Not implemented; the TCK falls back to `/start`. - "501": - description: Not implemented; the TCK falls back to `/start`. - - /healthz: - get: - tags: [health] - operationId: health - summary: "[OPTIONAL] Readiness of the control API" - description: | - Reports whether the control API is ready to accept commands. - - **Fallback when not implemented.** Readiness defaults to "the control - port accepts a TCP connection", which the TCK establishes with a - Testcontainers listening-port wait strategy before the first scenario. A - `404` here is therefore not a failure, and the reference implementation - does not serve this path. - - Note this reports the health of the **control API**, not of the backend. - The backend is deliberately unhealthy during outage scenarios while the - control API must stay reachable — otherwise the TCK could not end the - outage. - responses: - "200": - description: Control API ready. - content: - application/json: - schema: - $ref: "#/components/schemas/Health" - "404": - description: Not implemented; readiness falls back to a TCP port check. - "503": - description: Control API not ready yet. - -components: - schemas: - - Health: - type: object - properties: - status: - type: string - enum: [ok] - description: Present and equal to `ok` when the control API is ready. - required: [status] - - Error: - type: object - properties: - message: - type: string - description: Human-readable explanation. Never interpreted by the TCK. - required: [message]