From bcee1e0fda6809096bc0f44f26892cf04f168ca2 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 11:23:39 +0200 Subject: [PATCH 01/63] feat: add Appendix F, provider conformance (TCK) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OpenFeature's central promise is that swapping providers does not change application behaviour, and nothing verifies it. Every provider tests itself differently, against its own backend, with its own harness, so "implements the provider contract" is an unverified claim. Add the language-agnostic artifacts a shared provider conformance suite needs, plus an appendix explaining how they fit together for implementers in other languages. A new appendix rather than an extension of Appendix B, because the two answer different questions. Appendix B validates an SDK using an in-memory provider as a stand-in; this validates a provider against a real backend. Appendix F also needs more than Gherkin: the scenarios, the flag set they assume and the control API that produces outages are three co-dependent artifacts, and an OpenAPI document has no home under "Gherkin Suites". Both appendices now cross-reference each other. Assets live under specification/assets/provider-tck/ rather than alongside the existing SDK gherkin — those already contain an evaluation.feature that means something different. Marked experimental and explicitly non-normative. Nothing here is expressed as a numbered requirement yet; whether the control API contract and the capability tag vocabulary should become normative sections is a TSC decision, and is recorded as an open question rather than assumed. The first implementation is tools/provider-tck in open-feature/java-sdk-contrib, in review alongside this. Signed-off-by: Simon Schrottner --- specification/README.md | 1 + specification/appendix-b-gherkin-suites.md | 3 + .../appendix-f-provider-conformance.md | 222 +++++++++++ .../assets/provider-tck/.gitattributes | 6 + specification/assets/provider-tck/README.md | 27 ++ .../provider-tck/flags/canonical-flags.json | 82 ++++ .../provider-tck/gherkin/errors.feature | 80 ++++ .../provider-tck/gherkin/evaluation.feature | 59 +++ .../provider-tck/gherkin/events.feature | 42 ++ .../provider-tck/gherkin/lifecycle.feature | 33 ++ .../provider-tck/openapi/control-api.yaml | 368 ++++++++++++++++++ 11 files changed, 923 insertions(+) create mode 100644 specification/appendix-f-provider-conformance.md create mode 100644 specification/assets/provider-tck/.gitattributes create mode 100644 specification/assets/provider-tck/README.md create mode 100644 specification/assets/provider-tck/flags/canonical-flags.json create mode 100644 specification/assets/provider-tck/gherkin/errors.feature create mode 100644 specification/assets/provider-tck/gherkin/evaluation.feature create mode 100644 specification/assets/provider-tck/gherkin/events.feature create mode 100644 specification/assets/provider-tck/gherkin/lifecycle.feature create mode 100644 specification/assets/provider-tck/openapi/control-api.yaml diff --git a/specification/README.md b/specification/README.md index 6772899d1..ef803fda1 100644 --- a/specification/README.md +++ b/specification/README.md @@ -22,6 +22,7 @@ sidebar_position: 0 - [Appendix C: OFREP](./appendix-c/index.md) - [Appendix D: Observability](./appendix-d-observability.md) - [Appendix E: Migrations](./appendix-e-migrations.md) +- [Appendix F: Provider Conformance (TCK)](./appendix-f-provider-conformance.md) ## Conformance diff --git a/specification/appendix-b-gherkin-suites.md b/specification/appendix-b-gherkin-suites.md index 2e046dd46..70df67ea8 100644 --- a/specification/appendix-b-gherkin-suites.md +++ b/specification/appendix-b-gherkin-suites.md @@ -11,6 +11,9 @@ This section contains a set of language-agnostic end-to-end tests (defined in gh These tests can be used to validate the behavior of an OpenFeature implementation. "Features" (test suites) can be used in conjunction with an [in-memory provider](./appendix-a-included-utilities.md#in-memory-provider) and a cucumber test-runner for the language in question. +The suites below validate an **SDK**, using an in-memory provider as a stand-in for a real one. +To validate a **provider** against a real backend instead, see [Appendix F: Provider Conformance (TCK)](./appendix-f-provider-conformance.md). + ## Evaluation Feature The [evaluation feature](./assets/gherkin/evaluation.feature) contains tests for the basic functionality of the [Evaluation API](./sections/01-flag-evaluation.md). diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md new file mode 100644 index 000000000..906e4cf50 --- /dev/null +++ b/specification/appendix-f-provider-conformance.md @@ -0,0 +1,222 @@ +--- +id: appendix-f +title: "Appendix F: Provider Conformance (TCK)" +description: A language-agnostic conformance suite for validating OpenFeature providers +sidebar_position: 7 +--- + +# Appendix F: Provider Conformance (TCK) + +[![experimental](https://img.shields.io/static/v1?label=Status&message=experimental&color=orange)](https://github.com/open-feature/spec/tree/main/specification#experimental) + +> **Status: proof of concept.** The artifacts in this appendix are under active development and are +> not yet normative. The scenario set is a representative subset covering each architectural +> mechanism once, not exhaustive coverage. Breaking changes should be expected. + +OpenFeature's central promise is that swapping providers does not change application behaviour. +Nothing currently verifies that. Every provider tests itself differently, against its own backend, +with its own harness — so "implements the provider contract" is an unverified claim, and a +behavioural difference between two providers is discovered by the application that trips over it. + +This appendix defines a **provider conformance suite**: a shared set of scenarios, a shared flag set, +and a shared way to manipulate a backend under test, so that the same conformance question can be +asked of every provider in every language and get comparable answers. + +## Relationship to Appendix B + +[Appendix B](./appendix-b-gherkin-suites.md) also contains Gherkin, and the distinction matters: + +| | Appendix B | Appendix F | +|---|---|---| +| Subject under test | the **SDK** | the **provider** | +| Harness | an [in-memory provider](./appendix-a-included-utilities.md#in-memory-provider) | the provider under test, against a real backend | +| Answers | does this SDK implement the Evaluation API, hooks and events correctly? | does this provider map its backend onto the provider contract correctly? | + +They are complementary. An SDK passing Appendix B and a provider passing Appendix F are different +claims, and a language needs both suites to make both. + +## What it tests, and what it does not + +**In scope — the provider contract:** + +- mapping backend responses onto typed resolution details: value, variant, reason, error code +- keeping the integer and float types distinct rather than coercing between them +- error handling: a type mismatch and an unknown flag return the code default, report the right + error code, and never throw +- lifecycle: reaching `READY`, and settling into `ERROR` against an unreachable backend +- events: `PROVIDER_READY`, `PROVIDER_ERROR`, `PROVIDER_STALE`, `PROVIDER_CONFIGURATION_CHANGED` +- that a signalled configuration change is actually **applied** on re-evaluation, not merely + signalled + +**Out of scope:** + +- **Backend evaluation logic**, targeting and bucketing correctness. Every flag in the canonical set + resolves to its default variant with no targeting involved, so what is under test is the + provider's mapping of a response, not the backend's decision. +- **The provider↔backend wire protocol.** How a provider talks to its backend is its own business. +- **SDK behaviour.** That is Appendix B. + +## The three artifacts + +Conformance rests on three files, and they **travel together by necessity**. A feature file that +evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is +meaningless without the control endpoint that produces the disconnect. Changing one without the +others breaks the suite in every language at once. + +| Artifact | Location | What it defines | +|---|---|---| +| Gherkin scenarios | [`assets/provider-tck/gherkin/`](./assets/provider-tck/README.md) | the test cases themselves | +| Canonical flag set | [`assets/provider-tck/flags/canonical-flags.json`](./assets/provider-tck/flags/canonical-flags.json) | the flags those cases assume | +| Control API | [`assets/provider-tck/openapi/control-api.yaml`](./assets/provider-tck/openapi/control-api.yaml) | what a backend under test must expose | + +### Gherkin scenarios + +Four feature files: + +- [`evaluation.feature`](./assets/provider-tck/gherkin/evaluation.feature) — resolving each type with + the right value, variant and reason +- [`errors.feature`](./assets/provider-tck/gherkin/errors.feature) — the type-mismatch matrix and the + unknown-flag case +- [`events.feature`](./assets/provider-tck/gherkin/events.feature) — configuration change, and the + stale/ready transition across an outage +- [`lifecycle.feature`](./assets/provider-tck/gherkin/lifecycle.feature) — initialisation against a + healthy backend and against an unreachable one + +The step vocabulary is inherited from the +[flagd test harness](https://github.com/open-feature/test-harness) wherever it was already +provider-neutral, so an existing suite ports with a near-zero diff. + +### The canonical flag set + +A backend under test must serve an equivalent set under the configuration named `default`. The file +is expressed in the flagd flag-definition format because that is the only widely implemented +vendor-neutral format today — **the format is not what matters**, the keys, types, variant names and +resolved values are. Seed them however your backend seeds flags. + +Two properties are load-bearing and easy to break by accident: + +- **`missing-flag` must not exist.** Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding + it turns that scenario green for the wrong reason. +- **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests + the provider's mapping of a response, not the backend's decision. + +### The control API + +Scenarios need to change flags and simulate outages, and they need to do it identically across +vendors. The control API is a small HTTP surface the backend under test exposes for that purpose: + +| Endpoint | Required | Purpose | +|---|---|---| +| `POST /start?config=` | yes | start the backend, seeding flags to a named baseline | +| `POST /stop` | yes | make the backend unreachable | +| `POST /restart?seconds=` | yes | simulate an outage of a bounded duration | +| `POST /change` | yes | mutate flag configuration so the provider observes a change | +| `POST /reset` | no | restore the baseline without an availability blip | +| `GET /healthz` | no | readiness of the control API | + +See the [OpenAPI document](./assets/provider-tck/openapi/control-api.yaml) for the normative detail. + +Two invariants are worth stating here because they are the ones a TCK implementation gets wrong: + +- **No container is ever stopped or restarted between scenarios.** Unavailability is simulated + *inside* the running stack. Container orchestrators assign host ports dynamically and cannot + reliably preserve them across a restart, so restarting would silently invalidate every provider + already pointed at the old port. The failure looks like a flaky provider. +- **Scenario isolation comes from the control API**, not from cycling the stack. The backend is + started once per suite and reset before each scenario. + +## Capabilities: how a provider says what it cannot do + +Not every provider implements every optional part of the contract. A provider backed by a static +file has no meaningful notion of going stale; a provider without a streaming transport cannot emit +configuration-change events. Forcing those providers to fail scenarios they were never going to +satisfy makes the suite unadoptable. + +Instead, each scenario that exercises an optional capability carries a **tag**, and a provider +declares which capabilities it supports. Scenarios whose tag is not declared are reported as +**skipped, with the reason** — never as passed. + +| Tag | Meaning | +|---|---| +| `@events` | emits lifecycle events at all | +| `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | +| `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | +| `@object` | supports structured flag values | +| `@unavailable` | reports an error state instead of hanging against a dead backend | +| `@strict-numeric-typing` | does not coerce between integer and float | +| `@targeting` | reserved; no scenarios yet | +| `@caching` | reserved; no scenarios yet | + +Untagged scenarios are mandatory and always run. + +The design rule behind this: **a conformance suite that quietly goes green on scenarios it did not +run is worse than no suite at all.** A TCK implementation must report unsupported capabilities as +skipped and surface the reason, not silently pass or silently omit them. + +`@strict-numeric-typing` deserves a note, because unlike the others it is **not** an optional +feature. The specification requires a provider to report `TYPE_MISMATCH` when the requested type +cannot be satisfied, and narrowing `0.5` to `0` to satisfy an integer request loses information +silently — the worst failure mode for a feature flag, because the application sees a plausible value +and no error. It is a capability only so that a provider with this defect can adopt the suite today +and see the gap reported explicitly rather than being unable to adopt at all. Not declaring it is an +admission of a known bug. + +## Implementing the suite in a language + +A TCK implementation is the language-specific harness around these three artifacts. What it owns: + +1. **Ship the artifacts.** Package the Gherkin, the flag set and the control API document with the + library so that adopting providers need no submodule of their own. +2. **Implement the step definitions** against the language's OpenFeature SDK, using its Cucumber (or + equivalent) runner. +3. **Own the lifecycle** — start the backend stack once, register the provider under test with the + SDK, await events, tear down — so that an adopting provider writes no test infrastructure. If a + provider author finds themselves adding lifecycle code, that is a defect in the TCK. +4. **Drive the backend only through the control API.** This is the part that makes the conformance + claim portable: another language's TCK drives the same endpoints against the same stack and must + get the same answers. +5. **Gate on capabilities** and report undeclared ones as skipped with a reason. +6. **Run scenarios serially.** Backend state is global to the suite; concurrent scenarios corrupt + each other, and the symptom looks like a flaky provider rather than a broken test. + +### Providers with no backend + +An in-memory, environment-variable or file-based provider has nothing to connect to and no control +API to expose. A TCK implementation may offer an **in-process** control path for these, where flag +operations are direct manipulations of the provider's own state rather than HTTP calls. + +This is a narrow allowance, and worth being explicit about, because it is the obvious thing to abuse. +**A provider with an external backend must use the control API.** Reaching into an external backend +from inside the test process — a test-only admin client, a shared database handle, a hook inside the +provider — produces a suite that passes while proving nothing, because the path it exercised is not +the path the contract describes. + +Connection-dependent scenarios (`@stale`, `@unavailable`) have no meaning without a connection, so a +backend-less provider leaves those capabilities undeclared and they are skipped. An in-process +control path should **fail loudly** if a connection operation is reached anyway — that means a +capability was declared that the harness cannot back up, which is a test-configuration bug rather +than a provider defect. + +## Reference implementation + +The first implementation is `tools/provider-tck` in +[open-feature/java-sdk-contrib](https://github.com/open-feature/java-sdk-contrib), adopted by the +flagd provider for both its RPC and in-process resolvers. It is in review alongside this appendix. + +## Open questions + +This appendix is a proof of concept. Known gaps, all of which affect every language equally and so +belong here rather than in any one implementation: + +- **Evaluation context passthrough.** The scenarios build evaluation contexts but cannot assert the + context *reached* the backend intact. That needs an echo operation on the control API — something + like `GET /last-evaluation` returning the request the backend last received. Until then a provider + that silently drops the context passes. The `@targeting` tag is reserved for these scenarios. +- **Setting and removing individual flags.** The control API can reset to a baseline and mutate one + designated flag. Finer-grained flag manipulation would need new endpoints. +- **Caching.** Whether a stale provider keeps serving last-known values during an outage depends on + whether it holds a local copy of the ruleset. The `@caching` tag is reserved; no scenarios yet. +- **Hooks and flag metadata.** Not covered. +- **Normative status.** Nothing in this appendix is currently expressed as a numbered requirement. + Whether the control API contract and the capability vocabulary should become normative sections is + a decision for the TSC. diff --git a/specification/assets/provider-tck/.gitattributes b/specification/assets/provider-tck/.gitattributes new file mode 100644 index 000000000..c617195d2 --- /dev/null +++ b/specification/assets/provider-tck/.gitattributes @@ -0,0 +1,6 @@ +# These artifacts are consumed byte for byte by every language's provider TCK, and several +# are copied verbatim into published build artifacts. Normalise to LF so a checkout on +# Windows does not produce a different packaged file than one on Linux. +*.feature text eol=lf +*.json text eol=lf +*.yaml text eol=lf diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md new file mode 100644 index 000000000..292946c8a --- /dev/null +++ b/specification/assets/provider-tck/README.md @@ -0,0 +1,27 @@ +# Provider Conformance Assets + +Test assets for the provider conformance suite described in [Appendix F](../../appendix-f-provider-conformance.md). + +These validate a **provider** against a real backend. For assets that validate an **SDK**, see [`../gherkin/`](../gherkin/README.md) and [Appendix B](../../appendix-b-gherkin-suites.md). + +## Contents + +| Path | What it is | +| --- | --- | +| [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value, variant and reason | +| [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix and the unknown-flag case | +| [`gherkin/events.feature`](./gherkin/events.feature) | configuration change, and the stale/ready transition across an outage | +| [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one | +| [`flags/canonical-flags.json`](./flags/canonical-flags.json) | the flag set every scenario assumes | +| [`openapi/control-api.yaml`](./openapi/control-api.yaml) | the HTTP surface a backend under test must expose | + +## These three travel together + +A feature file that evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is meaningless without the control endpoint that produces the disconnect. Changing one without the others breaks the suite in every language at once. + +## Two properties that are load-bearing + +- **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. +- **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. + +The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json new file mode 100644 index 000000000..343b3ae52 --- /dev/null +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -0,0 +1,82 @@ +{ + "$comment": [ + "The canonical flag set the TCK's feature files assume. A backend under test MUST serve an", + "equivalent set under the configuration named 'default'.", + "", + "Expressed in the flagd flag-definition format because that is the only widely implemented", + "vendor-neutral format today. The format is not what matters — the keys, types, variant", + "names and resolved values are. Seed them however your backend seeds flags.", + "", + "Two things are load-bearing and easy to get wrong:", + " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", + " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", + " tests the provider's mapping of a response, not the backend's evaluation logic." + ], + "flags": { + "boolean-flag": { + "state": "ENABLED", + "variants": { + "on": true, + "off": false + }, + "defaultVariant": "on" + }, + "string-flag": { + "state": "ENABLED", + "variants": { + "greeting": "hi", + "parting": "bye" + }, + "defaultVariant": "greeting" + }, + "integer-flag": { + "state": "ENABLED", + "variants": { + "one": 1, + "ten": 10 + }, + "defaultVariant": "ten" + }, + "float-flag": { + "state": "ENABLED", + "variants": { + "tenth": 0.1, + "half": 0.5 + }, + "defaultVariant": "half" + }, + "object-flag": { + "state": "ENABLED", + "variants": { + "empty": {}, + "template": { + "showImages": true, + "title": "Check out these pics!", + "imagesPerPage": 100 + } + }, + "defaultVariant": "template" + }, + "wrong-flag": { + "$comment": "A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario.", + "state": "ENABLED", + "variants": { + "one": "uno", + "two": "dos" + }, + "defaultVariant": "one" + }, + "changing-flag": { + "$comment": [ + "The flag POST /change mutates. The TCK asserts only that its resolved value differs", + "after the change, so which of the two variants you start from does not matter." + ], + "state": "ENABLED", + "variants": { + "foo": "foo", + "bar": "bar" + }, + "defaultVariant": "foo" + } + } +} diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature new file mode 100644 index 000000000..0346df3da --- /dev/null +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -0,0 +1,80 @@ +Feature: Provider error handling + + # Every scenario here asserts the same three-part contract, because all three parts matter and + # providers routinely get one of them wrong: + # + # 1. the code default is returned — an application must keep working, + # 2. the correct error code is reported — an application must be able to tell what went wrong, + # 3. nothing is thrown — an unhandled exception from a flag evaluation is never acceptable. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: Requesting the wrong type returns the code default + # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered + # by the @strict-numeric-typing scenarios below, because "is 0.5 an integer?" has a defensible + # wrong answer whereas "is a string a boolean?" does not. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: a string flag requested as something else + | key | requested | default | + | string-flag | Boolean | false | + | string-flag | Integer | 1 | + | string-flag | Float | 0.1 | + | wrong-flag | Boolean | false | + + Examples: a boolean flag requested as something else + | key | requested | default | + | boolean-flag | String | fallback | + | boolean-flag | Integer | 1 | + | boolean-flag | Float | 0.1 | + + Examples: a numeric flag requested as a non-numeric type + | key | requested | default | + | integer-flag | Boolean | false | + | integer-flag | String | fallback | + | float-flag | Boolean | false | + | float-flag | String | fallback | + + @object + Scenario Outline: Requesting a structured flag as a scalar returns the code default + Given a -flag with key "object-flag" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: + | requested | default | + | Boolean | false | + | String | fallback | + | Integer | 1 | + | Float | 0.1 | + + @strict-numeric-typing + Scenario: A float flag is not silently narrowed to an integer + # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information + # silently, so it must be reported as a type mismatch rather than rounded. + Given a Integer-flag with key "float-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "1" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Scenario: An unknown flag key returns the code default + # 'missing-flag' is deliberately absent from the canonical flag set. + Given a String-flag with key "missing-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the reason should be "ERROR" + And the error-code should be "FLAG_NOT_FOUND" + And no exception should have been thrown diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature new file mode 100644 index 000000000..e89f174a5 --- /dev/null +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -0,0 +1,59 @@ +Feature: Provider flag evaluation + + # Verifies that a provider maps backend responses onto typed resolution details correctly. + # + # This does NOT test the backend's evaluation logic. Every flag in the canonical set resolves + # to its default variant with no targeting involved, so what is under test is purely the + # provider's mapping of a backend response to a value, a variant and a reason. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: Resolve values with variant and reason + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "" + And the error-code should be "" + And no exception should have been thrown + + Examples: + | key | type | default | value | variant | reason | + | boolean-flag | Boolean | false | true | on | STATIC | + | string-flag | String | bye | hi | greeting | STATIC | + | integer-flag | Integer | 1 | 10 | ten | STATIC | + | float-flag | Float | 0.1 | 0.5 | half | STATIC | + + Scenario: An integer flag resolves as an integer + # Paired with the float scenario below and with the narrowing scenario in errors.feature. + # Together they pin down that the two numeric types stay distinct rather than both being + # funnelled through one numeric representation. + Given a Integer-flag with key "integer-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "10" + And the error-code should be "" + And no exception should have been thrown + + Scenario: A float flag resolves as a float + Given a Float-flag with key "float-flag" and a default value "0.1" + When the flag was evaluated with details + Then the resolved details value should be "0.5" + And the error-code should be "" + And no exception should have been thrown + + @object + Scenario: Resolve a structured value + Given a Object-flag with key "object-flag" and a default value "{}" + When the flag was evaluated with details + Then the variant should be "template" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + And the resolved object value should contain + | key | type | value | + | showImages | Boolean | true | + | title | String | Check out these pics! | + | imagesPerPage | Integer | 100 | diff --git a/specification/assets/provider-tck/gherkin/events.feature b/specification/assets/provider-tck/gherkin/events.feature new file mode 100644 index 000000000..00e7e5ef6 --- /dev/null +++ b/specification/assets/provider-tck/gherkin/events.feature @@ -0,0 +1,42 @@ +@events +Feature: Provider events + + # Verifies that a provider notices changes in its backend and both signals them and acts on + # them. Signalling alone is not enough: a configuration-change event that is not followed by + # a changed evaluation result is a lie, so each scenario asserts the event AND the behaviour. + # + # Outages here are simulated inside the running stack via the control API. No container is + # ever stopped or restarted — see the invariant in openapi/control-api.yaml. + + Background: + Given a stable provider + + @configuration-change + Scenario: A configuration change is signalled and applied + Given a String-flag with key "changing-flag" and a default value "unset" + And a change event handler + When the flag was evaluated with details + And the resolved value is remembered + And the flag was modified + Then the change event handler should have been executed + And the flag should be part of the event payload + When the flag was evaluated with details + Then the resolved details value should have changed + And no exception should have been thrown + + @stale + Scenario: Losing the backend makes the provider stale, regaining it makes it ready again + Given a ready event handler + And a stale event handler + When a ready event was fired + And the connection is lost + Then the stale event handler should have been executed + And the client should be in stale state + When the connection is restored + Then the ready event handler should have been executed + And the client should be in ready state + + # Deliberately NOT covered here: whether a stale provider keeps serving last-known values + # during the outage. That is caching behaviour, which depends on whether the provider holds a + # local copy of the ruleset, and it belongs behind the @caching capability once those + # scenarios are written. See the "Known gaps" section of the README. diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature new file mode 100644 index 000000000..256164106 --- /dev/null +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -0,0 +1,33 @@ +@events +Feature: Provider lifecycle + + # Verifies the two terminal outcomes of provider initialisation: reaching READY against a + # healthy backend, and settling into ERROR against one that cannot be reached. + # + # The failure case matters more than it looks. A provider that blocks forever, or throws out + # of provider registration, takes the host application down with it — so the requirement is + # not merely that initialisation fails, but that it fails observably and promptly. + + Scenario: A provider reaching its backend becomes ready + Given a stable provider + And a ready event handler + Then the ready event handler should have been executed + And the client should be in ready state + + @unavailable + Scenario: A provider that cannot reach its backend reports an error + Given a unavailable provider + And a error event handler + Then the error event handler should have been executed within 10000ms + And the client should be in error state + + @unavailable + Scenario: A provider that cannot reach its backend still returns code defaults + Given a unavailable provider + And a error event handler + And a Boolean-flag with key "boolean-flag" and a default value "false" + Then the error event handler should have been executed within 10000ms + When the flag was evaluated with details + Then the resolved details value should be "false" + And the reason should be "ERROR" + And no exception should have been thrown diff --git a/specification/assets/provider-tck/openapi/control-api.yaml b/specification/assets/provider-tck/openapi/control-api.yaml new file mode 100644 index 000000000..fd9bc7000 --- /dev/null +++ b/specification/assets/provider-tck/openapi/control-api.yaml @@ -0,0 +1,368 @@ +openapi: 3.0.3 + +info: + title: OpenFeature Provider TCK — Backend Control API + version: 0.0.1 + description: | + The control API that a **backend under test** must expose so the OpenFeature + Provider TCK can drive it. + + The TCK verifies the *provider contract*: how a provider maps backend + responses to typed resolution details, lifecycle states and events. To do + that it must be able to put the backend into specific states on demand — + running, unreachable, reconfigured. This document standardises how. + + This specification is derived from the control endpoints already implemented + by [`flagd-testbed`](https://github.com/open-feature/flagd-testbed)'s + "launchpad" server, which is the reference implementation. + + ## Where this document should live + + This file currently ships inside the Java `provider-tck` artifact, but it is + not a Java artifact: it is a language-agnostic contract that every language's + TCK must implement identically, and that backend vendors implement in + whatever language their testbed is written in (Go, for flagd). + + It therefore belongs in the OpenFeature **spec** repository + (`open-feature/spec`), alongside the canonical Gherkin feature files and the + canonical flag set. Those three artifacts are a single unit — a feature file + that evaluates `boolean-flag` is meaningless without the flag definition, and + a disconnect scenario is meaningless without the endpoint that produces the + disconnect. Splitting them across repositories would let them drift. + + Each language's TCK then vendors the spec repo (git submodule or equivalent) + and packages these files into its own distribution format, so that adopting a + TCK never requires a consumer to check out a submodule of their own. + + ## Conformance language + + The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT and MAY are to be + interpreted as described in RFC 2119. + + Each operation below is tagged **REQUIRED** or **OPTIONAL**. A backend that + implements every REQUIRED operation can run the full TCK. OPTIONAL operations + have a defined fallback that the TCK applies automatically, so omitting them + costs nothing but precision. + + --- + + ## Normative requirement 1 — the no-container-restart invariant + + > **Container lifecycle operations MUST NOT be used to simulate backend + > unavailability. Backend unavailability MUST be simulated from inside the + > running stack.** + + The TCK starts the vendor's Docker Compose stack **once per test suite** and + reads the dynamically mapped host ports. Testcontainers cannot reliably + preserve mapped ports across a container stop/start in all language + bindings — a restarted container generally comes back on a *different* host + port, which silently invalidates every provider instance already pointed at + the old one. Any TCK implementation in any language hits this, so the + constraint is part of the contract rather than a Java detail. + + Therefore an implementation of `/stop`, `/restart` or any other outage + simulation MUST achieve the outage by one of: + + * killing or suspending the backend **process** inside its container + (the reference behaviour — this is what flagd-testbed does); + * a proxy in the stack refusing or blackholing connections + (e.g. a toxiproxy toxic, an envoy `direct_response`); + * an in-container firewall or socket-level block. + + An implementation MUST NOT `docker stop`, `docker kill`, `docker rm` or + recreate any container in the stack while the suite is running. The stack is + brought up before the first scenario and torn down after the last one, and + the mapped ports MUST remain stable for that entire window. + + --- + + ## Normative requirement 2 — flag state semantics across outages + + Outage simulation and flag-state seeding are orthogonal, and the TCK relies + on that separation for scenario isolation: + + * `POST /start` **MUST** (re)seed flag state to the baseline defined by the + named configuration. Any mutation previously applied by `POST /change` + MUST be discarded. This is what makes `/start` usable as a reset. + * `POST /restart` and a `POST /stop` followed by a `POST /start` **of the + same configuration** MUST leave the backend serving the same baseline + flag state it served before the outage. An outage MUST NOT be observable + as a change in flag *values* — only as a change in *availability*. + * `POST /change` mutations persist until the next `/start` or `/reset`. + + --- + + ## Normative requirement 3 — compose stack conventions + + The backend under test is delivered as a **Docker Compose stack**, not a + single image, so vendors can compose proxies, edge services or several + containers. The TCK only relies on these conventions: + + * One service — by default named `backend`, overridable by the provider + author — exposes the control API on container-internal port `8080` + (also overridable). + * The same stack exposes whatever port(s) the provider connects to. + * **All external ports are dynamically mapped.** A stack MUST NOT pin host + ports; the TCK discovers them after startup and hands them to the + provider factory. + * The stack MAY contain any number of additional services. + + --- + + ## Known gap — evaluation context passthrough + + There is currently no operation for asserting that an evaluation context sent + by the provider actually reached the backend intact. Verifying that requires + an echo mechanism (e.g. `GET /last-evaluation` returning the most recent + request the backend received). Until such an operation exists, context + passthrough is out of scope for the TCK. + + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0 + +servers: + - url: http://{host}:{port} + description: | + Resolved at runtime from the Compose stack. `host` is the Docker host and + `port` is the dynamically mapped host port for the control service's + internal port 8080. + variables: + host: + default: localhost + port: + default: "8080" + +tags: + - name: lifecycle + description: Start and stop the backend process. + - name: availability + description: Simulate outages without touching containers. + - name: flags + description: Seed and mutate flag configuration. + - name: health + description: Readiness of the control API itself. + +paths: + + /start: + post: + tags: [lifecycle] + operationId: start + summary: "[REQUIRED] Start the backend and seed flags to a named baseline" + description: | + Starts the backend process using the named configuration and seeds flag + state to that configuration's baseline. + + MUST be idempotent in the sense that calling it while the backend is + already running is not an error: the implementation restarts the process + (or otherwise ensures it is running) with the requested configuration. + + Because this operation resets flag state, the TCK uses it as its default + scenario-isolation mechanism when `/reset` is not implemented. + + The set of valid configuration names is vendor-defined. Every + implementation MUST support the name `default`, which MUST serve the + canonical flag set the TCK's feature files assume. + + Reference implementation: flagd-testbed launches the `flagd` binary with + the config file of that name from `launchpad/configs` and rewrites + `/flags/allFlags.json`. + parameters: + - name: config + in: query + required: false + description: | + Name of the configuration to start with. Defaults to `default`. + schema: + type: string + default: default + example: default + responses: + "200": + description: Backend started and flag state seeded. + "400": + description: Unknown configuration name. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + + /stop: + post: + tags: [availability] + operationId: stop + summary: "[REQUIRED] Make the backend unreachable" + description: | + Makes the backend unreachable to the provider, simulating an outage. + + **MUST NOT stop the container.** See normative requirement 1. The + reference implementation kills the flagd process while its container + keeps running. + + The backend stays unreachable until a subsequent `POST /start`. Calling + `/stop` when the backend is already stopped MUST succeed. + + The TCK uses this to drive providers into `STALE` and `ERROR` states and + to assert `PROVIDER_STALE` / `PROVIDER_ERROR` events. + responses: + "200": + description: Backend is now unreachable; container still running. + + /restart: + post: + tags: [availability] + operationId: restart + summary: "[REQUIRED] Simulate an outage of a bounded duration" + description: | + Makes the backend unreachable, waits `seconds`, then starts it again with + the configuration currently in effect. + + Flag state MUST be preserved across the outage — see normative + requirement 2. This is what distinguishes `/restart` from + `/stop` + `/start`: the former is an availability event, the latter is + also a reset. + + This operation MAY return as soon as the outage has begun rather than + blocking for the full duration; the TCK does not rely on the response + being delayed. It awaits provider events instead. + + The TCK uses this for the disconnect/reconnect scenarios: `STALE` → + `PROVIDER_STALE`, then back to `READY` → `PROVIDER_READY`. + parameters: + - name: seconds + in: query + required: false + description: | + How long the backend stays unreachable. Defaults to 5. + + Providers differ enormously in how fast they notice an outage — + a streaming provider may see it in milliseconds while a polling + provider needs up to a full poll interval. Feature files therefore + parameterise this value and provider authors tune the matching + await timeouts. + schema: + type: integer + format: int32 + minimum: 0 + default: 5 + example: 5 + responses: + "200": + description: Outage started (and, for blocking implementations, ended). + + /change: + post: + tags: [flags] + operationId: change + summary: "[REQUIRED] Mutate flag configuration so the provider observes a change" + description: | + Mutates the flag configuration such that a conforming provider observes a + configuration change and, on re-evaluation, resolves a **different value** + for the affected flag. + + The implementation MUST: + + * change the resolved value of the flag with key `changing-flag`; + * do so without restarting the backend process, so that a provider sees + a configuration-change signal rather than a reconnect; + * make the change durable until the next `/start` or `/reset`. + + The implementation SHOULD toggle between exactly two known values so that + repeated calls are meaningful and the test remains deterministic + regardless of how many times it has run against the same stack. The + reference implementation toggles `changing-flag`'s `defaultVariant` + between `foo` and `bar`. + + The TCK uses this to assert `PROVIDER_CONFIGURATION_CHANGED`, that the + changed flag key appears in the event payload, and that a subsequent + evaluation returns the new value. + responses: + "200": + description: Flag configuration mutated. + + /reset: + post: + tags: [flags] + operationId: reset + summary: "[OPTIONAL] Restore the seeded baseline without an outage" + description: | + Restores flag state to the baseline of the configuration currently in + effect, discarding any mutation applied by `/change`, **without** making + the backend unreachable at any point. + + This is the preferred scenario-isolation primitive: unlike `/start` it + causes no availability blip, so it cannot inject spurious lifecycle + events into the next scenario. + + **Scope.** This operation resets flag state only. It MUST NOT be + expected to start a backend that is currently stopped — that is what + `/start` is for. A TCK therefore uses `/reset` only when the backend is + known to be running, and `/start` otherwise. The reference client tracks + this: `/stop` and `/restart` mark the backend as possibly-unreachable, so + the scenario that follows either of them is prepared with `/start`. + + **Fallback when not implemented.** A backend that does not implement this + operation MUST respond `404` or `501`. The TCK then falls back to + `POST /start?config={defaultConfig}`, which resets flag state at the cost + of a process restart. The fallback is detected once per suite and cached. + + Implementing `/reset` is RECOMMENDED for providers whose reconnect + behaviour makes the `/start` blip hard to distinguish from a real event. + responses: + "200": + description: Flag state restored to the baseline. + "404": + description: Not implemented; the TCK falls back to `/start`. + "501": + description: Not implemented; the TCK falls back to `/start`. + + /healthz: + get: + tags: [health] + operationId: health + summary: "[OPTIONAL] Readiness of the control API" + description: | + Reports whether the control API is ready to accept commands. + + **Fallback when not implemented.** Readiness defaults to "the control + port accepts a TCP connection", which the TCK establishes with a + Testcontainers listening-port wait strategy before the first scenario. A + `404` here is therefore not a failure, and the reference implementation + does not serve this path. + + Note this reports the health of the **control API**, not of the backend. + The backend is deliberately unhealthy during outage scenarios while the + control API must stay reachable — otherwise the TCK could not end the + outage. + responses: + "200": + description: Control API ready. + content: + application/json: + schema: + $ref: "#/components/schemas/Health" + "404": + description: Not implemented; readiness falls back to a TCP port check. + "503": + description: Control API not ready yet. + +components: + schemas: + + Health: + type: object + properties: + status: + type: string + enum: [ok] + description: Present and equal to `ok` when the control API is ready. + required: [status] + + Error: + type: object + properties: + message: + type: string + description: Human-readable explanation. Never interpreted by the TCK. + required: [message] From dbb790ef44a5ca94e62928643515cb8ffd309da5 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:30:08 +0200 Subject: [PATCH 02/63] feat: separate @lifecycle from @events in the provider conformance suite lifecycle.feature was gated by @events, which is wrong in both directions. Too strict: a stateless HTTP provider such as OFREP emits no events of its own and so cannot declare @events, yet the readiness scenario is not really about events at all. Too lax: every SDK synthesises PROVIDER_READY for a provider with no initialisation step -- the Go SDK says so in a comment, "a provider without state handling capability can be assumed to be ready immediately" -- so a provider that declares @events passes the readiness scenario without demonstrating anything. A no-op provider passes it identically. The two are different claims. @events is "emits lifecycle events at all"; @lifecycle is "performs an initialisation that reaches its backend, with an observable outcome". A provider may do either without the other. Found by adopting the suite in the OFREP providers for Go and Java, where two independent adoptions withheld @events for the same reason and neither could express what they actually lacked. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 10 ++++++++++ .../assets/provider-tck/gherkin/lifecycle.feature | 9 ++++++++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 906e4cf50..941d3f40f 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -139,6 +139,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | Tag | Meaning | |---|---| | `@events` | emits lifecycle events at all | +| `@lifecycle` | performs an initialisation that reaches its backend, with an observable outcome | | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `@object` | supports structured flag values | @@ -149,6 +150,15 @@ declares which capabilities it supports. Scenarios whose tag is not declared are Untagged scenarios are mandatory and always run. +`@lifecycle` and `@events` are deliberately separate, and conflating them is the mistake this +vocabulary exists to prevent. Every SDK synthesises `PROVIDER_READY` for a provider that has no +initialisation step -- the Go SDK's comment says so outright, *"a provider without state handling +capability can be assumed to be ready immediately"* -- so a provider with no lifecycle passes the +readiness scenario without demonstrating anything, exactly as a no-op provider would. A stateless +HTTP provider such as OFREP is the common case: it observably emits nothing of its own and cannot +fail initialisation, yet its client still reports `READY`. Such a provider declares neither tag, and +the lifecycle scenarios are reported as skipped rather than passing vacuously. + The design rule behind this: **a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all.** A TCK implementation must report unsupported capabilities as skipped and surface the reason, not silently pass or silently omit them. diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature index 256164106..338b2c052 100644 --- a/specification/assets/provider-tck/gherkin/lifecycle.feature +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -1,9 +1,16 @@ -@events +@lifecycle Feature: Provider lifecycle # Verifies the two terminal outcomes of provider initialisation: reaching READY against a # healthy backend, and settling into ERROR against one that cannot be reached. # + # Gated by @lifecycle rather than @events, and the distinction is load-bearing. Every SDK + # synthesises PROVIDER_READY for a provider that has no initialisation step, so a provider + # without a lifecycle passes the readiness scenario below without demonstrating anything -- + # a NoOpProvider passes it identically. @lifecycle asserts that the provider actually reaches + # its backend during initialisation and that the outcome is observable; a provider that merely + # emits events does not necessarily do that. + # # The failure case matters more than it looks. A provider that blocks forever, or throws out # of provider registration, takes the host application down with it — so the requirement is # not merely that initialisation fails, but that it fails observably and promptly. From 4c342c43d3db889b4873f196640d715bf2490f27 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 15:17:27 +0200 Subject: [PATCH 03/63] fix: require POST /start to return only once the flag state is being served Found by adopting the suite in the Python OFREP provider, which is the first stateless adopter. The launchpad returns from /start as soon as flagd answers its readiness probe, roughly 40ms before its file sources reach the flag store. An evaluation in that window gets FLAG_NOT_FOUND for a flag the configuration plainly defines. Every previous adopter is a provider that blocks during initialisation -- streaming, or syncing a ruleset -- which absorbs the window silently. A stateless provider evaluates over HTTP with no initialisation at all, so it has nothing to hide the warm-up behind: the first run failed 22 of 24 scenarios and read as a catastrophically broken provider rather than as a racing testbed. The contract said /start MUST reseed flag state, but never said it returns only once that state is being served. Those should be the same instant. Leaving it implicit means every stateless adopter reimplements the same defensive probe, and the ones that do not get a suite that looks broken for reasons that have nothing to do with the provider under test. Part of https://github.com/open-feature/spec/issues/417 Signed-off-by: Simon Schrottner --- .../provider-tck/openapi/control-api.yaml | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/specification/assets/provider-tck/openapi/control-api.yaml b/specification/assets/provider-tck/openapi/control-api.yaml index fd9bc7000..d21191067 100644 --- a/specification/assets/provider-tck/openapi/control-api.yaml +++ b/specification/assets/provider-tck/openapi/control-api.yaml @@ -158,6 +158,26 @@ paths: already running is not an error: the implementation restarts the process (or otherwise ensures it is running) with the requested configuration. + **MUST NOT return until the seeded flag state is actually being served.** + A 200 is a promise that the very next evaluation will resolve against the + new baseline. Returning as soon as the process reports healthy is not + enough: a backend can accept connections and answer a readiness probe + while its flag store is still empty, and an evaluation in that window + gets `FLAG_NOT_FOUND` for a flag the configuration plainly defines. + + This is easy to get wrong and easy to miss. A provider that blocks during + initialisation -- streaming, or syncing a ruleset -- absorbs the window + and never sees it. A **stateless** provider, which evaluates over HTTP + with no initialisation at all, has nothing to hide it behind and fails + essentially every scenario, which reads as a catastrophically broken + provider rather than as a racing testbed. The reference implementation + exhibits this: its `/start` returns roughly 40ms before flagd's file + sources reach the flag store. + + A TCK MAY defensively probe after `/start`, but it should not have to, + and requiring every stateless adopter to reimplement that probe is worse + than stating the requirement here. + Because this operation resets flag state, the TCK uses it as its default scenario-isolation mechanism when `/reset` is not implemented. From e9bf1d7d4605e82ad0c0bdac734ea9e7e3020a22 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Thu, 10 Sep 2026 18:35:27 +0200 Subject: [PATCH 04/63] Apply suggestion from @toddbaert Co-authored-by: Todd Baert Signed-off-by: Simon Schrottner --- specification/assets/provider-tck/gherkin/lifecycle.feature | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature index 338b2c052..3e168565a 100644 --- a/specification/assets/provider-tck/gherkin/lifecycle.feature +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -15,7 +15,7 @@ Feature: Provider lifecycle # of provider registration, takes the host application down with it — so the requirement is # not merely that initialisation fails, but that it fails observably and promptly. - Scenario: A provider reaching its backend becomes ready + Scenario: A provider that successfully initializes becomes ready Given a stable provider And a ready event handler Then the ready event handler should have been executed From 685008674619571f1e0ea54d0e25e2e1525be962 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Thu, 10 Sep 2026 21:08:12 +0200 Subject: [PATCH 05/63] fix: a reserved capability must not be declarable The table listed @targeting and @caching as "reserved; no scenarios yet" and left it there, which reads as documentation rather than as a rule. It is a rule: a capability no scenario carries cannot be verified, cannot produce a skip, and tells a reader of a conformance report only that something was claimed and nothing examined. Leaving it implicit had a real consequence. A conformance report from one implementation asserts both tags as declared, not because anyone decided to claim them but because that adopter declares "every capability except X" and collects every reserved tag on the way past. So the mechanism is called out alongside the rule. An implementation offering a declare-everything convenience has to exclude reserved tags from it, and has to tell an adopter who names one directly rather than passing it quietly into a published report. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 941d3f40f..b35e9ae28 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -145,11 +145,23 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@object` | supports structured flag values | | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@strict-numeric-typing` | does not coerce between integer and float | -| `@targeting` | reserved; no scenarios yet | -| `@caching` | reserved; no scenarios yet | +| `@targeting` | reserved; **not declarable** -- no scenarios yet | +| `@caching` | reserved; **not declarable** -- no scenarios yet | Untagged scenarios are mandatory and always run. +A reserved tag is documented so the vocabulary has a place for the capability when scenarios exist, +but it **must not be declared** and must not appear in a conformance report's declaration. No +scenario carries it, so declaring it cannot be verified, cannot produce a skip, and tells a reader +only that something was claimed and nothing examined -- the vacuous conformance claim this whole +vocabulary exists to prevent. + +This is easy to reintroduce by accident rather than by intent. An adopter who declares "every +capability except X" picks up every reserved tag on the way past, which is exactly how one +implementation came to report `@targeting` and `@caching` as declared. An implementation offering a +"declare everything" convenience should exclude reserved tags from it, and should tell an adopter who +names one directly rather than passing it silently into a report. + `@lifecycle` and `@events` are deliberately separate, and conflating them is the mistake this vocabulary exists to prevent. Every SDK synthesises `PROVIDER_READY` for a provider that has no initialisation step -- the Go SDK's comment says so outright, *"a provider without state handling From cb22edbfd83d2c99cece633d2cc50f7544cffafe Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 09:12:49 +0200 Subject: [PATCH 06/63] feat: rename @strict-numeric-typing to @numeric-coercion, and fix the rule The tag was named for a stricter rule than the specification actually wants, and it was about to collide with a second vocabulary for the same property. flagd is implementing an accepted numeric coercion ADR (open-feature/flagd#1996) whose rule is that coercion is permitted when lossless and must fail with TYPE_MISMATCH only when information would be lost: 10.0 requested as an integer succeeds, 0.5 does not. This appendix said "does not coerce between integer and float", which forbids the case the ADR requires to work. The existing scenario survives the difference only by luck, because it asks about 0.5, which does have a fractional part. That work also introduces @numeric-coercion and @int32-bounded scenarios into flagd's own testbed. Two capability vocabularies describing one property, written in the same week, is the drift a shared vocabulary exists to prevent, so the tag takes flagd's name rather than leaving the reference implementation and the specification disagreeing about what a rule is called. Renaming now costs four draft implementations an enum change; renaming later costs a published one. Two gaps are recorded rather than closed. The lossless half of the contract has no scenario, because the canonical flag set contains no integral float to ask it of and adding one changes the flag set for every language at once -- so a provider that wrongly rejects 10.0 as an integer still passes. And accessor width is not modelled at all: the ADR distinguishes a 64-bit integer accessor from a 32-bit one, which flagd tags @int32-bounded, and that is a real source of cross-language disagreement this appendix is currently silent about. This also settles an open question on open-feature/spec#417. @strict-numeric-typing was the one capability tracing to no numbered requirement, existing only because flagd had a defect. It now has a definition to trace to, and can stop being a known-deviation marker wearing a capability's clothes. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 35 ++++++++++++++----- .../provider-tck/gherkin/errors.feature | 10 ++++-- 2 files changed, 35 insertions(+), 10 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index b35e9ae28..cb69e3ae3 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -144,7 +144,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `@object` | supports structured flag values | | `@unavailable` | reports an error state instead of hanging against a dead backend | -| `@strict-numeric-typing` | does not coerce between integer and float | +| `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `@targeting` | reserved; **not declarable** -- no scenarios yet | | `@caching` | reserved; **not declarable** -- no scenarios yet | @@ -175,13 +175,32 @@ The design rule behind this: **a conformance suite that quietly goes green on sc run is worse than no suite at all.** A TCK implementation must report unsupported capabilities as skipped and surface the reason, not silently pass or silently omit them. -`@strict-numeric-typing` deserves a note, because unlike the others it is **not** an optional -feature. The specification requires a provider to report `TYPE_MISMATCH` when the requested type -cannot be satisfied, and narrowing `0.5` to `0` to satisfy an integer request loses information -silently — the worst failure mode for a feature flag, because the application sees a plausible value -and no error. It is a capability only so that a provider with this defect can adopt the suite today -and see the gap reported explicitly rather than being unable to adopt at all. Not declaring it is an -admission of a known bug. +`@numeric-coercion` deserves a note, because unlike the others it is **not** an optional feature. +The specification requires a provider to report `TYPE_MISMATCH` when the requested type cannot be +satisfied, and narrowing `0.5` to `0` to satisfy an integer request loses information silently — the +worst failure mode for a feature flag, because the application sees a plausible value and no error. +It is a capability only so that a provider with this defect can adopt the suite today and see the +gap reported explicitly rather than being unable to adopt at all. Not declaring it is an admission +of a known bug. + +The rule is **lossless coercion is permitted; lossy coercion must fail**, and not the stricter "never +coerce" this tag was originally named for. An integral float such as `10.0` requested as an integer +must succeed; `0.5` must not. The distinction is flagd's +[numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), +and the tag was renamed from `@strict-numeric-typing` to match it rather than leave two vocabularies +describing one property — flagd's own testbed is gaining `@numeric-coercion` scenarios, and a +capability vocabulary that disagrees with the reference implementation about the name of a rule is +worse than having no name for it. + +Two gaps follow from adopting that rule, and both are open rather than fixed here: + +- **The lossless case has no scenario.** Only the lossy half is tested, so a provider that wrongly + rejects `10.0` as an integer passes. Closing it needs an integral float in the canonical flag set, + which changes the flag set for every language at once. +- **Accessor width is not modelled.** The ADR distinguishes the width of a language's integer + accessor — Go's `ResolveIntValue` is `int64` and so is the canonical `Long`, whereas a 32-bit + accessor needs its own scenarios, which flagd's testbed tags `@int32-bounded`. This appendix has + nothing equivalent, and it is a real source of cross-language disagreement. ## Implementing the suite in a language diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature index 0346df3da..0efbe5726 100644 --- a/specification/assets/provider-tck/gherkin/errors.feature +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -14,7 +14,7 @@ Feature: Provider error handling Scenario Outline: Requesting the wrong type returns the code default # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered - # by the @strict-numeric-typing scenarios below, because "is 0.5 an integer?" has a defensible + # by the @numeric-coercion scenarios below, because "is 0.5 an integer?" has a defensible # wrong answer whereas "is a string a boolean?" does not. Given a -flag with key "" and a default value "" When the flag was evaluated with details @@ -59,10 +59,16 @@ Feature: Provider error handling | Integer | 1 | | Float | 0.1 | - @strict-numeric-typing + @numeric-coercion Scenario: A float flag is not silently narrowed to an integer # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information # silently, so it must be reported as a type mismatch rather than rounded. + # + # This is the lossy half of the coercion contract. The lossless half -- that an + # integral float such as 10.0 requested as an integer MUST succeed -- has no scenario + # yet, because the canonical flag set has no integral float to ask it of. Adding one + # is a change to the flag set and so to every language at once; see the tag's entry in + # Appendix F. Given a Integer-flag with key "float-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "1" From 595bbbbeb0b419b238bf0e90eb983c0193275f25 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 09:22:35 +0200 Subject: [PATCH 07/63] fix: stop presenting the numeric coercion rule as normative OpenFeature The previous text opened with "the specification requires a provider to report TYPE_MISMATCH when the requested type cannot be satisfied" and concluded that not declaring the tag was "an admission of a known bug". The first claim is wrong, and the second followed from it. OpenFeature has one numeric type, deliberately. `number` is "a numeric value of unspecified type or size", and implementation languages MAY differentiate between integers and floating point numbers "as idioms dictate"; both the client and provider requirements say "boolean, numeric, string, and structure". Typed languages expose two accessors anyway, and no requirement says what a provider must do when a value does not fit the accessor it was asked through. That gap is now open-feature/spec#430. So the rule this capability is tested against is borrowed rather than normative. It comes from flagd's numeric coercion ADR, which is scoped to flagd's own implementations, and the tag took that ADR's name because flagd's testbed is gaining @numeric-coercion scenarios and two vocabularies for one observable property is worse than one borrowed name. The appendix now says so plainly, including that a provider behaving differently is not violating the specification -- which a conformance appendix has no business implying when the requirement does not exist. This also corrects why the capability is optional. It is genuinely optional, because the specification declines to specify the behaviour, and not optional as a concession to a provider with a defect. That distinction was inverted before. What remains true is unchanged and still worth stating: flagd narrows 0.5 to 0 with no error code at all, in two languages and both resolvers, so an application sees a plausible value and no signal. That is open-feature/flagd#1996, and a provider withholding the capability should say whether it is a choice or a tracked defect -- for which a report carries knownDeviations. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 51 ++++++++++++------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index cb69e3ae3..409cb4875 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -175,24 +175,41 @@ The design rule behind this: **a conformance suite that quietly goes green on sc run is worse than no suite at all.** A TCK implementation must report unsupported capabilities as skipped and surface the reason, not silently pass or silently omit them. -`@numeric-coercion` deserves a note, because unlike the others it is **not** an optional feature. -The specification requires a provider to report `TYPE_MISMATCH` when the requested type cannot be -satisfied, and narrowing `0.5` to `0` to satisfy an integer request loses information silently — the -worst failure mode for a feature flag, because the application sees a plausible value and no error. -It is a capability only so that a provider with this defect can adopt the suite today and see the -gap reported explicitly rather than being unable to adopt at all. Not declaring it is an admission -of a known bug. - -The rule is **lossless coercion is permitted; lossy coercion must fail**, and not the stricter "never -coerce" this tag was originally named for. An integral float such as `10.0` requested as an integer -must succeed; `0.5` must not. The distinction is flagd's +`@numeric-coercion` deserves a note, because it is the one capability here that **the specification +does not define**, and readers should not mistake it for one that does. + +OpenFeature has a single numeric type, deliberately: `number` is +[*"a numeric value of unspecified type or size"*](../types.md), and implementation languages **may** +further differentiate between integers and floating point numbers *"as idioms dictate"*. Both the +client and provider requirements say "boolean, numeric, string, and structure" — one numeric type, +not two. Typed-language SDKs take up that idiom and expose two accessors anyway, and at that point +no requirement answers the obvious question: what must a provider do when a value does not fit the +accessor it was asked through? `0.5` requested as an integer is not a corner case, it is the ordinary +consequence of a two-accessor SDK over a one-type wire format. That gap is +[open-feature/spec#430](https://github.com/open-feature/spec/issues/430). + +So the rule this tag is tested against is **borrowed, not normative**: lossless coercion is +permitted, lossy coercion must return `TYPE_MISMATCH`. An integral float such as `10.0` requested as +an integer must succeed; `0.5` must not. It comes from flagd's [numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), -and the tag was renamed from `@strict-numeric-typing` to match it rather than leave two vocabularies -describing one property — flagd's own testbed is gaining `@numeric-coercion` scenarios, and a -capability vocabulary that disagrees with the reference implementation about the name of a rule is -worse than having no name for it. - -Two gaps follow from adopting that rule, and both are open rather than fixed here: +which is scoped to flagd's own implementations, and the tag took the ADR's name — it was +`@strict-numeric-typing` — because flagd's testbed is gaining `@numeric-coercion` scenarios and two +vocabularies for one observable property is worse than one borrowed name. **A provider that behaves +differently is not violating the specification**, and this suite must not be read as saying it is. + +That is also why the capability is genuinely optional, rather than optional as a concession to a +known defect. An earlier draft of this appendix claimed the specification required the behaviour and +that not declaring the tag was "an admission of a known bug". That was wrong on the first count, and +therefore on the second. + +What remains true is that the observed behaviour is bad for users: flagd narrows `0.5` to `0` with no +error code at all, in Go and in Java, in both resolvers, so an application receives a plausible value +and no signal. That is being fixed in +[open-feature/flagd#1996](https://github.com/open-feature/flagd/issues/1996). A provider withholding +this capability should say which it is — a deliberate choice, or a tracked defect — and a conformance +report has `knownDeviations` for the second. + +Two further gaps, both open rather than fixed here: - **The lossless case has no scenario.** Only the lossy half is tested, so a provider that wrongly rejects `10.0` as an integer passes. Closing it needs an integral float in the canonical flag set, From a62f935e8649eb3368f085168dde7e6c9520ba67 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 09:25:42 +0200 Subject: [PATCH 08/63] style: pad the appendix tables to the repository's table style markdownlint's MD060 wants one space either side of a table pipe, and the appendix's tables were written without it -- `|---|---|` rather than `| --- | --- |`, and an empty leading cell as `| |` rather than `| |`. That failed the lint job, which is named json-lint but runs `make lint` and so covers markdown too. Applied with the repository's own `markdownlint --fix`, and only to the appendix. The tool also rewrites specification/assets/gherkin/README.md and specification/sections/01-flag-evaluation.md, which are pre-existing and belong to nobody's PR; those were reverted. CI never sees them in any case, because the Makefile globs `**/*.md` unquoted and without globstar, which matches one level deep only. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 409cb4875..746e8c9ea 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -26,8 +26,8 @@ asked of every provider in every language and get comparable answers. [Appendix B](./appendix-b-gherkin-suites.md) also contains Gherkin, and the distinction matters: -| | Appendix B | Appendix F | -|---|---|---| +| | Appendix B | Appendix F | +| --- | --- | --- | | Subject under test | the **SDK** | the **provider** | | Harness | an [in-memory provider](./appendix-a-included-utilities.md#in-memory-provider) | the provider under test, against a real backend | | Answers | does this SDK implement the Evaluation API, hooks and events correctly? | does this provider map its backend onto the provider contract correctly? | @@ -64,7 +64,7 @@ meaningless without the control endpoint that produces the disconnect. Changing others breaks the suite in every language at once. | Artifact | Location | What it defines | -|---|---|---| +| --- | --- | --- | | Gherkin scenarios | [`assets/provider-tck/gherkin/`](./assets/provider-tck/README.md) | the test cases themselves | | Canonical flag set | [`assets/provider-tck/flags/canonical-flags.json`](./assets/provider-tck/flags/canonical-flags.json) | the flags those cases assume | | Control API | [`assets/provider-tck/openapi/control-api.yaml`](./assets/provider-tck/openapi/control-api.yaml) | what a backend under test must expose | @@ -106,7 +106,7 @@ Scenarios need to change flags and simulate outages, and they need to do it iden vendors. The control API is a small HTTP surface the backend under test exposes for that purpose: | Endpoint | Required | Purpose | -|---|---|---| +| --- | --- | --- | | `POST /start?config=` | yes | start the backend, seeding flags to a named baseline | | `POST /stop` | yes | make the backend unreachable | | `POST /restart?seconds=` | yes | simulate an outage of a bounded duration | @@ -137,7 +137,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are **skipped, with the reason** — never as passed. | Tag | Meaning | -|---|---| +| --- | --- | | `@events` | emits lifecycle events at all | | `@lifecycle` | performs an initialisation that reaches its backend, with an observable outcome | | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | From c4fb21a48160c6e79be7c09462475af603bbb4d0 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 10:58:02 +0200 Subject: [PATCH 09/63] feat: say how the suite may be extended, and what an extension may not do Implementations have grown an extension point -- the adopter supplies feature files and step definitions and they run inside the suite, against the same backend, in the same lifecycle -- because the alternative an adopter reaches for is a parallel harness that reimplements provider registration, the readiness wait and the per-scenario reset, and then drifts from this one. The mechanism differs by language and the appendix does not prescribe one: Java and Python discover extensions by convention, Go by configuration. What it does prescribe is the four properties that stop an extension quietly becoming a conformance claim. Extension scenarios must be distinguishable from canonical ones, or an adopter's own passing scenarios flatter the result. They must not shadow a canonical scenario. They must never satisfy one -- a rule under which supplying enough scenarios of your own made the canonical ones optional would defeat the point of a canonical set. And a run that did not execute the canonical set in full must fail. That last one is stated because it is not obvious and because it was found by accident: a test selector matching a single scenario name produced a green suite and a well-formed conformance report describing one scenario out of twenty-nine. A mis-wired extension filesystem does the same. The report cannot defend itself here -- the schema is closed and the envelope carries no expected count, so there is no field a consumer could read to notice -- which leaves failing the run as the only lever an implementation has. Two details of that check are recorded because both are easy to get wrong. The expectation has to come from the same parser the runner uses, since a second parser disagrees about precisely the cases that matter -- an Examples block with its own tags, scenarios nested in a Rule. And a capability-gated skip is not a gap: the question was put and declined, and counting it as a gap would force every provider to declare every capability. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 746e8c9ea..778879fac 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -255,6 +255,49 @@ control path should **fail loudly** if a connection operation is reached anyway capability was declared that the harness cannot back up, which is a test-configuration bug rather than a provider defect. +## Extending the suite + +A provider often has behaviour this specification does not describe — flagd's fractional targeting, +a vendor's own segment rules — and no way to test it inside this suite. The alternative an adopter +reaches for is a parallel harness that reimplements provider registration, the readiness wait and +the per-scenario backend reset, and then drifts from the one here. So a TCK implementation **MAY** +offer an extension point: the adopter supplies feature files and step definitions, and they run +inside the same suite, against the same backend, in the same lifecycle. + +The mechanism is the implementation's own — a classpath scan, a `conftest.py`, two configuration +fields — and this appendix does not prescribe one. What it does prescribe is the four properties that +keep an extension from quietly becoming a conformance claim. + +**Extension scenarios must be distinguishable from canonical ones.** A results payload that mixes +them with no way to tell which is which lets an adopter's own passing scenarios flatter the +conformance result. Partitioning by path is enough — canonical features keep the path they have in +this repository and extensions mount under a reserved prefix — and it is what a consumer reads to +separate the two. + +**An extension must not shadow a canonical scenario.** The same partition provides this: an +extension file cannot occupy a canonical path, so it can add questions but never replace one. + +**Extension scenarios must never satisfy a canonical scenario.** An adopter's feature is an addition +to the canonical set, not a substitute for part of it. A rule under which supplying enough scenarios +of your own made the canonical ones optional would defeat the point of having a canonical set. + +**A run that did not execute the canonical set in full must fail.** This is the one that needs +stating because it is not obvious, and because it was found by accident rather than by design: a +test selector matching a single scenario name produced a green suite and a **well-formed conformance +report describing one scenario out of twenty-nine**. A mis-wired extension filesystem does the same. +There is no field in the report a consumer could read to notice — the schema is closed and the +envelope carries no expected count — so failing the run is the only lever the implementation has. + +Two details of that check are worth recording, because both are easy to get wrong: + +- **The expectation must come from the same parser the runner uses**, not a second one written for + the check. A parser of the implementation's own will disagree with the runner about exactly the + cases that matter — an `Examples` block carrying its own tags, scenarios nested in a `Rule` — and + the expectation has to be what a full run would actually have produced. +- **A capability-gated skip is not a gap.** The scenario ran the gate and is reported as skipped with + its reason, so the question was put and declined. Treating that as a gap would force every provider + to declare every capability, which is the opposite of what the vocabulary is for. + ## Reference implementation The first implementation is `tools/provider-tck` in From 17207c7cb742c80eb00e374bd6ac2cdc74c4cecd Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:09:50 +0200 Subject: [PATCH 10/63] feat: publish the provider conformance assets as a Go module The Go conformance suite cannot embed the artifacts out of a git submodule: a Go module is distributed as a zip of the VCS tree, where a submodule is only a gitlink, so the embed would arrive empty for anyone running go get. Its workaround so far was to commit a generated copy of every artifact and fail CI whenever the copy drifted from the submodule. A go.mod and an embed.FS in this directory make it a nested Go module the suite can depend on directly, by tag or by commit, with the checksum database guaranteeing that the pinned revision is immutable. The module has no code beyond the embed directives and no dependencies, and the other language suites keep consuming this directory through the submodule; the two files are inert for them. This is the same arrangement open-feature/flagd-schemas uses to ship its JSON schemas to flagd. Signed-off-by: Simon Schrottner --- .../assets/provider-tck/.gitattributes | 4 +++ specification/assets/provider-tck/README.md | 12 +++++++ specification/assets/provider-tck/embed.go | 35 +++++++++++++++++++ specification/assets/provider-tck/go.mod | 3 ++ 4 files changed, 54 insertions(+) create mode 100644 specification/assets/provider-tck/embed.go create mode 100644 specification/assets/provider-tck/go.mod diff --git a/specification/assets/provider-tck/.gitattributes b/specification/assets/provider-tck/.gitattributes index c617195d2..e1bb310bf 100644 --- a/specification/assets/provider-tck/.gitattributes +++ b/specification/assets/provider-tck/.gitattributes @@ -4,3 +4,7 @@ *.feature text eol=lf *.json text eol=lf *.yaml text eol=lf + +# embed.go ships inside the Go module zip, whose checksum is content-addressed, so it too +# must be identical whichever platform it is committed from. +*.go text eol=lf diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 292946c8a..b77bca321 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -25,3 +25,15 @@ A feature file that evaluates `boolean-flag` is meaningless without the flag def - **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. + +## Consuming from Go + +This directory is also a Go module, `github.com/open-feature/spec/specification/assets/provider-tck`, whose only content is an `embed.FS` of the artifacts above. The Go conformance suite depends on it instead of vendoring a copy: a Go module ships as a zip of the VCS tree, in which a git submodule is only a gitlink, so an embed from a submodule would arrive empty for anyone running `go get`. The other languages build from a working tree and keep using the submodule; `go.mod` and `embed.go` are inert for them. + +A consumer pins a revision the usual way, by tag or by commit: + +```console +go get github.com/open-feature/spec/specification/assets/provider-tck@ +``` + +Nested Go modules are tagged with their path as a prefix, so a release of these assets is tagged `specification/assets/provider-tck/vX.Y.Z`; the specification's own `vX.Y.Z` tags do not apply to it. diff --git a/specification/assets/provider-tck/embed.go b/specification/assets/provider-tck/embed.go new file mode 100644 index 000000000..926e0f756 --- /dev/null +++ b/specification/assets/provider-tck/embed.go @@ -0,0 +1,35 @@ +// Package providertck carries the provider conformance assets as a Go module, +// so a Go conformance suite can depend on a specific revision of them the way +// it depends on any other module. +// +// The other language suites consume this directory through a git submodule, +// which works because a wheel or a JAR is built from a working tree where the +// submodule is present. A Go module is distributed as a zip built from the +// VCS tree, where a submodule is only a gitlink and its files are absent, so +// the Go suite would otherwise have to commit a copy of every artifact and +// police it against drift. Publishing the artifacts as a module removes the +// copy: the consumer pins a commit or tag in its go.mod, the Go checksum +// database makes that revision immutable, and the embedded bytes are the same +// bytes every other language reads out of the submodule. +// +// The module contains no code beyond this file and has no dependencies. It is +// inert for every consumer that is not Go. See +// https://github.com/open-feature/spec/issues/417. +package providertck + +import "embed" + +// FS holds the conformance artifacts, keyed by their path relative to this +// directory, so that they are addressed here exactly as they are documented: +// +// gherkin/*.feature the canonical scenarios +// flags/canonical-flags.json the flag set those scenarios assume +// openapi/control-api.yaml the HTTP surface a backend under test exposes +// +// The README and .gitattributes beside them are not artifacts and are not +// embedded. +// +//go:embed gherkin/*.feature +//go:embed flags/canonical-flags.json +//go:embed openapi/control-api.yaml +var FS embed.FS diff --git a/specification/assets/provider-tck/go.mod b/specification/assets/provider-tck/go.mod new file mode 100644 index 000000000..1080ae17e --- /dev/null +++ b/specification/assets/provider-tck/go.mod @@ -0,0 +1,3 @@ +module github.com/open-feature/spec/specification/assets/provider-tck + +go 1.21 From d1c30aca550ef1d5aaea228cc11ea38f972c1150 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:46:47 +0200 Subject: [PATCH 11/63] feat: release the provider conformance assets on their own tags The assets are consumed by four language suites, and each of them has to pin a revision: Go through go.mod, the other three through a submodule. A bare commit works and says nothing. Nobody can tell from `dc4d7ae` whether a scenario was added, and no bot can tell whether the pin is behind. So release them, with release-please in manifest mode and one package. A nested Go module is tagged with its path as a prefix, so `tag-separator: "/"` and a package name of the directory produce `specification/assets/provider-tck/vX.Y.Z` -- the same shape as `providers/flagd/v0.6.0` in go-sdk-contrib, and the only shape the Go toolchain will resolve for a module in a subdirectory. A submodule can carry the same tag in its `branch` field, which leaves the recorded gitlink a commit -- `git submodule update --init` and `actions/checkout` are unaffected -- and lets Renovate's git-submodules manager see the release. No root package is configured. The specification's own vX.Y.Z tags stay hand-cut, and release-please will neither create nor advance them; it has no opinion about anything outside the one configured path. One deviation from the sibling packages in the contrib repositories, which set `bump-patch-for-minor-pre-major` and so release features as patches: here a new scenario raises the bar a provider must clear, and an adopter whose build goes red did not change anything. That is a minor, not a patch. Patches are reserved for changes that cannot alter a result. Starting the manifest at 0.0.0 rather than seeding a version means the first tag is cut from the commits like every later one, so the manifest and the tags cannot disagree. With the configuration above a feature lands it at v0.1.0. Signed-off-by: Simon Schrottner --- .github/workflows/release-please.yaml | 31 +++++++++++++++ .release-please-manifest.json | 3 ++ release-please-config.json | 17 ++++++++ specification/assets/provider-tck/README.md | 43 +++++++++++++++++++-- 4 files changed, 91 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/release-please.yaml create mode 100644 .release-please-manifest.json create mode 100644 release-please-config.json diff --git a/.github/workflows/release-please.yaml b/.github/workflows/release-please.yaml new file mode 100644 index 000000000..3cf33b832 --- /dev/null +++ b/.github/workflows/release-please.yaml @@ -0,0 +1,31 @@ +name: Run Release Please + +# Releases the versioned artifacts kept in this repository. It does NOT release the +# specification itself: the specification's own vX.Y.Z tags are cut by hand, and no root +# package is configured here, so release-please neither creates nor advances them. +# +# Today the one package is specification/assets/provider-tck, the provider conformance +# assets. Those are consumed by four language suites which each have to pin a revision of +# them, and a pinned revision needs a name a human can read and a bot can compare. See +# release-please-config.json. + +on: + push: + branches: + - main + +permissions: + contents: write + pull-requests: write + +jobs: + release-please: + runs-on: ubuntu-latest + steps: + - uses: googleapis/release-please-action@v5 + with: + command: manifest + token: ${{ secrets.GITHUB_TOKEN }} + default-branch: main + # This repository enforces DCO, so the bot's own commits need a sign-off too. + signoff: "OpenFeature Bot <109696520+openfeaturebot@users.noreply.github.com>" diff --git a/.release-please-manifest.json b/.release-please-manifest.json new file mode 100644 index 000000000..39a253925 --- /dev/null +++ b/.release-please-manifest.json @@ -0,0 +1,3 @@ +{ + "specification/assets/provider-tck": "0.0.0" +} diff --git a/release-please-config.json b/release-please-config.json new file mode 100644 index 000000000..56c67995c --- /dev/null +++ b/release-please-config.json @@ -0,0 +1,17 @@ +{ + "$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json", + "monorepo-tags": true, + "tag-separator": "/", + "bootstrap-sha": "dd235837f7b2ddb1120ab6e1bde940bf4d532313", + "separate-pull-requests": true, + "packages": { + "specification/assets/provider-tck": { + "release-type": "go", + "package-name": "specification/assets/provider-tck", + "changelog-path": "CHANGELOG.md", + "bump-minor-pre-major": true, + "versioning": "default", + "extra-files": [] + } + } +} diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index b77bca321..fd67c1e6f 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -30,10 +30,47 @@ The flag set is expressed in the flagd flag-definition format because that is th This directory is also a Go module, `github.com/open-feature/spec/specification/assets/provider-tck`, whose only content is an `embed.FS` of the artifacts above. The Go conformance suite depends on it instead of vendoring a copy: a Go module ships as a zip of the VCS tree, in which a git submodule is only a gitlink, so an embed from a submodule would arrive empty for anyone running `go get`. The other languages build from a working tree and keep using the submodule; `go.mod` and `embed.go` are inert for them. -A consumer pins a revision the usual way, by tag or by commit: +A consumer pins a release the usual way: ```console -go get github.com/open-feature/spec/specification/assets/provider-tck@ +go get github.com/open-feature/spec/specification/assets/provider-tck@v0.1.0 ``` -Nested Go modules are tagged with their path as a prefix, so a release of these assets is tagged `specification/assets/provider-tck/vX.Y.Z`; the specification's own `vX.Y.Z` tags do not apply to it. +## Releases + +These assets are released independently of the specification, by [release-please](../../../release-please-config.json). A nested Go module is tagged with its path as a prefix, so a release is tagged `specification/assets/provider-tck/vX.Y.Z` — the same shape as `providers/flagd/v0.6.0` in the SDK contrib repositories. The specification's own `vX.Y.Z` tags are cut by hand and do not apply here; nothing about the two numbering schemes is related. + +What a bump means is not the usual thing, because this is a test suite rather than a library: + +- **A minor bump may turn a passing suite red.** Adding a scenario, or tightening one, raises the bar a provider has to clear. Nothing changed on the adopter's side and their build can still go from green to red, which is the point of adopting a conformance suite and is why new scenarios are released as minors rather than as patches. +- **A patch bump cannot.** Patches are editorial: a clarified scenario name, a comment, a fix to something that never ran. + +So pinning is not optional bookkeeping. A suite that floats on the latest assets cannot distinguish a regression in the provider from a new question being asked of it. + +## Consuming from the other three languages + +Java, Python and JavaScript reach these files through a git submodule of this repository, because a JAR, a wheel and an npm package are all built from a working tree where the submodule is present. A submodule can track the release tag rather than a bare commit, which makes the pin readable in review: + +```ini +[submodule "spec"] + path = tools/provider-tck/spec + url = https://github.com/open-feature/spec.git + branch = specification/assets/provider-tck/v0.1.0 +``` + +The recorded gitlink is still a commit, so `git submodule update --init` and `actions/checkout` with `submodules: recursive` behave exactly as before. Only `git submodule update --remote` is affected, which resolves `branch` and will report that the tag is not a branch — do not use it on a submodule pinned this way. + +## Keeping a pin current + +Both forms are updatable by [Renovate](https://docs.renovatebot.com), so an adopting repository is told about a new release rather than discovering it: + +- Go: the `gomod` manager, on by default, raises a PR for a new `specification/assets/provider-tck/vX.Y.Z`. +- The submodule: the `git-submodules` manager, which is opt-in and reads the tag out of `branch` above. + +```json +{ + "git-submodules": { "enabled": true } +} +``` + +Let those PRs run the suite. A red one is the report that conformance narrowed, and reading it is the work — which is why it is worth *not* automerging these. From 6636e1ddacac8b3313690f42d4dfeb609e0fbf4c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:49:10 +0200 Subject: [PATCH 12/63] docs: state the suite's coverage of the numbered provider requirements The open questions said only that hooks and flag metadata were not covered. Mapped against 02-providers.md the picture is 8 of 14 MUST, 3 of 5 SHOULD and 1 of 6 MAY, with the six MUST gaps named, so a reader of this appendix can see what a green run does and does not claim without reading the tracking issue. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 778879fac..8faec2ff0 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -317,7 +317,17 @@ belong here rather than in any one implementation: designated flag. Finer-grained flag manipulation would need new endpoints. - **Caching.** Whether a stale provider keeps serving last-known values during an outage depends on whether it holds a local copy of the ruleset. The `@caching` tag is reserved; no scenarios yet. -- **Hooks and flag metadata.** Not covered. +- **Coverage of the numbered requirements.** Mapped against + [the provider requirements](./sections/02-providers.md), leaving out 2.8.5.1 (it constrains the SDK) + and 2.2.8.1 (a language-binding property, not observable at runtime), the suite covers 8 of the 14 + `MUST` requirements in scope, 3 of 5 `SHOULD`, and 1 of 6 `MAY`. The `MUST` gaps are 2.1.1 (a + non-empty metadata `name`), 2.3.1 (the provider hook mechanism), 2.3.2 (no `error message` on + normal execution), 2.2.10 (flag metadata structure), 2.4.4 (a domain-scoped provider accepts its + bound domain) and 2.8.4 (`PROVIDER_CONTEXT_CHANGED`). Shutdown is not tested at all (2.5.2 and + 2.5.3, both `SHOULD`), although the suite already performs one to release connections at the end of + a run. Flag metadata (2.2.9, 2.2.10) is blocked on the canonical flag set defining none. Of these, + 2.8.4 is the largest hole: context reconciliation is where a provider is most likely to serve values + computed for the *previous* context, and the failure is silent. - **Normative status.** Nothing in this appendix is currently expressed as a numbered requirement. Whether the control API contract and the capability vocabulary should become normative sections is a decision for the TSC. From ad7b8eb10020d3497a163b953a563f05ba2002ef Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:54:15 +0200 Subject: [PATCH 13/63] feat: close the cheap provider-contract gaps and test the edge cases Stacked on the Appendix F PR, which stays the method; this is the tests. Closes two MUST gaps and the two SHOULD gaps the coverage mapping named: - 2.3.2: every success path now asserts the error message is empty. - 2.1.1: metadata.feature asserts a non-empty provider name, which the conformance report keys on. - 2.5.2 and 2.5.3: shutdown is idempotent, a shut-down provider can be initialized again and then serves flags, and shutdown against an unreachable backend completes rather than hanging. Adds the edge cases review asked for: - false, 0 and "" resolve as values, each with a default that differs, so a provider that treats a falsy result as an absence is caught twice. - Integer precision at 2^31 - 1, untagged, which a float32 round trip changes; and at 2^53 - 1 under a new @large-integers tag, because Java's integer accessor is a 32-bit Integer and cannot ask for it at all. - The lossless half of @numeric-coercion in both directions: 10.0 as an integer and 10 as a float must succeed. Rejecting every float passed the lossy scenario alone. Five flags join the canonical set. The README says which of them a seeding step is likely to mangle and how. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 53 +++++++++------ specification/assets/provider-tck/README.md | 9 ++- .../provider-tck/flags/canonical-flags.json | 68 ++++++++++++++++++- .../provider-tck/gherkin/errors.feature | 31 +++++++-- .../provider-tck/gherkin/evaluation.feature | 59 ++++++++++++++++ .../provider-tck/gherkin/lifecycle.feature | 46 ++++++++++++- .../provider-tck/gherkin/metadata.feature | 9 +++ 7 files changed, 244 insertions(+), 31 deletions(-) create mode 100644 specification/assets/provider-tck/gherkin/metadata.feature diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 8faec2ff0..bfb6ba1ff 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -39,11 +39,16 @@ claims, and a language needs both suites to make both. **In scope — the provider contract:** -- mapping backend responses onto typed resolution details: value, variant, reason, error code +- mapping backend responses onto typed resolution details: value, variant, reason, error code, and + no error message on a normal evaluation +- the values most often mistaken for an absence — `false`, `0` and `""` — resolving as values +- integer precision: a 32-bit maximum for every language, and 2^53 − 1 where the accessor allows it - keeping the integer and float types distinct rather than coercing between them - error handling: a type mismatch and an unknown flag return the code default, report the right error code, and never throw -- lifecycle: reaching `READY`, and settling into `ERROR` against an unreachable backend +- identity: a non-empty metadata name +- lifecycle: reaching `READY`, settling into `ERROR` against an unreachable backend, and a shutdown + that is idempotent, reversible by initialising again, and prompt when the backend is gone - events: `PROVIDER_READY`, `PROVIDER_ERROR`, `PROVIDER_STALE`, `PROVIDER_CONFIGURATION_CHANGED` - that a signalled configuration change is actually **applied** on re-evaluation, not merely signalled @@ -145,6 +150,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@object` | supports structured flag values | | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | +| `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `@targeting` | reserved; **not declarable** -- no scenarios yet | | `@caching` | reserved; **not declarable** -- no scenarios yet | @@ -209,15 +215,21 @@ and no signal. That is being fixed in this capability should say which it is — a deliberate choice, or a tracked defect — and a conformance report has `knownDeviations` for the second. -Two further gaps, both open rather than fixed here: - -- **The lossless case has no scenario.** Only the lossy half is tested, so a provider that wrongly - rejects `10.0` as an integer passes. Closing it needs an integral float in the canonical flag set, - which changes the flag set for every language at once. -- **Accessor width is not modelled.** The ADR distinguishes the width of a language's integer - accessor — Go's `ResolveIntValue` is `int64` and so is the canonical `Long`, whereas a 32-bit - accessor needs its own scenarios, which flagd's testbed tags `@int32-bounded`. This appendix has - nothing equivalent, and it is a real source of cross-language disagreement. +Both halves of the rule have scenarios. The lossy half asks for `float-flag` (`0.5`) as an integer +and expects `TYPE_MISMATCH`; the lossless half asks for `integral-float-flag` (`10.0`) as an integer +and for `integer-flag` (`10`) as a float, and expects both to succeed. A provider declaring the tag +must satisfy all three — rejecting every float is an easy way to pass the first, and the other two +are what stop it. A provider whose SDK has a single numeric type, such as JavaScript, cannot +distinguish the cases and reports the tag as not applicable rather than declaring it. + +**Accessor width** is the related property the ADR distinguishes, and it is modelled separately +because it is a property of the SDK rather than of the provider. Every language's integer accessor +can ask for 2^31 − 1, so that precision scenario is untagged. Only some can ask for 2^53 − 1: Go's +`ResolveIntValue` is `int64`, but Java's accessor is a 32-bit `Integer`, and a provider cannot +resolve a value the accessor has no room for. That scenario carries `@large-integers`, which a +provider on a 32-bit accessor leaves undeclared. Nothing above 2^53 − 1 is asked for: JavaScript +cannot represent it, and what a provider owes a value that does not fit the requested accessor is +the open question in [open-feature/spec#430](https://github.com/open-feature/spec/issues/430). ## Implementing the suite in a language @@ -319,15 +331,16 @@ belong here rather than in any one implementation: whether it holds a local copy of the ruleset. The `@caching` tag is reserved; no scenarios yet. - **Coverage of the numbered requirements.** Mapped against [the provider requirements](./sections/02-providers.md), leaving out 2.8.5.1 (it constrains the SDK) - and 2.2.8.1 (a language-binding property, not observable at runtime), the suite covers 8 of the 14 - `MUST` requirements in scope, 3 of 5 `SHOULD`, and 1 of 6 `MAY`. The `MUST` gaps are 2.1.1 (a - non-empty metadata `name`), 2.3.1 (the provider hook mechanism), 2.3.2 (no `error message` on - normal execution), 2.2.10 (flag metadata structure), 2.4.4 (a domain-scoped provider accepts its - bound domain) and 2.8.4 (`PROVIDER_CONTEXT_CHANGED`). Shutdown is not tested at all (2.5.2 and - 2.5.3, both `SHOULD`), although the suite already performs one to release connections at the end of - a run. Flag metadata (2.2.9, 2.2.10) is blocked on the canonical flag set defining none. Of these, - 2.8.4 is the largest hole: context reconciliation is where a provider is most likely to serve values - computed for the *previous* context, and the failure is silent. + and 2.2.8.1 (a language-binding property, not observable at runtime), the suite covers 10 of the + 14 `MUST` requirements in scope, all 5 `SHOULD`, and 1 of 6 `MAY`. The `MUST` gaps are 2.3.1 (the + provider hook mechanism, a compile-time property in typed languages with little to observe at + runtime), 2.2.10 (flag metadata structure, blocked with 2.2.9 on the canonical flag set defining + none), 2.4.4 (a domain-scoped provider accepts its bound domain, which is as much SDK as provider + behaviour) and 2.8.4 (`PROVIDER_CONTEXT_CHANGED`). The last is the largest hole: context + reconciliation is where a provider is most likely to serve values computed for the *previous* + context, and the failure is silent. It wants its own capability tag, and until the control API has + an echo operation a scenario can show only that reconciliation was signalled, not that the values + that follow are the new context's. - **Normative status.** Nothing in this appendix is currently expressed as a numbered requirement. Whether the control API contract and the capability vocabulary should become normative sections is a decision for the TSC. diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index fd67c1e6f..413bc10b5 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -8,10 +8,11 @@ These validate a **provider** against a real backend. For assets that validate a | Path | What it is | | --- | --- | -| [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value, variant and reason | -| [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix and the unknown-flag case | +| [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value, variant and reason; falsy values; integer precision | +| [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix, numeric coercion and the unknown-flag case | | [`gherkin/events.feature`](./gherkin/events.feature) | configuration change, and the stale/ready transition across an outage | -| [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one | +| [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one; shutdown | +| [`gherkin/metadata.feature`](./gherkin/metadata.feature) | the provider identifies itself by name | | [`flags/canonical-flags.json`](./flags/canonical-flags.json) | the flag set every scenario assumes | | [`openapi/control-api.yaml`](./openapi/control-api.yaml) | the HTTP surface a backend under test must expose | @@ -25,6 +26,8 @@ A feature file that evaluates `boolean-flag` is meaningless without the flag def - **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. +- **`false-flag`, `zero-flag` and `empty-string-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. +- **`integral-float-flag` is a float and `huge-integer-flag` is an integer.** Seeding `10.0` as `10` makes the lossless-coercion scenario pass without coercing; seeding `9007199254740991` through a float rounds it. ## Consuming from Go diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index 343b3ae52..ff73357d9 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -10,7 +10,19 @@ "Two things are load-bearing and easy to get wrong:", " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", - " tests the provider's mapping of a response, not the backend's evaluation logic." + " tests the provider's mapping of a response, not the backend's evaluation logic.", + "", + "Three more are easy to lose in translation, because a seeding step that 'cleans up' values", + "destroys exactly what they test:", + " * 'false-flag', 'zero-flag' and 'empty-string-flag' resolve to false, 0 and \"\". They are", + " values, not absences. A backend that drops them, or a provider that treats them as", + " missing, is what the falsy-value scenarios catch.", + " * 'large-integer-flag' resolves to 9007199254740991, which is 2^53 - 1: the largest integer", + " every language can represent exactly, including JavaScript. Seed it as an integer, not a", + " float, and do not round it.", + " * 'integral-float-flag' resolves to 10.0, a float with no fractional part. It MUST be seeded", + " as a float. A backend that stores it as the integer 10 makes the lossless-coercion scenario", + " pass without coercing anything." ], "flags": { "boolean-flag": { @@ -45,6 +57,60 @@ }, "defaultVariant": "half" }, + "large-integer-flag": { + "$comment": "2^31 - 1, the largest 32-bit signed integer. Every language represents it exactly; a float32 round trip does not.", + "state": "ENABLED", + "variants": { + "one": 1, + "max-int32": 2147483647 + }, + "defaultVariant": "max-int32" + }, + "huge-integer-flag": { + "$comment": "2^53 - 1, the largest integer JavaScript represents exactly. Only asked for under @large-integers, because a 32-bit integer accessor cannot request it at all.", + "state": "ENABLED", + "variants": { + "one": 1, + "max-safe": 9007199254740991 + }, + "defaultVariant": "max-safe" + }, + "integral-float-flag": { + "$comment": "A float with no fractional part, for the lossless half of @numeric-coercion. Seed as a float.", + "state": "ENABLED", + "variants": { + "tenth": 0.1, + "ten": 10.0 + }, + "defaultVariant": "ten" + }, + "false-flag": { + "$comment": "Resolves to false. The default in the scenario is true, so a provider that treats false as missing is caught.", + "state": "ENABLED", + "variants": { + "on": true, + "off": false + }, + "defaultVariant": "off" + }, + "zero-flag": { + "$comment": "Resolves to 0. The default in the scenario is 1, so a provider that treats 0 as missing is caught.", + "state": "ENABLED", + "variants": { + "one": 1, + "zero": 0 + }, + "defaultVariant": "zero" + }, + "empty-string-flag": { + "$comment": "Resolves to the empty string. The default in the scenario is 'fallback', so a provider that treats \"\" as missing is caught.", + "state": "ENABLED", + "variants": { + "greeting": "hi", + "empty": "" + }, + "defaultVariant": "empty" + }, "object-flag": { "state": "ENABLED", "variants": { diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature index 0efbe5726..fd696092d 100644 --- a/specification/assets/provider-tck/gherkin/errors.feature +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -64,11 +64,10 @@ Feature: Provider error handling # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information # silently, so it must be reported as a type mismatch rather than rounded. # - # This is the lossy half of the coercion contract. The lossless half -- that an - # integral float such as 10.0 requested as an integer MUST succeed -- has no scenario - # yet, because the canonical flag set has no integral float to ask it of. Adding one - # is a change to the flag set and so to every language at once; see the tag's entry in - # Appendix F. + # This is the lossy half of the coercion contract; the two scenarios that follow are the + # lossless half. A provider declaring @numeric-coercion must satisfy all three. Rejecting + # 0.5 is easy to get right by rejecting every float, and the lossless scenarios are what + # stop that shortcut from passing. Given a Integer-flag with key "float-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "1" @@ -76,6 +75,28 @@ Feature: Provider error handling And the error-code should be "TYPE_MISMATCH" And no exception should have been thrown + @numeric-coercion + Scenario: An integral float requested as an integer is coerced without loss + # 'integral-float-flag' resolves to 10.0. Nothing is lost by returning it as the integer + # 10, so the coercion rule permits it and a provider declaring the tag must perform it. + Given a Integer-flag with key "integral-float-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "10" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + + @numeric-coercion + Scenario: An integer requested as a float is widened without loss + # The other direction. 'integer-flag' resolves to 10; every integer this suite asks for + # is exactly representable as a float, so a provider declaring the tag must widen it. + Given a Float-flag with key "integer-flag" and a default value "0.1" + When the flag was evaluated with details + Then the resolved details value should be "10" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + Scenario: An unknown flag key returns the code default # 'missing-flag' is deliberately absent from the canonical flag set. Given a String-flag with key "missing-flag" and a default value "fallback" diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index e89f174a5..a1b01da2d 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -6,6 +6,10 @@ Feature: Provider flag evaluation # to its default variant with no targeting involved, so what is under test is purely the # provider's mapping of a backend response to a value, a variant and a reason. # + # Every success path also asserts that no error message was set (requirement 2.3.2). A + # provider that reports a value AND an error message is sending two contradictory signals, + # and an application reading the message will believe the wrong one. + # # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. Background: @@ -18,6 +22,7 @@ Feature: Provider flag evaluation And the variant should be "" And the reason should be "" And the error-code should be "" + And the error message should be empty And no exception should have been thrown Examples: @@ -27,6 +32,57 @@ Feature: Provider flag evaluation | integer-flag | Integer | 1 | 10 | ten | STATIC | | float-flag | Float | 0.1 | 0.5 | half | STATIC | + Scenario Outline: A falsy value is a value, not an absence + # false, 0 and "" are the values most likely to be mistaken for "nothing came back": a + # `value || default` in JavaScript, a zero-value check in Go, an `if not value` in Python. + # Each row's default differs from its resolved value, so a provider that falls back on a + # falsy result returns the wrong value AND the wrong reason, and is caught twice over. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "STATIC" + And the error-code should be "" + And the error message should be empty + And no exception should have been thrown + + Examples: + | key | type | default | value | variant | + | false-flag | Boolean | true | false | off | + | zero-flag | Integer | 1 | 0 | zero | + | empty-string-flag | String | fallback | | empty | + + Scenario: A large integer resolves without loss of precision + # 2147483647 is 2^31 - 1, the largest 32-bit signed integer, so every language's integer + # accessor can ask for it. It is also outside what a 32-bit float represents exactly, so a + # provider that routes integers through float32 and back returns 2147483648. + Given a Integer-flag with key "large-integer-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "2147483647" + And the variant should be "max-int32" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + + @large-integers + Scenario: An integer beyond 32 bits resolves without loss of precision + # 9007199254740991 is 2^53 - 1: the largest integer JavaScript represents exactly, and + # comfortably inside a 64-bit integer. Anything that routes the value through a 32-bit + # integer, or through a 64-bit float and back with rounding, changes it. + # + # Tagged, because whether it can be asked for at all is a property of the language's + # SDK rather than of the provider: Java's integer accessor is a 32-bit Integer, and a + # provider cannot resolve a value the accessor has no room for. Values above 2^53 - 1 are + # deliberately not asked for. What a provider must do with a value that does not fit the + # requested accessor is an open question of the provider contract (open-feature/spec#430). + Given a Integer-flag with key "huge-integer-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "9007199254740991" + And the variant should be "max-safe" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + Scenario: An integer flag resolves as an integer # Paired with the float scenario below and with the narrowing scenario in errors.feature. # Together they pin down that the two numeric types stay distinct rather than both being @@ -35,6 +91,7 @@ Feature: Provider flag evaluation When the flag was evaluated with details Then the resolved details value should be "10" And the error-code should be "" + And the error message should be empty And no exception should have been thrown Scenario: A float flag resolves as a float @@ -42,6 +99,7 @@ Feature: Provider flag evaluation When the flag was evaluated with details Then the resolved details value should be "0.5" And the error-code should be "" + And the error message should be empty And no exception should have been thrown @object @@ -51,6 +109,7 @@ Feature: Provider flag evaluation Then the variant should be "template" And the reason should be "STATIC" And the error-code should be "" + And the error message should be empty And no exception should have been thrown And the resolved object value should contain | key | type | value | diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature index 3e168565a..15eb735f8 100644 --- a/specification/assets/provider-tck/gherkin/lifecycle.feature +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -2,7 +2,9 @@ Feature: Provider lifecycle # Verifies the two terminal outcomes of provider initialisation: reaching READY against a - # healthy backend, and settling into ERROR against one that cannot be reached. + # healthy backend, and settling into ERROR against one that cannot be reached. And the other + # end of the lifecycle: that shutdown releases what initialisation acquired, can be repeated, + # and does not hang when the backend is gone. # # Gated by @lifecycle rather than @events, and the distinction is load-bearing. Every SDK # synthesises PROVIDER_READY for a provider that has no initialisation step, so a provider @@ -12,8 +14,12 @@ Feature: Provider lifecycle # emits events does not necessarily do that. # # The failure case matters more than it looks. A provider that blocks forever, or throws out - # of provider registration, takes the host application down with it — so the requirement is + # of provider registration, takes the host application down with it -- so the requirement is # not merely that initialisation fails, but that it fails observably and promptly. + # + # The shutdown steps call the provider's own shutdown and initialize functions directly, not + # the SDK's. The SDK shuts a provider down when it is replaced, but wrapping that in a scenario + # would test the SDK's bookkeeping as much as the provider's, and Appendix B already does that. Scenario: A provider that successfully initializes becomes ready Given a stable provider @@ -38,3 +44,39 @@ Feature: Provider lifecycle Then the resolved details value should be "false" And the reason should be "ERROR" And no exception should have been thrown + + Scenario: Shutting down a provider twice has no further effect + # Requirement 2.5.3. Double-close is the classic shutdown bug: the second call finds a + # closed channel, a null connection or a disposed client, and throws from a code path the + # application runs during its own shutdown -- where an exception is least welcome. + Given a stable provider + When the provider is shut down + And the provider is shut down + Then no exception should have been thrown + + Scenario: A provider that was shut down can be initialized again + # Requirement 2.5.2: after shutdown the provider reverts to its uninitialized state, which + # is observable as exactly one thing -- it can be initialized again and then serves flags. + # A provider that shuts down by discarding its client and never recreating it passes the + # scenario above and fails this one. + Given a stable provider + And a Boolean-flag with key "boolean-flag" and a default value "false" + When the provider is shut down + And the provider is initialized again + And the flag was evaluated with details + Then the resolved details value should be "true" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + + @unavailable + Scenario: Shutting down a provider that cannot reach its backend completes promptly + # A shutdown that waits for a graceful close of a connection that will never answer hangs + # the host application's own shutdown. The bound is generous; what is being asserted is + # that shutdown returns at all rather than blocking on the backend. + Given a unavailable provider + And a error event handler + Then the error event handler should have been executed within 10000ms + When the provider is shut down + Then the shutdown should have completed within 10000ms + And no exception should have been thrown diff --git a/specification/assets/provider-tck/gherkin/metadata.feature b/specification/assets/provider-tck/gherkin/metadata.feature new file mode 100644 index 000000000..b8422447e --- /dev/null +++ b/specification/assets/provider-tck/gherkin/metadata.feature @@ -0,0 +1,9 @@ +Feature: Provider metadata + + # Verifies that a provider identifies itself (requirement 2.1.1). This looks too small to + # test, and was left untested for exactly that reason -- until a conformance report keyed on + # the provider's metadata name made an empty name into a report nobody can attribute. + + Scenario: A provider identifies itself by name + Given a stable provider + Then the provider metadata name should not be empty From f68677f3e786d461c7a2829b805724f579d59b9e Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:56:41 +0200 Subject: [PATCH 14/63] docs: list metadata.feature among the feature files Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index bfb6ba1ff..a33242e6a 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -76,16 +76,18 @@ others breaks the suite in every language at once. ### Gherkin scenarios -Four feature files: +Five feature files: - [`evaluation.feature`](./assets/provider-tck/gherkin/evaluation.feature) — resolving each type with - the right value, variant and reason -- [`errors.feature`](./assets/provider-tck/gherkin/errors.feature) — the type-mismatch matrix and the - unknown-flag case + the right value, variant and reason; falsy values; integer precision +- [`errors.feature`](./assets/provider-tck/gherkin/errors.feature) — the type-mismatch matrix, numeric + coercion and the unknown-flag case - [`events.feature`](./assets/provider-tck/gherkin/events.feature) — configuration change, and the stale/ready transition across an outage - [`lifecycle.feature`](./assets/provider-tck/gherkin/lifecycle.feature) — initialisation against a - healthy backend and against an unreachable one + healthy backend and against an unreachable one; shutdown +- [`metadata.feature`](./assets/provider-tck/gherkin/metadata.feature) — the provider identifies + itself by name The step vocabulary is inherited from the [flagd test harness](https://github.com/open-feature/test-harness) wherever it was already From cd85816bdfae5fb109d40bb1ed54ae5c0ef6ee80 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:58:34 +0200 Subject: [PATCH 15/63] docs: keep the load-bearing flag properties in one list Two bullets added with the falsy-value and precision flags landed below the paragraph that closes the list, which rendered them as a stranded list after prose. Moved them up with the other two, and the heading now counts four. Signed-off-by: Simon Schrottner --- specification/assets/provider-tck/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 413bc10b5..f99f4860b 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -20,15 +20,15 @@ These validate a **provider** against a real backend. For assets that validate a A feature file that evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is meaningless without the control endpoint that produces the disconnect. Changing one without the others breaks the suite in every language at once. -## Two properties that are load-bearing +## Four properties that are load-bearing - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. - **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. - -The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. - **`false-flag`, `zero-flag` and `empty-string-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. - **`integral-float-flag` is a float and `huge-integer-flag` is an integer.** Seeding `10.0` as `10` makes the lossless-coercion scenario pass without coercing; seeding `9007199254740991` through a float rounds it. +The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. + ## Consuming from Go This directory is also a Go module, `github.com/open-feature/spec/specification/assets/provider-tck`, whose only content is an `embed.FS` of the artifacts above. The Go conformance suite depends on it instead of vendoring a copy: a Go module ships as a zip of the VCS tree, in which a git submodule is only a gitlink, so an embed from a submodule would arrive empty for anyone running `go get`. The other languages build from a working tree and keep using the submodule; `go.mod` and `embed.go` are inert for them. From aa3196d40fa2b1df37b24d70dfef6109875eb2bd Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 12:21:27 +0200 Subject: [PATCH 16/63] docs: name the right flag in the canonical set's precision note The top-level comment attributed 2^53 - 1 to large-integer-flag, which resolves to 2^31 - 1; the 2^53 - 1 flag is huge-integer-flag. Signed-off-by: Simon Schrottner --- .../assets/provider-tck/flags/canonical-flags.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index ff73357d9..20bb9ef44 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -17,9 +17,9 @@ " * 'false-flag', 'zero-flag' and 'empty-string-flag' resolve to false, 0 and \"\". They are", " values, not absences. A backend that drops them, or a provider that treats them as", " missing, is what the falsy-value scenarios catch.", - " * 'large-integer-flag' resolves to 9007199254740991, which is 2^53 - 1: the largest integer", - " every language can represent exactly, including JavaScript. Seed it as an integer, not a", - " float, and do not round it.", + " * 'large-integer-flag' and 'huge-integer-flag' resolve to 2147483647 (2^31 - 1) and", + " 9007199254740991 (2^53 - 1). Seed both as integers, not floats, and do not round them: a", + " float32 round trip changes the first, and a 32-bit integer cannot hold the second.", " * 'integral-float-flag' resolves to 10.0, a float with no fractional part. It MUST be seeded", " as a float. A backend that stores it as the integer 10 makes the lossless-coercion scenario", " pass without coercing anything." From 667856acbbe69e23fe938247b5045b017e5a9488 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 12:47:21 +0200 Subject: [PATCH 17/63] fix: point the number-type quote at specification/types.md The link was written as ../types.md from a file that already sits in specification/, so it resolved outside the repo and failed link check. Made it a sibling link and anchored it to the Number section the quote comes from. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index a33242e6a..64a1263ae 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -187,7 +187,7 @@ skipped and surface the reason, not silently pass or silently omit them. does not define**, and readers should not mistake it for one that does. OpenFeature has a single numeric type, deliberately: `number` is -[*"a numeric value of unspecified type or size"*](../types.md), and implementation languages **may** +[*"a numeric value of unspecified type or size"*](./types.md#number), and implementation languages **may** further differentiate between integers and floating point numbers *"as idioms dictate"*. Both the client and provider requirements say "boolean, numeric, string, and structure" — one numeric type, not two. Typed-language SDKs take up that idiom and expose two accessors anyway, and at that point From bd60449bd79dcbd38493366ae70db43ef970c255 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 13:32:17 +0200 Subject: [PATCH 18/63] fix: name the falsy flags what the SDK suite already names them The canonical set called them 'false-flag', 'zero-flag' and 'empty-string-flag'. Appendix B's SDK suite, on main, already calls the same three flags 'boolean-zero-flag', 'integer-zero-flag' and 'string-zero-flag', with variants 'zero' and 'non-zero' -- and so does flagd-testbed, which serves that flag set to every SDK's e2e suite. So this appendix had invented a second vocabulary for a property the repository next door already had names for. That cost something real rather than being untidy. Running the Go flagd adoption against flagd-testbed fails four scenarios with FLAG_NOT_FOUND, because the backend serves the falsy flags under the established names and the suite asks for the new ones. The alternative fix was to add the new names to flagd-testbed, which would leave two fixtures for one property in the harness every SDK runs, forever. Ten of the thirteen canonical flags were already borrowed from that suite verbatim, which is the evidence that the divergence was accidental. The variants move with the keys, because a scenario asserts the variant name too: the three entries here are now byte-identical to their counterparts in assets/gherkin/test-flags.json, so a backend that can serve the SDK suite can serve this one without adding a flag. No scenario changes meaning. The falsy-value outline still resolves false, 0 and "" against defaults of true, 1 and 'fallback', and still asserts reason STATIC. Signed-off-by: Simon Schrottner --- specification/assets/provider-tck/README.md | 2 +- .../provider-tck/flags/canonical-flags.json | 28 +++++++++---------- .../provider-tck/gherkin/evaluation.feature | 6 ++-- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index f99f4860b..1dce5b06e 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -24,7 +24,7 @@ A feature file that evaluates `boolean-flag` is meaningless without the flag def - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. - **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. -- **`false-flag`, `zero-flag` and `empty-string-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. +- **`boolean-zero-flag`, `integer-zero-flag` and `string-zero-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. These names, and their `zero`/`non-zero` variants, are the ones [Appendix B's SDK suite](../gherkin/test-flags.json) already uses, so a backend serving that flag set already serves these. - **`integral-float-flag` is a float and `huge-integer-flag` is an integer.** Seeding `10.0` as `10` makes the lossless-coercion scenario pass without coercing; seeding `9007199254740991` through a float rounds it. The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index 20bb9ef44..1a880fb91 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -14,9 +14,9 @@ "", "Three more are easy to lose in translation, because a seeding step that 'cleans up' values", "destroys exactly what they test:", - " * 'false-flag', 'zero-flag' and 'empty-string-flag' resolve to false, 0 and \"\". They are", - " values, not absences. A backend that drops them, or a provider that treats them as", - " missing, is what the falsy-value scenarios catch.", + " * 'boolean-zero-flag', 'integer-zero-flag' and 'string-zero-flag' resolve to false, 0 and", + " \"\". They are values, not absences. A backend that drops them, or a provider that", + " treats them as missing, is what the falsy-value scenarios catch.", " * 'large-integer-flag' and 'huge-integer-flag' resolve to 2147483647 (2^31 - 1) and", " 9007199254740991 (2^53 - 1). Seed both as integers, not floats, and do not round them: a", " float32 round trip changes the first, and a 32-bit integer cannot hold the second.", @@ -84,32 +84,32 @@ }, "defaultVariant": "ten" }, - "false-flag": { + "boolean-zero-flag": { "$comment": "Resolves to false. The default in the scenario is true, so a provider that treats false as missing is caught.", "state": "ENABLED", "variants": { - "on": true, - "off": false + "zero": false, + "non-zero": true }, - "defaultVariant": "off" + "defaultVariant": "zero" }, - "zero-flag": { + "integer-zero-flag": { "$comment": "Resolves to 0. The default in the scenario is 1, so a provider that treats 0 as missing is caught.", "state": "ENABLED", "variants": { - "one": 1, - "zero": 0 + "zero": 0, + "non-zero": 1 }, "defaultVariant": "zero" }, - "empty-string-flag": { + "string-zero-flag": { "$comment": "Resolves to the empty string. The default in the scenario is 'fallback', so a provider that treats \"\" as missing is caught.", "state": "ENABLED", "variants": { - "greeting": "hi", - "empty": "" + "zero": "", + "non-zero": "str" }, - "defaultVariant": "empty" + "defaultVariant": "zero" }, "object-flag": { "state": "ENABLED", diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index a1b01da2d..15bfc7ec8 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -48,9 +48,9 @@ Feature: Provider flag evaluation Examples: | key | type | default | value | variant | - | false-flag | Boolean | true | false | off | - | zero-flag | Integer | 1 | 0 | zero | - | empty-string-flag | String | fallback | | empty | + | boolean-zero-flag | Boolean | true | false | zero | + | integer-zero-flag | Integer | 1 | 0 | zero | + | string-zero-flag | String | fallback | | zero | Scenario: A large integer resolves without loss of precision # 2147483647 is 2^31 - 1, the largest 32-bit signed integer, so every language's integer From 2dc0b251c29ecd32b6861c68108e9842d40910d7 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 13:57:22 +0200 Subject: [PATCH 19/63] feat: release the specification with release-please as well The config only knew about the conformance assets, so the specification's own vX.Y.Z tags stayed a manual step. With a versioned package now living in the repository that manual step is easy to forget, and the tag the ecosystem actually depends on is the one that would go missing. Adds a root package that keeps the existing tag shape: - include-component-in-tag is false, so it tags vX.Y.Z, not spec/vX.Y.Z. The assets set it true and keep their path-prefixed Go module tag. - exclude-paths drops the assets from it, so a commit confined to the suite bumps the suite and leaves the specification alone. - The manifest starts at 0.9.0, the current release, so numbering continues rather than restarting. - release-type simple, because the root package.json is private and carries no version; it tracks the version in version.txt. separate-pull-requests was already set, so the two release as two pull requests. Also drops monorepo-tags, which is not a manifest config key -- the schema rejects it and release-please ignores it. Component tagging was coming from the include-component-in-tag default all along, which is now explicit on both packages. Signed-off-by: Simon Schrottner --- .github/workflows/release-please.yaml | 16 +++++++++------- .release-please-manifest.json | 1 + release-please-config.json | 12 +++++++++++- specification/assets/provider-tck/README.md | 2 +- version.txt | 1 + 5 files changed, 23 insertions(+), 9 deletions(-) create mode 100644 version.txt diff --git a/.github/workflows/release-please.yaml b/.github/workflows/release-please.yaml index 3cf33b832..13362410a 100644 --- a/.github/workflows/release-please.yaml +++ b/.github/workflows/release-please.yaml @@ -1,13 +1,15 @@ name: Run Release Please -# Releases the versioned artifacts kept in this repository. It does NOT release the -# specification itself: the specification's own vX.Y.Z tags are cut by hand, and no root -# package is configured here, so release-please neither creates nor advances them. +# Releases everything versioned in this repository. Two packages are configured in +# release-please-config.json, each with its own release pull request: # -# Today the one package is specification/assets/provider-tck, the provider conformance -# assets. Those are consumed by four language suites which each have to pin a revision of -# them, and a pinned revision needs a name a human can read and a bot can compare. See -# release-please-config.json. +# . the specification, tagged vX.Y.Z as it always has been +# specification/assets/provider-tck the provider conformance assets, tagged with their path +# +# The two are independent. The specification package excludes the conformance assets, so a +# change confined to the suite never bumps the specification; the assets carry their own +# version because four language suites pin a revision of them, and a pinned revision needs a +# name a human can read and a bot can compare. on: push: diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 39a253925..8a76893c9 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -1,3 +1,4 @@ { + ".": "0.9.0", "specification/assets/provider-tck": "0.0.0" } diff --git a/release-please-config.json b/release-please-config.json index 56c67995c..a37bae492 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -1,13 +1,23 @@ { "$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json", - "monorepo-tags": true, "tag-separator": "/", "bootstrap-sha": "dd235837f7b2ddb1120ab6e1bde940bf4d532313", "separate-pull-requests": true, "packages": { + ".": { + "release-type": "simple", + "package-name": "spec", + "include-component-in-tag": false, + "changelog-path": "CHANGELOG.md", + "bump-minor-pre-major": true, + "versioning": "default", + "exclude-paths": ["specification/assets/provider-tck"], + "extra-files": [] + }, "specification/assets/provider-tck": { "release-type": "go", "package-name": "specification/assets/provider-tck", + "include-component-in-tag": true, "changelog-path": "CHANGELOG.md", "bump-minor-pre-major": true, "versioning": "default", diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 1dce5b06e..c17cea9d1 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -41,7 +41,7 @@ go get github.com/open-feature/spec/specification/assets/provider-tck@v0.1.0 ## Releases -These assets are released independently of the specification, by [release-please](../../../release-please-config.json). A nested Go module is tagged with its path as a prefix, so a release is tagged `specification/assets/provider-tck/vX.Y.Z` — the same shape as `providers/flagd/v0.6.0` in the SDK contrib repositories. The specification's own `vX.Y.Z` tags are cut by hand and do not apply here; nothing about the two numbering schemes is related. +These assets are released independently of the specification, by [release-please](../../../release-please-config.json). A nested Go module is tagged with its path as a prefix, so a release is tagged `specification/assets/provider-tck/vX.Y.Z` — the same shape as `providers/flagd/v0.6.0` in the SDK contrib repositories. The specification's own `vX.Y.Z` tags are cut by release-please too, from a separate release pull request that excludes this directory, and do not apply here; nothing about the two numbering schemes is related. What a bump means is not the usual thing, because this is a test suite rather than a library: diff --git a/version.txt b/version.txt new file mode 100644 index 000000000..ac39a106c --- /dev/null +++ b/version.txt @@ -0,0 +1 @@ +0.9.0 From 0d387e2d21a0a347e371e76b114d6fcdc5d52544 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 14:45:00 +0200 Subject: [PATCH 20/63] fix: gate reinitialisation, which 2.5.2 permits rather than requires The scenario "A provider that was shut down can be initialized again" was untagged, and so mandatory. Its own comment justified that: reverting to the uninitialized state "is observable as exactly one thing -- it can be initialized again and then serves flags". That inference does not hold. Requirement 2.5.2 says a provider SHOULD revert to its uninitialized state, and its supporting text says "some providers MAY allow reinitialization from this state". Reuse is permitted, not required. A provider that releases its client on shutdown and declines to start again is taking an option the specification offers it. The cost was not theoretical. Run against flagd, the scenario failed on the RPC resolver, was recorded as a known deviation, and was one step from being filed as a defect against a provider that is doing nothing wrong. A false failure is the mirror image of a vacuous pass, and this appendix already had a great deal to say about the latter. So it is gated on @reinitialization. Keeping it rather than deleting it is deliberate: a provider that does offer reuse has somewhere to be held to it, and "shutdown() releases the client, initialize() returns early because an initialised flag was never cleared" is easy to write and leaves the provider evaluating against a closed connection rather than failing outright. Reverting the state is not separately testable -- a provider that reverts but refuses reuse presents exactly as one that did neither -- so a gated reuse scenario is the only assertion the requirement admits. The appendix records how this was caught, because the method generalises: mapping each scenario back to a numbered requirement is what exposed it, and it is the third rule in this suite found to be asserted more strongly than the specification states it. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 28 +++++++++++++++++-- .../provider-tck/gherkin/lifecycle.feature | 19 ++++++++++--- 2 files changed, 41 insertions(+), 6 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 64a1263ae..0bb9ffa4d 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -153,6 +153,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | +| `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](./sections/02-providers.md#requirement-252) permits rather than requires | | `@targeting` | reserved; **not declarable** -- no scenarios yet | | `@caching` | reserved; **not declarable** -- no scenarios yet | @@ -179,9 +180,32 @@ HTTP provider such as OFREP is the common case: it observably emits nothing of i fail initialisation, yet its client still reports `READY`. Such a provider declares neither tag, and the lifecycle scenarios are reported as skipped rather than passing vacuously. +`@reinitialization` is separate from `@lifecycle` for a subtler reason, and it is worth recording how +it came to be separate. [Requirement 2.5.2](./sections/02-providers.md#requirement-252) says a +provider **SHOULD** revert to its uninitialized state after `shutdown`, and its supporting text adds +that *"some providers **may** allow reinitialization from this state"*. Reuse is therefore permitted, +not required. A provider that releases its client on shutdown and refuses to be started again is +exercising a choice the specification offers it. + +The scenario was originally untagged, on the reading that reverting to the uninitialized state "is +observable as exactly one thing — it can be initialized again and then serves flags". That inference +does not hold, and the cost of it was concrete: a provider making a permitted choice was reported as +failing conformance, and the failure was on its way to being filed as a defect against the +implementation. A false failure is the mirror image of a vacuous pass, and this appendix cares about +both. + +Reverting the state is not separately observable either — a provider that reverts but refuses reuse +presents exactly as one that did neither — so a gated reuse scenario is the only assertion the +requirement admits. It is worth keeping for the providers that do offer reuse, because releasing the +client on shutdown while leaving an initialised flag set is easy to write and leaves the provider +evaluating against a closed connection rather than failing outright. + The design rule behind this: **a conformance suite that quietly goes green on scenarios it did not -run is worse than no suite at all.** A TCK implementation must report unsupported capabilities as -skipped and surface the reason, not silently pass or silently omit them. +run is worse than no suite at all** — and, learned later and at some cost, one that reports a +permitted choice as a failure is not much better. A TCK implementation must report unsupported +capabilities as skipped and surface the reason, not silently pass or silently omit them; and a +scenario must be gated whenever the behaviour it asserts is one the specification allows a provider +to decline. `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**, and readers should not mistake it for one that does. diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature index 15eb735f8..635e5fd95 100644 --- a/specification/assets/provider-tck/gherkin/lifecycle.feature +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -54,11 +54,22 @@ Feature: Provider lifecycle And the provider is shut down Then no exception should have been thrown + @reinitialization Scenario: A provider that was shut down can be initialized again - # Requirement 2.5.2: after shutdown the provider reverts to its uninitialized state, which - # is observable as exactly one thing -- it can be initialized again and then serves flags. - # A provider that shuts down by discarding its client and never recreating it passes the - # scenario above and fails this one. + # Requirement 2.5.2 says a provider SHOULD revert to its uninitialized state after shutdown, + # and its supporting text says "some providers MAY allow reinitialization from this state". + # Reuse is therefore permitted, not required, and this scenario is gated accordingly: a + # provider that shuts down by discarding its client and never recreating it is making a + # choice the specification allows, not exhibiting a defect. + # + # What the tag buys is the other direction. A provider that does claim to be reusable has + # somewhere to be held to it, because "shutdown() releases the client and initialize() + # returns early because an initialised flag was never cleared" is easy to write and leaves + # the provider evaluating against a closed connection rather than failing outright. + # + # Reverting the state itself is not separately observable: a provider that reverts but + # refuses reuse presents exactly as one that did neither. So this is the only assertion the + # requirement admits, and it only applies where reuse is offered. Given a stable provider And a Boolean-flag with key "boolean-flag" and a default value "false" When the provider is shut down From 00578fc1e4d768027c44cb66edfcf3cbea5e06de Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 17:45:23 +0200 Subject: [PATCH 21/63] fix: state the rules four implementations answered differently Every divergence found while building the four language suites traced back to this appendix implying something instead of stating it. Each of the five below cost real time, and three of them produced a report or a bug report that was wrong. **Which path identifies a canonical feature.** The rule said canonical features "keep the path they have in this repository". That produced three answers: gherkin/errors.feature Go, Python specification/assets/provider-tck/gherkin/errors.feature JavaScript features/errors.feature Java JavaScript followed the sentence literally, which is the tell that the sentence is the defect. A consumer joining two languages' results keys on the uri and the scenario name, so today nothing joins across languages -- the partition this rule exists to guarantee is the thing that does not survive it. The rule now names the form: the path relative to this asset directory, and `extensions/` for extensions. Relative to the directory rather than the repository because these assets are also a released Go module, whose root *is* this directory: the embed keys inside it are `gherkin/*.feature`, and nothing in it knows or should know where it sits in a checkout. Comparison is on the path after any uri scheme, because Cucumber-JVM resolves from the classpath and reports `classpath:gherkin/errors.feature` -- the scheme belongs to the runner and to the results format, neither of which this appendix defines, and requiring an implementation to rewrite a foreign format to satisfy a rule of ours is the wrong trade. **That tags compose, and a scenario needs all of them.** `lifecycle.feature` carries `@lifecycle` at feature level, so the re-initialization scenario inside it needs `@lifecycle` and `@reinitialization` both. Declaring only the second leaves it skipped -- a declaration that looks satisfied and examines nothing. That was mistaken for a passing scenario, and a provider was described as supporting reuse on the strength of it. **That a capability is declared on evidence from running, never from reading the provider.** Source inspection was wrong in both directions on the same question: one provider reverts its initialised flag on shutdown, which reads as support, and fails on a deadline because the transport cannot restart; another closes its client with nothing visibly reconstructing it, which reads as a refusal, and works. **When a deviation is a deviation.** Twice a permitted choice was recorded as a defect against a provider, and one of those was nearly filed upstream. An entry is for a behaviour the provider is required to have and does not -- a numbered MUST, or a rule the implementation bound itself to elsewhere. Where the specification permits the choice, withholding the capability is itself the honest report. **That `knownDeviations` is omitted when empty.** An empty array asserts that deviations were considered and none found, which no suite can know on the adopter's behalf. One implementation emits nothing at all today, which is the opposite failure. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 54 +++++++++++++++++-- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 0bb9ffa4d..52368ae11 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -207,6 +207,39 @@ capabilities as skipped and surface the reason, not silently pass or silently om scenario must be gated whenever the behaviour it asserts is one the specification allows a provider to decline. +### Rules for declaring + +The four rules below are stated rather than implied because each was discovered by four +implementations answering the same question differently. They are what makes two reports comparable. + +**A scenario is gated by every capability tag that applies to it, including tags inherited from its +feature.** Tags compose: a tag on a `Feature` applies to every scenario in it, and a scenario +carrying its own tag is gated by both. A scenario runs only when **all** of its capabilities are +declared, and is otherwise reported as skipped naming one of the undeclared ones. This matters for +reading a result: `lifecycle.feature` carries `@lifecycle` at feature level, so the re-initialization +scenario inside it needs `@lifecycle` *and* `@reinitialization`, and declaring only the second leaves +it skipped — a declaration that looks satisfied and examines nothing. + +**Declare a capability only on evidence from running the suite, never from reading the provider's +source.** Source inspection is unreliable here in both directions, and demonstrably so: one +provider's shutdown explicitly reverts its own initialised flag, which reads as support for reuse, +while the transport underneath cannot be restarted and initialization fails on a deadline. Another +closes its client with nothing visibly reconstructing it, which reads as a refusal, and works. Run +the cycle. + +**A `knownDeviations` entry is for a behaviour the provider is required to have and does not.** The +requirement must be a numbered `MUST`, or a rule the implementation has bound itself to elsewhere — +a vendor's own architecture decision, say. Where the specification permits the choice, withholding +the capability *is* the honest report and a deviation entry would assert a defect that does not +exist. A false failure is the mirror image of a vacuous pass, and a reader cannot tell them apart +from the outside. When a scenario fails, find the numbered requirement before concluding anything: +check whether it is a `MUST`, a `SHOULD`, or explicitly optional. + +**Emit `knownDeviations` only when there is at least one.** An empty array and an absent field are +not the same claim: stating none asserts that deviations were considered and none found, which no +suite can know on the adopter's behalf. Omit the field when the list is empty, and never synthesise +an empty one. + `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**, and readers should not mistake it for one that does. @@ -308,9 +341,24 @@ keep an extension from quietly becoming a conformance claim. **Extension scenarios must be distinguishable from canonical ones.** A results payload that mixes them with no way to tell which is which lets an adopter's own passing scenarios flatter the -conformance result. Partitioning by path is enough — canonical features keep the path they have in -this repository and extensions mount under a reserved prefix — and it is what a consumer reads to -separate the two. +conformance result. Partitioning by path is enough, and it is what a consumer reads to separate the +two: + +- a canonical feature is identified by its path **relative to this asset directory** — + `gherkin/errors.feature`, not a path relative to the repository root; +- an extension mounts under the reserved prefix **`extensions/`**. + +Both are stated exactly because a phrasing that merely implied them produced three different answers +in four implementations. "The path it has in this repository" is the phrasing that did it, and it is +wrong in a way worth recording: these assets are also consumed as a released Go module, where the +module root *is* this directory and nothing inside it knows or should know where the directory sits +in a repository. `gherkin/errors.feature` is the only form every consumer can produce from what it +actually has. + +Compare on the **path component**, after any URI scheme. A runner that resolves features from a +classpath or a bundle legitimately reports `classpath:gherkin/errors.feature`; the scheme belongs to +the runner and the results format, neither of which this appendix defines, and stripping it is the +consumer's job rather than the implementation's. **An extension must not shadow a canonical scenario.** The same partition provides this: an extension file cannot occupy a canonical path, so it can add questions but never replace one. From 11c2ab4254aadd57bc8c2504f296c2e4887e07e1 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:07:35 +0200 Subject: [PATCH 22/63] fix: one skip with a reason, not a second way to say the same thing "Not declared" and "not applicable" are both skips. A results payload that gives them separate statuses, or a declaration that carries a parallel not-applicable field, asks four implementations to agree on more vocabulary without telling a reader anything the skip's own reason does not already say -- the scenario's tags say what was asked, the declaration says whether it was claimed, and the reason says why it was skipped. The two cases that motivated a separate field are both properties of an SDK rather than of a provider: @numeric-coercion cannot hold where the language has a single numeric type, and @large-integers cannot hold on a 32-bit accessor. Those are true of every provider written against that SDK, and for as long as the accessor is what it is. So this appendix records them, once, which is also what the field's own description in the report schema argued for -- it reserved itself for provider-specific impossibility, and no adoption in four languages has produced one. All four implementations built the field and none of their adoptions populate it. Removing it costs nothing and removes a thing they would otherwise have to keep in step. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 52368ae11..9f599c92a 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -279,7 +279,21 @@ and expects `TYPE_MISMATCH`; the lossless half asks for `integral-float-flag` (` and for `integer-flag` (`10`) as a float, and expects both to succeed. A provider declaring the tag must satisfy all three — rejecting every float is an easy way to pass the first, and the other two are what stop it. A provider whose SDK has a single numeric type, such as JavaScript, cannot -distinguish the cases and reports the tag as not applicable rather than declaring it. +distinguish the cases at all, so it leaves the tag undeclared and the scenarios are reported as +skipped with that reason. + +**Where a capability cannot hold in a language, this appendix is where that is recorded** — not a +field in every report. `@numeric-coercion` in a single-numeric-type language and `@large-integers` on +a 32-bit accessor are properties of the SDK, true of every provider written against it and for as +long as the accessor is what it is. Stating them here says it once; a per-report field would restate +a language fact on every provider's behalf, and would still say nothing in a run where no scenario +carried the tag. + +That leaves one skip, carrying its reason, as the whole mechanism. A results payload does not need a +second status to distinguish "undeclared" from "cannot apply": both are skips, the reason says which, +and the scenario's own tags say what was being asked. Splitting them into separate statuses, or into +a parallel declaration field, multiplies the vocabulary that four implementations have to agree on +without telling a reader anything the reason does not. **Accessor width** is the related property the ADR distinguishes, and it is modelled separately because it is a property of the SDK rather than of the provider. Every language's integer accessor From 75ecde2105a3cd0889ca343803a967e0a089ecba Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 20:20:47 +0200 Subject: [PATCH 23/63] fix: configure release-please with inputs the action actually declares Three of the four settings passed to googleapis/release-please-action@v5 are not inputs it declares, and GitHub Actions ignores an undeclared input without saying so: command: manifest removed after v3; v5 is manifest-driven already default-branch renamed to target-branch signoff moved into release-please-config.json The signoff one had teeth. This repository enforces DCO, so a release commit without a sign-off cannot merge -- and the setting that was meant to provide it was being dropped on the floor. The release pull request for the conformance assets would have been raised and then blocked, which is a slow way to find out, because nothing fails until the first release is attempted. Verified against the action's own action.yml: token, release-type, path, target-branch, config-file, manifest-file, repo-url, github-api-url, github-graphql-url, fork, include-component-in-tag, proxy-server, skip-github-release, skip-github-pull-request, skip-labeling, changelog-host, versioning-strategy, release-as. And against release-please's config schema, where signoff is a declared top-level key. Also adds issues: write, which release-please needs to manage its labels. Signed-off-by: Simon Schrottner --- .github/workflows/release-please.yaml | 10 ++++++---- release-please-config.json | 1 + 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/workflows/release-please.yaml b/.github/workflows/release-please.yaml index 13362410a..4b95c3259 100644 --- a/.github/workflows/release-please.yaml +++ b/.github/workflows/release-please.yaml @@ -10,6 +10,10 @@ name: Run Release Please # change confined to the suite never bumps the specification; the assets carry their own # version because four language suites pin a revision of them, and a pinned revision needs a # name a human can read and a bot can compare. +# +# This repository enforces DCO, so the bot's own commits need a sign-off. That is the signoff +# key in release-please-config.json: v5 takes it from the configuration file, and an action +# input of the same name is not declared and would be silently ignored. on: push: @@ -18,6 +22,7 @@ on: permissions: contents: write + issues: write pull-requests: write jobs: @@ -26,8 +31,5 @@ jobs: steps: - uses: googleapis/release-please-action@v5 with: - command: manifest token: ${{ secrets.GITHUB_TOKEN }} - default-branch: main - # This repository enforces DCO, so the bot's own commits need a sign-off too. - signoff: "OpenFeature Bot <109696520+openfeaturebot@users.noreply.github.com>" + target-branch: main diff --git a/release-please-config.json b/release-please-config.json index a37bae492..b346fefdf 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -1,5 +1,6 @@ { "$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json", + "signoff": "OpenFeature Bot <109696520+openfeaturebot@users.noreply.github.com>", "tag-separator": "/", "bootstrap-sha": "dd235837f7b2ddb1120ab6e1bde940bf4d532313", "separate-pull-requests": true, From 91d093bdcb174218e0503eb54ad390f236a5e0dc Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 22:16:19 +0200 Subject: [PATCH 24/63] ci: run pull request checks for pull requests based on feat/provider-tck* These filters match a pull request's BASE branch, and the report schema pull request is based on this one rather than on main. So it ran no checks at all -- not the linter, not the link checker, not the table-of-contents check -- while appearing green, which is a worse state than failing. Found by adding a section to the appendix on that branch and noticing that nothing had linted it. Temporary, and the comment says so: the filter comes out when the chain lands. Signed-off-by: Simon Schrottner --- .github/workflows/pr-checks.yaml | 5 +++++ .github/workflows/pr-python.yaml | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/.github/workflows/pr-checks.yaml b/.github/workflows/pr-checks.yaml index 47605cc26..9f6e78583 100644 --- a/.github/workflows/pr-checks.yaml +++ b/.github/workflows/pr-checks.yaml @@ -5,6 +5,11 @@ on: types: [opened, synchronize, reopened] branches: - main + # Temporary, for the duration of the provider conformance suite's review. + # This filter matches the pull request's BASE branch, so without it a pull + # request stacked on feat/provider-tck-appendix gets no checks at all. + # Remove once the chain has landed. See open-feature/spec#417. + - 'feat/provider-tck*' jobs: # This job ensures that any PR targeting the main branch has up-to-date generated JSON files. diff --git a/.github/workflows/pr-python.yaml b/.github/workflows/pr-python.yaml index ecfd0e5a3..18f3a33b7 100644 --- a/.github/workflows/pr-python.yaml +++ b/.github/workflows/pr-python.yaml @@ -4,6 +4,11 @@ on: pull_request: branches: - main + # Temporary, for the duration of the provider conformance suite's review. + # This filter matches the pull request's BASE branch, so without it a pull + # request stacked on feat/provider-tck-appendix gets no checks at all. + # Remove once the chain has landed. See open-feature/spec#417. + - 'feat/provider-tck*' permissions: contents: read From 178db5b7af8e4ac3d5379b5c9c69fd6eae3b99c6 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 23:06:07 +0200 Subject: [PATCH 25/63] fix: gate the variant assertions, which 2.2.4 makes a SHOULD Every evaluation scenario asserted a variant. That reads as obviously correct until a backend with no variant concept for a plain flag is put under test: its evaluation response carries no such key, so the provider never receives one and no seeding can produce one. Ten scenario instances failed a conformant provider for something its author cannot fix. Requirement 2.2.4 is a SHOULD -- a provider "SHOULD populate the resolution details structure's variant field" -- and types.md types the field as "variant (string, optional)". The same section adds that the value "might only be meaningful in the context of the flag management system associated with the provider". The suite was asserting a MUST that neither states, which is the error this appendix's own rules say to check for: look up the requirement and see whether it is a MUST, a SHOULD, or explicitly optional. Worse than the failures was that nothing could be said about them. A knownDeviations entry needs a capability to hang on, and there was none, so a provider author had no way to record that the gap was a property of the backend rather than a defect. That absence is the actual bug; the ten failures were a symptom. The variant assertions now live in one @variants scenario outline covering the eight flags that need no other capability. The value and reason assertions stay untagged, because 2.2.3 makes the value a MUST. A provider whose backend names its variants declares the tag and the rows run; one whose backend does not leaves it undeclared and they are skipped with that reason rather than passed. Found by running the suite against a Flagsmith provider, where ten of twelve failures were `variant was ""`. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 16 ++++- specification/assets/provider-tck/README.md | 2 +- .../provider-tck/gherkin/evaluation.feature | 63 ++++++++++++++----- 3 files changed, 61 insertions(+), 20 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 9f599c92a..9e598f482 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -150,6 +150,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `@object` | supports structured flag values | +| `@variants` | names the variant it resolved, which [Requirement 2.2.4](./sections/02-providers.md#requirement-224) makes a `SHOULD` and `types.md` types as optional | | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | @@ -235,6 +236,14 @@ exist. A false failure is the mirror image of a vacuous pass, and a reader canno from the outside. When a scenario fails, find the numbered requirement before concluding anything: check whether it is a `MUST`, a `SHOULD`, or explicitly optional. +`@variants` is the clearest case, and it was found the hard way. Every evaluation scenario asserted +a variant, which reads as obviously correct until a backend with no variant concept for a plain flag +is put under test: its evaluation response carries no such key, the provider never receives one, and +no seeding can produce one. Ten scenarios failed a conformant provider for something its author +could not fix, and nothing could be recorded as a known deviation because there was no capability to +hang one on. [Requirement 2.2.4](./sections/02-providers.md#requirement-224) is a `SHOULD` and +`types.md` types the field as optional; the suite was asserting a `MUST` neither of them states. + **Emit `knownDeviations` only when there is at least one.** An empty array and an absent field are not the same claim: stating none asserts that deviations were considered and none found, which no suite can know on the adopter's behalf. Omit the field when the list is empty, and never synthesise @@ -420,9 +429,10 @@ belong here rather than in any one implementation: - **Coverage of the numbered requirements.** Mapped against [the provider requirements](./sections/02-providers.md), leaving out 2.8.5.1 (it constrains the SDK) and 2.2.8.1 (a language-binding property, not observable at runtime), the suite covers 10 of the - 14 `MUST` requirements in scope, all 5 `SHOULD`, and 1 of 6 `MAY`. The `MUST` gaps are 2.3.1 (the - provider hook mechanism, a compile-time property in typed languages with little to observe at - runtime), 2.2.10 (flag metadata structure, blocked with 2.2.9 on the canonical flag set defining + 14 `MUST` requirements in scope, all 5 `SHOULD` — 2.2.4 only for a provider declaring `@variants` + — and 1 of 6 `MAY`. The `MUST` gaps are 2.3.1 (the provider hook mechanism, a compile-time + property in typed languages with little to observe at runtime), 2.2.10 (flag metadata structure, + blocked with 2.2.9 on the canonical flag set defining none), 2.4.4 (a domain-scoped provider accepts its bound domain, which is as much SDK as provider behaviour) and 2.8.4 (`PROVIDER_CONTEXT_CHANGED`). The last is the largest hole: context reconciliation is where a provider is most likely to serve values computed for the *previous* diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index c17cea9d1..bf8803c0d 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -8,7 +8,7 @@ These validate a **provider** against a real backend. For assets that validate a | Path | What it is | | --- | --- | -| [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value, variant and reason; falsy values; integer precision | +| [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value and reason; the variant where the backend names one; falsy values; integer precision | | [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix, numeric coercion and the unknown-flag case | | [`gherkin/events.feature`](./gherkin/events.feature) | configuration change, and the stale/ready transition across an outage | | [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one; shutdown | diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index 15bfc7ec8..2ffaa690d 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -15,22 +15,21 @@ Feature: Provider flag evaluation Background: Given a stable provider - Scenario Outline: Resolve values with variant and reason + Scenario Outline: Resolve values with reason Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" - And the variant should be "" And the reason should be "" And the error-code should be "" And the error message should be empty And no exception should have been thrown Examples: - | key | type | default | value | variant | reason | - | boolean-flag | Boolean | false | true | on | STATIC | - | string-flag | String | bye | hi | greeting | STATIC | - | integer-flag | Integer | 1 | 10 | ten | STATIC | - | float-flag | Float | 0.1 | 0.5 | half | STATIC | + | key | type | default | value | reason | + | boolean-flag | Boolean | false | true | STATIC | + | string-flag | String | bye | hi | STATIC | + | integer-flag | Integer | 1 | 10 | STATIC | + | float-flag | Float | 0.1 | 0.5 | STATIC | Scenario Outline: A falsy value is a value, not an absence # false, 0 and "" are the values most likely to be mistaken for "nothing came back": a @@ -40,17 +39,52 @@ Feature: Provider flag evaluation Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" - And the variant should be "" And the reason should be "STATIC" And the error-code should be "" And the error message should be empty And no exception should have been thrown Examples: - | key | type | default | value | variant | - | boolean-zero-flag | Boolean | true | false | zero | - | integer-zero-flag | Integer | 1 | 0 | zero | - | string-zero-flag | String | fallback | | zero | + | key | type | default | value | + | boolean-zero-flag | Boolean | true | false | + | integer-zero-flag | Integer | 1 | 0 | + | string-zero-flag | String | fallback | | + + @variants + Scenario Outline: The resolved details name the variant + # Gated, because a variant is optional rather than required. types.md declares the field + # "variant (string, optional)", and Requirement 2.2.4 is a SHOULD: in normal execution a + # provider "SHOULD populate the resolution details structure's variant field". The same + # section goes further and says the value "might only be meaningful in the context of the + # flag management system associated with the provider". + # + # Some systems have no variant concept for a plain flag at all. Their evaluation response + # carries no such key, so the provider never receives one and no amount of seeding can + # produce one. Asserting a variant in every scenario failed such a backend ten times over + # for something that is not a defect and that no provider author can fix — and left nothing + # to record as a known deviation, because there was no capability to hang one on. + # + # A provider whose backend names its variants declares this tag and these rows run. One + # whose backend does not leaves it undeclared, and they are skipped with that reason rather + # than passed. Either way the value and reason assertions above are unaffected: they are + # untagged, and 2.2.3 makes the value a MUST. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the variant should be "" + And the error-code should be "" + And the error message should be empty + And no exception should have been thrown + + Examples: + | key | type | default | variant | + | boolean-flag | Boolean | false | on | + | string-flag | String | bye | greeting | + | integer-flag | Integer | 1 | ten | + | float-flag | Float | 0.1 | half | + | boolean-zero-flag | Boolean | true | zero | + | integer-zero-flag | Integer | 1 | zero | + | string-zero-flag | String | fallback | zero | + | large-integer-flag | Integer | 1 | max-int32 | Scenario: A large integer resolves without loss of precision # 2147483647 is 2^31 - 1, the largest 32-bit signed integer, so every language's integer @@ -59,7 +93,6 @@ Feature: Provider flag evaluation Given a Integer-flag with key "large-integer-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "2147483647" - And the variant should be "max-int32" And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown @@ -78,7 +111,6 @@ Feature: Provider flag evaluation Given a Integer-flag with key "huge-integer-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "9007199254740991" - And the variant should be "max-safe" And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown @@ -106,8 +138,7 @@ Feature: Provider flag evaluation Scenario: Resolve a structured value Given a Object-flag with key "object-flag" and a default value "{}" When the flag was evaluated with details - Then the variant should be "template" - And the reason should be "STATIC" + Then the reason should be "STATIC" And the error-code should be "" And the error message should be empty And no exception should have been thrown From 9cb29f2788e52e4ea471ed4130e21576cad207f3 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 23:10:23 +0200 Subject: [PATCH 26/63] feat: test evaluation context, and make @targeting a real capability Nothing in the suite passed an evaluation context. Requirement 2.2.1 makes it a parameter of every resolve method, so a provider that threw on any context, or serialised it into a malformed request, passed every scenario. One mandatory scenario now supplies a targeting key to a flag with no rule and asserts the resolution is undisturbed, which closes that half of 2.2.1. The rest is what @targeting was reserved for. One new flag carries the only targeting rule in the set, and the rule is specified by its behaviour rather than by a syntax: resolve variant "targeted" when the evaluation context's targeting key is exactly "tck-targeted-user", "untargeted" otherwise. Express it however your backend expresses targeting -- the same latitude the flag set already gives for values, where the format "is not what matters". That turns out to solve a problem this appendix had filed as needing new API surface. The passthrough open question asked for an echo operation, something like GET /last-evaluation, on the grounds that a provider silently dropping the context would otherwise pass. That is only true while every flag resolves the same way whatever the context. A flag that resolves differently for a matching context is caught by the resolved value itself, so no echo operation is needed for the basic case. What remains unverified is narrower and now stated as such: that the whole context arrives, not just the targeting key. Three scenarios rather than one, because the matching case alone would pass for a provider that always served the targeted variant: a non-matching context and an absent context must both resolve the default. The scenarios assert values and the absence of an error, not reasons. 2.2.3 makes the value a MUST and 2.2.6 forbids an error code in normal execution, whereas 2.2.5 is a SHOULD that explicitly permits "some other string" -- and with a context supplied and nothing matching, both STATIC and DEFAULT are defensible. The flag's variant names double as its values so the scenarios do not depend on @variants either. The invariant that no flag has targeting rules is now stated as the exception it became, in all three places it appeared, because every untargeted scenario's STATIC expectation still rests on it. The `targeting` member is inert for the in-memory decoders, which read only state, variants and defaultVariant, so an in-memory provider leaves @targeting undeclared and these scenarios skip. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 36 ++++++---- specification/assets/provider-tck/README.md | 2 +- .../provider-tck/flags/canonical-flags.json | 28 ++++++++ .../provider-tck/gherkin/evaluation.feature | 67 ++++++++++++++++++- 4 files changed, 115 insertions(+), 18 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 9e598f482..a9eb4ee74 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -55,9 +55,11 @@ claims, and a language needs both suites to make both. **Out of scope:** -- **Backend evaluation logic**, targeting and bucketing correctness. Every flag in the canonical set - resolves to its default variant with no targeting involved, so what is under test is the - provider's mapping of a response, not the backend's decision. +- **Backend evaluation logic**, bucketing and rule-language correctness. Every flag in the canonical + set except `targeted-flag` resolves to its default variant whatever the context, so what is under + test is the provider's mapping of a response, not the backend's decision. `targeted-flag` carries + the one rule, and it is there to prove the context reached the backend rather than to test how the + backend evaluated it — which is why the rule is stated as behaviour and not as a syntax. - **The provider↔backend wire protocol.** How a provider talks to its backend is its own business. - **SDK behaviour.** That is Appendix B. @@ -104,8 +106,9 @@ Two properties are load-bearing and easy to break by accident: - **`missing-flag` must not exist.** Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. -- **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests - the provider's mapping of a response, not the backend's decision. +- **Only `targeted-flag` has a targeting rule.** Every other flag resolves to its default variant + whatever the evaluation context, which is what lets the untargeted scenarios expect reason + `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. ### The control API @@ -155,7 +158,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](./sections/02-providers.md#requirement-252) permits rather than requires | -| `@targeting` | reserved; **not declarable** -- no scenarios yet | +| `@targeting` | resolves a flag differently for a matching evaluation context | | `@caching` | reserved; **not declarable** -- no scenarios yet | Untagged scenarios are mandatory and always run. @@ -164,13 +167,14 @@ A reserved tag is documented so the vocabulary has a place for the capability wh but it **must not be declared** and must not appear in a conformance report's declaration. No scenario carries it, so declaring it cannot be verified, cannot produce a skip, and tells a reader only that something was claimed and nothing examined -- the vacuous conformance claim this whole -vocabulary exists to prevent. +vocabulary exists to prevent. `@caching` is the only reserved tag left. This is easy to reintroduce by accident rather than by intent. An adopter who declares "every capability except X" picks up every reserved tag on the way past, which is exactly how one -implementation came to report `@targeting` and `@caching` as declared. An implementation offering a -"declare everything" convenience should exclude reserved tags from it, and should tell an adopter who -names one directly rather than passing it silently into a report. +implementation came to report `@targeting` and `@caching` as declared -- back when both were +reserved. An implementation offering a "declare everything" convenience should exclude reserved tags +from it, and should tell an adopter who names one directly rather than passing it silently into a +report. `@lifecycle` and `@events` are deliberately separate, and conflating them is the mistake this vocabulary exists to prevent. Every SDK synthesises `PROVIDER_READY` for a provider that has no @@ -418,10 +422,14 @@ flagd provider for both its RPC and in-process resolvers. It is in review alongs This appendix is a proof of concept. Known gaps, all of which affect every language equally and so belong here rather than in any one implementation: -- **Evaluation context passthrough.** The scenarios build evaluation contexts but cannot assert the - context *reached* the backend intact. That needs an echo operation on the control API — something - like `GET /last-evaluation` returning the request the backend last received. Until then a provider - that silently drops the context passes. The `@targeting` tag is reserved for these scenarios. +- **Evaluation context passthrough, beyond the targeting key.** `targeted-flag` resolves differently + for a matching context, so a provider that drops the context is caught by the resolved value + itself — no echo operation needed for the basic case, which is how the `@targeting` scenarios + work. What is still unverified is that the *whole* context arrives intact: a provider that + forwards the targeting key and silently discards every other attribute passes. Asserting that + needs either an echo operation on the control API, something like `GET /last-evaluation` returning + the request the backend last received, or a second flag whose rule keys on a custom attribute. + The latter is cheaper and worth doing first, since attributes are where dropping is most likely. - **Setting and removing individual flags.** The control API can reset to a baseline and mutate one designated flag. Finer-grained flag manipulation would need new endpoints. - **Caching.** Whether a stale provider keeps serving last-known values during an outage depends on diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index bf8803c0d..01f0dc2cd 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -23,7 +23,7 @@ A feature file that evaluates `boolean-flag` is meaningless without the flag def ## Four properties that are load-bearing - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. -- **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. +- **Only `targeted-flag` has a targeting rule.** Every other flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. `targeted-flag`'s rule is specified by behaviour — resolve `targeted` when the targeting key is exactly `tck-targeted-user`, `untargeted` otherwise — so express it however your backend expresses targeting. - **`boolean-zero-flag`, `integer-zero-flag` and `string-zero-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. These names, and their `zero`/`non-zero` variants, are the ones [Appendix B's SDK suite](../gherkin/test-flags.json) already uses, so a backend serving that flag set already serves these. - **`integral-float-flag` is a float and `huge-integer-flag` is an integer.** Seeding `10.0` as `10` makes the lossless-coercion scenario pass without coercing; seeding `9007199254740991` through a float rounds it. diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index 1a880fb91..6410e7d5b 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -132,6 +132,34 @@ }, "defaultVariant": "one" }, + "targeted-flag": { + "$comment": [ + "The one flag in this set with a targeting rule. Everything else resolves to its", + "defaultVariant regardless of context, which is what lets every other scenario expect", + "reason STATIC.", + "", + "The rule is specified by its behaviour, not by this encoding: resolve variant", + "'targeted' when the evaluation context's targeting key is exactly 'tck-targeted-user',", + "and 'untargeted' otherwise. Express that however your backend expresses targeting.", + "", + "It is what makes context passthrough observable. A matching context resolves to a", + "different variant and value, so a provider that drops the context on the floor is", + "caught by the resolved value itself rather than needing an echo endpoint." + ], + "state": "ENABLED", + "variants": { + "untargeted": "untargeted", + "targeted": "targeted" + }, + "defaultVariant": "untargeted", + "targeting": { + "if": [ + { "==": [{ "var": "targetingKey" }, "tck-targeted-user"] }, + "targeted", + null + ] + } + }, "changing-flag": { "$comment": [ "The flag POST /change mutates. The TCK asserts only that its resolved value differs", diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index 2ffaa690d..d760aa022 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -2,9 +2,10 @@ Feature: Provider flag evaluation # Verifies that a provider maps backend responses onto typed resolution details correctly. # - # This does NOT test the backend's evaluation logic. Every flag in the canonical set resolves - # to its default variant with no targeting involved, so what is under test is purely the - # provider's mapping of a backend response to a value, a variant and a reason. + # This does NOT test the backend's evaluation logic. Every flag in the canonical set except + # targeted-flag resolves to its default variant whatever the context, so what is under test is + # purely the provider's mapping of a backend response to a value, a variant and a reason. + # targeted-flag carries the one rule, and only to show the context reached the backend. # # Every success path also asserts that no error message was set (requirement 2.3.2). A # provider that reports a value AND an error message is sending two contradictory signals, @@ -147,3 +148,63 @@ Feature: Provider flag evaluation | showImages | Boolean | true | | title | String | Check out these pics! | | imagesPerPage | Integer | 100 | + Scenario: Supplying an evaluation context does not disturb an untargeted resolution + # Mandatory, and the only scenario that passes a context to a provider with no targeting + # involved. Requirement 2.2.1 makes the evaluation context a parameter of every resolve + # method, but until this scenario existed no scenario supplied one — so a provider that + # threw on any context, or serialised it into a malformed request, passed the whole suite. + # + # string-flag has no targeting rule, so the context cannot change the outcome. What is + # under test is only that supplying one is harmless. + # + # Deliberately asserts the value and the absence of an error rather than the reason. + # 2.2.3 makes the value a MUST and 2.2.6 forbids an error code in normal execution, while + # the reason is a SHOULD that 2.2.5 lets a provider populate with "some other string" — + # and with a context supplied and nothing matching, both "STATIC" and "DEFAULT" are + # defensible readings. + Given a String-flag with key "string-flag" and a default value "bye" + And an evaluation context with targeting key "tck-other-user" + When the flag was evaluated with details + Then the resolved details value should be "hi" + And the error-code should be "" + And the error message should be empty + And no exception should have been thrown + + @targeting + Scenario: A matching evaluation context resolves the targeted variant + # This is what makes context passthrough observable. Every other flag resolves the same way + # whatever the context, so a provider that drops the context entirely passes them all. Here + # a matching context resolves to a different value, so dropping it is caught by the resolved + # value itself — no echo endpoint on the control API required. + # + # targeted-flag's rule is specified by behaviour, not by syntax: resolve "targeted" when the + # targeting key is exactly "tck-targeted-user", "untargeted" otherwise. Express it however + # your backend expresses targeting. + Given a String-flag with key "targeted-flag" and a default value "fallback" + And an evaluation context with targeting key "tck-targeted-user" + When the flag was evaluated with details + Then the resolved details value should be "targeted" + And the error-code should be "" + And no exception should have been thrown + + @targeting + Scenario: A non-matching evaluation context resolves the default variant + # Paired with the scenario above, and the reason it is not enough on its own: a provider + # that always returned the targeted value would pass that one. This is what pins down that + # the rule was evaluated rather than the targeted variant simply being served. + Given a String-flag with key "targeted-flag" and a default value "fallback" + And an evaluation context with targeting key "tck-other-user" + When the flag was evaluated with details + Then the resolved details value should be "untargeted" + And the error-code should be "" + And no exception should have been thrown + + @targeting + Scenario: No evaluation context resolves the default variant + # A targeting rule that cannot match must not error. A provider that requires a targeting + # key, or that fails to evaluate a rule when the context is absent, is caught here. + Given a String-flag with key "targeted-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "untargeted" + And the error-code should be "" + And no exception should have been thrown From a2bbda4ed47efefac4c51aca5c7f0bcad7511bda Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 23:13:06 +0200 Subject: [PATCH 27/63] fix: use Appendix B's wording for the targeting-key step I had invented "an evaluation context with targeting key". Appendix B's SDK suite already says "a context containing a targeting key with value", and the Java TCK already carries a step definition matching it -- dead until now, because no scenario used it. Two step phrasings for one idea is the divergence this appendix exists to prevent, and it would have cost four implementations a new step definition each for a step one of them had already written. Signed-off-by: Simon Schrottner --- .../assets/provider-tck/gherkin/evaluation.feature | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index d760aa022..214475712 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -157,13 +157,17 @@ Feature: Provider flag evaluation # string-flag has no targeting rule, so the context cannot change the outcome. What is # under test is only that supplying one is harmless. # + # The step wording is Appendix B's, and the Java TCK already carries a step definition for + # it, because inventing a second way to say "a context containing a targeting key" is the + # kind of divergence this appendix exists to prevent. + # # Deliberately asserts the value and the absence of an error rather than the reason. # 2.2.3 makes the value a MUST and 2.2.6 forbids an error code in normal execution, while # the reason is a SHOULD that 2.2.5 lets a provider populate with "some other string" — # and with a context supplied and nothing matching, both "STATIC" and "DEFAULT" are # defensible readings. Given a String-flag with key "string-flag" and a default value "bye" - And an evaluation context with targeting key "tck-other-user" + And a context containing a targeting key with value "tck-other-user" When the flag was evaluated with details Then the resolved details value should be "hi" And the error-code should be "" @@ -181,7 +185,7 @@ Feature: Provider flag evaluation # targeting key is exactly "tck-targeted-user", "untargeted" otherwise. Express it however # your backend expresses targeting. Given a String-flag with key "targeted-flag" and a default value "fallback" - And an evaluation context with targeting key "tck-targeted-user" + And a context containing a targeting key with value "tck-targeted-user" When the flag was evaluated with details Then the resolved details value should be "targeted" And the error-code should be "" @@ -193,7 +197,7 @@ Feature: Provider flag evaluation # that always returned the targeted value would pass that one. This is what pins down that # the rule was evaluated rather than the targeted variant simply being served. Given a String-flag with key "targeted-flag" and a default value "fallback" - And an evaluation context with targeting key "tck-other-user" + And a context containing a targeting key with value "tck-other-user" When the flag was evaluated with details Then the resolved details value should be "untargeted" And the error-code should be "" From ec2cfd738b9a75aac3139a0ee5c7d006e82d34a0 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 23:17:50 +0200 Subject: [PATCH 28/63] fix: use flagd-testbed's targeting-key-flag, and state the reason narrowing I invented targeted-flag. flagd-testbed already serves targeting-key-flag with the same rule shape -- variants hit and miss, defaultVariant miss, a rule on the targeting key -- exercised by its own targeting.feature and present in both its flag sets. Adopting it instead of a new flag is the same reasoning that named the falsy flags after Appendix B's: a backend serving that harness already serves this one, so the canonical set costs it nothing. It also removes an upstream dependency this change would otherwise have had, since nothing needs adding to the testbed at all. Its own scenario settles a question these scenarios had to guess at. flagd reports TARGETING_MATCH for the hit and DEFAULT for the miss, so with a rule present but unmatched there is no single reason to assert. Had these scenarios pinned STATIC, as every other evaluation scenario does, the non-matching one would have failed against flagd. They assert values, which carry the whole signal. That leaves the reason assertions elsewhere as a narrowing of the specification, and this states it rather than leaving it to be discovered. 2.2.5 is a SHOULD that explicitly permits "some other string indicating the semantic reason", and the suite pins an exact value in thirty-one scenario instances. It stays that way for now because the reason is the cheapest diagnosis of silent fallback, and because gating it would mean a second capability that nearly every provider would declare anyway. What is written down is that a provider failed by it is not thereby defective, and that the fix is to revisit this decision rather than to record a deviation against that provider. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 27 ++++++++++++-- specification/assets/provider-tck/README.md | 2 +- .../provider-tck/flags/canonical-flags.json | 28 ++++++++------ .../provider-tck/gherkin/evaluation.feature | 37 +++++++++++-------- 4 files changed, 62 insertions(+), 32 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index a9eb4ee74..1ce5e53a5 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -56,8 +56,8 @@ claims, and a language needs both suites to make both. **Out of scope:** - **Backend evaluation logic**, bucketing and rule-language correctness. Every flag in the canonical - set except `targeted-flag` resolves to its default variant whatever the context, so what is under - test is the provider's mapping of a response, not the backend's decision. `targeted-flag` carries + set except `targeting-key-flag` resolves to its default variant whatever the context, so what is + under test is the provider's mapping of a response, not the backend's decision. It carries the one rule, and it is there to prove the context reached the backend rather than to test how the backend evaluated it — which is why the rule is stated as behaviour and not as a syntax. - **The provider↔backend wire protocol.** How a provider talks to its backend is its own business. @@ -106,7 +106,7 @@ Two properties are load-bearing and easy to break by accident: - **`missing-flag` must not exist.** Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. -- **Only `targeted-flag` has a targeting rule.** Every other flag resolves to its default variant +- **Only `targeting-key-flag` has a targeting rule.** Every other flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. @@ -240,6 +240,25 @@ exist. A false failure is the mirror image of a vacuous pass, and a reader canno from the outside. When a scenario fails, find the numbered requirement before concluding anything: check whether it is a `MUST`, a `SHOULD`, or explicitly optional. +**The `reason` field is the deliberate exception, and it is stated here so it is a decision rather +than an oversight.** [Requirement 2.2.5](./sections/02-providers.md#requirement-225) is also a +`SHOULD`, and it goes further than 2.2.4 does: it lets a provider populate the field with one of the +listed values *"or some other string indicating the semantic reason for the returned flag value"*. +The suite nonetheless requires a reason, and requires a specific one, in every scenario that asserts +it. A provider whose backend reports vendor-specific reason strings will fail those scenarios. + +That is a narrowing of the specification, and it is accepted for now because the reason is the +suite's cheapest diagnosis of a whole class of silent failure: a provider that quietly falls back to +the code default reports a different reason, and the assertion names the problem where a value +assertion alone only says the number was wrong. Gating it would mean a second capability, a second +set of scenarios to keep in step, and a declaration nearly every provider would make anyway. + +A reader comparing reports should therefore treat a reason failure differently from a value failure: +the value assertions rest on `MUST` requirements, the reason assertions rest on a house rule. If a +conformant provider is failed by one, that is this suite's narrowing and not that provider's defect +-- and the right response is to revisit this decision, not to record a deviation against the +provider. + `@variants` is the clearest case, and it was found the hard way. Every evaluation scenario asserted a variant, which reads as obviously correct until a backend with no variant concept for a plain flag is put under test: its evaluation response carries no such key, the provider never receives one, and @@ -422,7 +441,7 @@ flagd provider for both its RPC and in-process resolvers. It is in review alongs This appendix is a proof of concept. Known gaps, all of which affect every language equally and so belong here rather than in any one implementation: -- **Evaluation context passthrough, beyond the targeting key.** `targeted-flag` resolves differently +- **Evaluation context passthrough, beyond the targeting key.** `targeting-key-flag` resolves differently for a matching context, so a provider that drops the context is caught by the resolved value itself — no echo operation needed for the basic case, which is how the `@targeting` scenarios work. What is still unverified is that the *whole* context arrives intact: a provider that diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 01f0dc2cd..b57f563cf 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -23,7 +23,7 @@ A feature file that evaluates `boolean-flag` is meaningless without the flag def ## Four properties that are load-bearing - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. -- **Only `targeted-flag` has a targeting rule.** Every other flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. `targeted-flag`'s rule is specified by behaviour — resolve `targeted` when the targeting key is exactly `tck-targeted-user`, `untargeted` otherwise — so express it however your backend expresses targeting. +- **Only `targeting-key-flag` has a targeting rule.** Every other flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. Its rule is specified by behaviour — resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`, `miss` otherwise — so express it however your backend expresses targeting. The flag, its variants and the uuid are the ones [flagd-testbed's `targeting.feature`](https://github.com/open-feature/flagd-testbed) already uses, on the same reasoning as the zero flags: a backend serving that harness already serves this. - **`boolean-zero-flag`, `integer-zero-flag` and `string-zero-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. These names, and their `zero`/`non-zero` variants, are the ones [Appendix B's SDK suite](../gherkin/test-flags.json) already uses, so a backend serving that flag set already serves these. - **`integral-float-flag` is a float and `huge-integer-flag` is an integer.** Seeding `10.0` as `10` makes the lossless-coercion scenario pass without coercing; seeding `9007199254740991` through a float rounds it. diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index 6410e7d5b..429005b0b 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -132,30 +132,34 @@ }, "defaultVariant": "one" }, - "targeted-flag": { + "targeting-key-flag": { "$comment": [ "The one flag in this set with a targeting rule. Everything else resolves to its", - "defaultVariant regardless of context, which is what lets every other scenario expect", + "defaultVariant whatever the context, which is what lets every other scenario expect", "reason STATIC.", "", - "The rule is specified by its behaviour, not by this encoding: resolve variant", - "'targeted' when the evaluation context's targeting key is exactly 'tck-targeted-user',", - "and 'untargeted' otherwise. Express that however your backend expresses targeting.", + "The rule is specified by its behaviour, not by this encoding: resolve variant 'hit'", + "when the evaluation context's targeting key is exactly the uuid below, and 'miss'", + "otherwise. Express that however your backend expresses targeting.", + "", + "Name, variants and uuid are the ones flagd-testbed's own targeting.feature already", + "uses, on the same reasoning as the zero flags: a backend serving that harness already", + "serves this, so adopting the canonical set costs it nothing.", "", "It is what makes context passthrough observable. A matching context resolves to a", - "different variant and value, so a provider that drops the context on the floor is", - "caught by the resolved value itself rather than needing an echo endpoint." + "different value, so a provider that drops the context on the floor is caught by the", + "resolved value itself rather than needing an echo endpoint." ], "state": "ENABLED", "variants": { - "untargeted": "untargeted", - "targeted": "targeted" + "miss": "miss", + "hit": "hit" }, - "defaultVariant": "untargeted", + "defaultVariant": "miss", "targeting": { "if": [ - { "==": [{ "var": "targetingKey" }, "tck-targeted-user"] }, - "targeted", + { "==": [{ "var": "targetingKey" }, "5c3d8535-f81a-4478-a6d3-afaa4d51199e"] }, + "hit", null ] } diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index 214475712..de50b9597 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -3,9 +3,10 @@ Feature: Provider flag evaluation # Verifies that a provider maps backend responses onto typed resolution details correctly. # # This does NOT test the backend's evaluation logic. Every flag in the canonical set except - # targeted-flag resolves to its default variant whatever the context, so what is under test is - # purely the provider's mapping of a backend response to a value, a variant and a reason. - # targeted-flag carries the one rule, and only to show the context reached the backend. + # targeting-key-flag resolves to its default variant whatever the context, so what is under + # test is purely the provider's mapping of a backend response to a value, a variant and a + # reason. targeting-key-flag carries the one rule, only to show the context reached the + # backend. # # Every success path also asserts that no error message was set (requirement 2.3.2). A # provider that reports a value AND an error message is sending two contradictory signals, @@ -167,7 +168,7 @@ Feature: Provider flag evaluation # and with a context supplied and nothing matching, both "STATIC" and "DEFAULT" are # defensible readings. Given a String-flag with key "string-flag" and a default value "bye" - And a context containing a targeting key with value "tck-other-user" + And a context containing a targeting key with value "f20bd32d-703b-48b6-bc8e-79d53c85134a" When the flag was evaluated with details Then the resolved details value should be "hi" And the error-code should be "" @@ -181,13 +182,19 @@ Feature: Provider flag evaluation # a matching context resolves to a different value, so dropping it is caught by the resolved # value itself — no echo endpoint on the control API required. # - # targeted-flag's rule is specified by behaviour, not by syntax: resolve "targeted" when the - # targeting key is exactly "tck-targeted-user", "untargeted" otherwise. Express it however - # your backend expresses targeting. - Given a String-flag with key "targeted-flag" and a default value "fallback" - And a context containing a targeting key with value "tck-targeted-user" + # targeting-key-flag's rule is specified by behaviour, not by syntax: resolve "hit" when the + # targeting key is exactly this uuid, "miss" otherwise. Express it however your backend + # expresses targeting. The flag, its variants and the uuid are flagd-testbed's own, so a + # backend serving that harness already serves this one. + # + # Asserts the value, not the reason. Elsewhere the suite does pin the reason, deliberately + # — see Appendix F. Here it cannot: flagd reports TARGETING_MATCH for this hit and DEFAULT + # for the miss below, and with a rule present but unmatched both readings are right, so + # there is no single value to pin. The resolved value carries the whole signal anyway. + Given a String-flag with key "targeting-key-flag" and a default value "fallback" + And a context containing a targeting key with value "5c3d8535-f81a-4478-a6d3-afaa4d51199e" When the flag was evaluated with details - Then the resolved details value should be "targeted" + Then the resolved details value should be "hit" And the error-code should be "" And no exception should have been thrown @@ -196,10 +203,10 @@ Feature: Provider flag evaluation # Paired with the scenario above, and the reason it is not enough on its own: a provider # that always returned the targeted value would pass that one. This is what pins down that # the rule was evaluated rather than the targeted variant simply being served. - Given a String-flag with key "targeted-flag" and a default value "fallback" - And a context containing a targeting key with value "tck-other-user" + Given a String-flag with key "targeting-key-flag" and a default value "fallback" + And a context containing a targeting key with value "f20bd32d-703b-48b6-bc8e-79d53c85134a" When the flag was evaluated with details - Then the resolved details value should be "untargeted" + Then the resolved details value should be "miss" And the error-code should be "" And no exception should have been thrown @@ -207,8 +214,8 @@ Feature: Provider flag evaluation Scenario: No evaluation context resolves the default variant # A targeting rule that cannot match must not error. A provider that requires a targeting # key, or that fails to evaluate a rule when the context is absent, is caught here. - Given a String-flag with key "targeted-flag" and a default value "fallback" + Given a String-flag with key "targeting-key-flag" and a default value "fallback" When the flag was evaluated with details - Then the resolved details value should be "untargeted" + Then the resolved details value should be "miss" And the error-code should be "" And no exception should have been thrown From 7f0ca75959ff9e8151713955f369e57ac598de41 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 08:44:12 +0200 Subject: [PATCH 29/63] docs: stop pinning the canonical set's size in an anecdote The extension rule recounted a real incident -- a selector matching one scenario name produced a green suite and a well-formed report -- and gave the set size as twenty-nine. That was true when it was written. Three count changes later it reads as a claim about the current set, and Java's TCK now counts fifty-two scenario instances against twenty-seven definitions, so any number here is a third place for the same fact to drift. The anecdote's force never depended on the figure: one scenario out of the whole set is the point. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 1ce5e53a5..df3509a88 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -416,7 +416,8 @@ of your own made the canonical ones optional would defeat the point of having a **A run that did not execute the canonical set in full must fail.** This is the one that needs stating because it is not obvious, and because it was found by accident rather than by design: a test selector matching a single scenario name produced a green suite and a **well-formed conformance -report describing one scenario out of twenty-nine**. A mis-wired extension filesystem does the same. +report describing one scenario out of the whole canonical set**. A mis-wired extension +filesystem does the same. There is no field in the report a consumer could read to notice — the schema is closed and the envelope carries no expected count — so failing the run is the only lever the implementation has. From d1b66720b8e84485a5027fcb102d06ca40df8ffc Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 12:01:58 +0200 Subject: [PATCH 30/63] feat: test disabled flags, gated, because the answer depends on architecture A flag disabled in the management system resolves to nothing and the caller's default stands in. No scenario covered it. Appendix B does, for SDKs, and gates it behind @reason-codes-disabled -- and the only trace in the provider section is DISABLED appearing in 2.2.5's list of reason strings. Requirement 1.4.7 is about the SDK propagating whatever reason arrived, not about what a provider owes a disabled flag. So this appendix states the behaviour, as it does for @numeric-coercion, and gates it. Gated rather than mandatory because the substitution happens in different places. A provider that evaluates locally -- flagd's two resolvers, an in-memory provider -- can return the value the caller passed in. A provider whose backend decides, such as one speaking OFREP, cannot: the server never sees the caller's default. The same flag cannot behave the same way across those two, and neither is wrong. That is what a capability is for. The scenarios assert the value and the absence of an error, not the reason. Each row's caller default differs from the flag's configured value, so a provider that ignores the state returns the configured value and fails on the value alone, which rests on 2.2.3, a MUST. Pinning reason DISABLED would rest on 2.2.5, a SHOULD that permits "some other string", and would be the seventh place this suite asserts more than the specification states. No variant is asserted: a disabled flag has resolved no variant, so @disabled-flags and @variants deliberately do not compose. There is no disabled-object-flag for the same reason in reverse -- an Object row would need @object as well, and one row of an outline cannot carry its own tag. Four flags, mirroring boolean-flag, string-flag, integer-flag and float-flag and differing only in state. Names, variants and values are flagd-testbed's own from flags/disabled-flags.json, which the launchpad already serves, so nothing needs adding upstream -- the same reasoning that made targeting-key-flag free. Two statements this falsifies are corrected: the canonical set no longer has every flag resolving to its default variant, and ENABLED is no longer universal. Both are now stated as the load-bearing properties they are, in the appendix and in the asset README. Scenario instances go 52 to 56. Note for the language suites: a test that iterates the packaged flags and asserts each resolves to its own defaultVariant now has four exceptions. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 11 +++-- specification/assets/provider-tck/README.md | 3 +- .../provider-tck/flags/canonical-flags.json | 45 +++++++++++++++++++ .../provider-tck/gherkin/evaluation.feature | 36 +++++++++++++++ 4 files changed, 91 insertions(+), 4 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index df3509a88..03a51fa84 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -106,9 +106,13 @@ Two properties are load-bearing and easy to break by accident: - **`missing-flag` must not exist.** Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. -- **Only `targeting-key-flag` has a targeting rule.** Every other flag resolves to its default variant - whatever the evaluation context, which is what lets the untargeted scenarios expect reason - `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. +- **Only `targeting-key-flag` has a targeting rule.** Every other enabled flag resolves to its + default variant whatever the evaluation context, which is what lets the untargeted scenarios + expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at + once. +- **The four `disabled-*` flags are the only ones whose state is not `ENABLED`.** They resolve to + nothing: the caller's default stands in. Every other scenario assumes a flag serves its own + value, so enabling one of these, or disabling anything else, breaks that assumption silently. ### The control API @@ -154,6 +158,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `@object` | supports structured flag values | | `@variants` | names the variant it resolved, which [Requirement 2.2.4](./sections/02-providers.md#requirement-224) makes a `SHOULD` and `types.md` types as optional | +| `@disabled-flags` | resolves a flag disabled in the management system to the code default | | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index b57f563cf..85ba14ee1 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -23,8 +23,9 @@ A feature file that evaluates `boolean-flag` is meaningless without the flag def ## Four properties that are load-bearing - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. -- **Only `targeting-key-flag` has a targeting rule.** Every other flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. Its rule is specified by behaviour — resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`, `miss` otherwise — so express it however your backend expresses targeting. The flag, its variants and the uuid are the ones [flagd-testbed's `targeting.feature`](https://github.com/open-feature/flagd-testbed) already uses, on the same reasoning as the zero flags: a backend serving that harness already serves this. +- **Only `targeting-key-flag` has a targeting rule.** Every other enabled flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. Its rule is specified by behaviour — resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`, `miss` otherwise — so express it however your backend expresses targeting. The flag, its variants and the uuid are the ones [flagd-testbed's `targeting.feature`](https://github.com/open-feature/flagd-testbed) already uses, on the same reasoning as the zero flags: a backend serving that harness already serves this. - **`boolean-zero-flag`, `integer-zero-flag` and `string-zero-flag` resolve to falsy values on purpose.** A seeding step that treats `false`, `0` or `""` as "unset" and drops them turns the falsy-value scenarios into `FLAG_NOT_FOUND` failures that look like provider defects. These names, and their `zero`/`non-zero` variants, are the ones [Appendix B's SDK suite](../gherkin/test-flags.json) already uses, so a backend serving that flag set already serves these. +- **The four `disabled-*` flags are the only ones whose state is not `ENABLED`.** They resolve to nothing — the caller's default stands in, and no variant is named. Every other scenario assumes a flag serves its own value, so enabling one of these, or disabling anything else, breaks that assumption silently. Names, variants and values are [flagd-testbed's own](https://github.com/open-feature/flagd-testbed), from `flags/disabled-flags.json`. - **`integral-float-flag` is a float and `huge-integer-flag` is an integer.** Seeding `10.0` as `10` makes the lossless-coercion scenario pass without coercing; seeding `9007199254740991` through a float rounds it. The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index 429005b0b..9876547d0 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -132,6 +132,51 @@ }, "defaultVariant": "one" }, + "disabled-boolean-flag": { + "$comment": [ + "The four disabled flags mirror boolean-flag, string-flag, integer-flag and float-flag", + "exactly, differing only in state. Each scenario's caller default differs from the", + "flag's configured value, so a provider that ignores the state returns the configured", + "value and is caught on the value alone.", + "", + "Names, variants and values are flagd-testbed's own, from flags/disabled-flags.json,", + "so a backend serving that harness already serves these.", + "", + "There is deliberately no disabled-object-flag. An Object resolution needs @object, and", + "a scenario needing both tags cannot be one row of a single outline -- the scalar rows", + "already establish the behaviour." + ], + "state": "DISABLED", + "variants": { + "on": true, + "off": false + }, + "defaultVariant": "on" + }, + "disabled-string-flag": { + "state": "DISABLED", + "variants": { + "greeting": "hi", + "parting": "bye" + }, + "defaultVariant": "greeting" + }, + "disabled-integer-flag": { + "state": "DISABLED", + "variants": { + "one": 1, + "ten": 10 + }, + "defaultVariant": "ten" + }, + "disabled-float-flag": { + "state": "DISABLED", + "variants": { + "tenth": 0.1, + "half": 0.5 + }, + "defaultVariant": "half" + }, "targeting-key-flag": { "$comment": [ "The one flag in this set with a targeting rule. Everything else resolves to its", diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index de50b9597..bebf3ed23 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -136,6 +136,42 @@ Feature: Provider flag evaluation And the error message should be empty And no exception should have been thrown + @disabled-flags + Scenario Outline: A disabled flag resolves to the code default + # Gated, because what a disabled flag resolves to is a property of where the substitution + # happens rather than of provider quality. A provider that evaluates locally — flagd's RPC + # and in-process resolvers, an in-memory provider — can substitute the value the caller + # passed in. A provider whose backend decides, such as one speaking OFREP, cannot: the + # server never sees the caller's default, so it has no way to return it. The same flag + # cannot behave the same way across those two architectures, and neither is wrong. + # + # Nothing in the specification says what a provider owes a disabled flag. Requirement + # 1.4.7 is about the SDK propagating whatever reason arrived, and 2.2.5 only lists + # DISABLED among the reason strings a provider may use. So this appendix states the + # behaviour, the way it does for @numeric-coercion, and gates it. + # + # Asserts the value and the absence of an error, not the reason. Each row's caller default + # differs from the flag's configured value, so a provider that ignores the state returns + # the configured value and fails on the value alone — which rests on 2.2.3, a MUST. + # Pinning reason "DISABLED" would rest on 2.2.5, a SHOULD that permits "some other + # string". + # + # No variant is asserted. A disabled flag has resolved no variant, so there is none to + # name; this scenario and @variants deliberately do not compose. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the error-code should be "" + And the error message should be empty + And no exception should have been thrown + + Examples: + | key | type | default | + | disabled-boolean-flag | Boolean | false | + | disabled-string-flag | String | bye | + | disabled-integer-flag | Integer | 1 | + | disabled-float-flag | Float | 0.1 | + @object Scenario: Resolve a structured value Given a Object-flag with key "object-flag" and a default value "{}" From 70128f86cc3ea7859b5172bd9adb8c0968105c6c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 13:04:11 +0200 Subject: [PATCH 31/63] fix: state why @disabled-flags is gated, correctly this time The comment said a provider whose backend decides "cannot" substitute the caller's default, "such as one speaking OFREP", because the server never sees that default. The conclusion was right and the reasoning was wrong, and three implementations disproved it within a day of it being written. flagd's RPC resolver is remote by exactly that description and passes: the Python provider substitutes locally when the reason is DISABLED and no variant came back (resolvers/grpc.py). The Go and JavaScript OFREP providers pass too, on the same signal -- OFREP's codeDefaultFlag schema omits `value` precisely so the provider can use the code default, and says so normatively. Where evaluation happens is not the axis. What the axis actually is: the backend has to distinguish a disabled flag at all, and the provider has to substitute on the strength of that signal. Either half can be missing. Both OFREP providers that fail today fail on the second half and for unrelated reasons -- one returns FLAG_NOT_FOUND for any code-default response, the other indexes an optional field unconditionally -- which is what a capability is supposed to separate from a defect. Only the comment changes. No scenario, flag or assertion is touched, and the scenario count stays at 56. Signed-off-by: Simon Schrottner --- .../provider-tck/gherkin/evaluation.feature | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index bebf3ed23..ee3bba5d0 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -138,12 +138,16 @@ Feature: Provider flag evaluation @disabled-flags Scenario Outline: A disabled flag resolves to the code default - # Gated, because what a disabled flag resolves to is a property of where the substitution - # happens rather than of provider quality. A provider that evaluates locally — flagd's RPC - # and in-process resolvers, an in-memory provider — can substitute the value the caller - # passed in. A provider whose backend decides, such as one speaking OFREP, cannot: the - # server never sees the caller's default, so it has no way to return it. The same flag - # cannot behave the same way across those two architectures, and neither is wrong. + # Gated, because it takes a provider and its backend together. The backend has to + # distinguish a disabled flag at all — flagd says so with reason DISABLED and no variant, + # and OFREP's codeDefaultFlag schema carries the same signal by omitting `value` — and the + # provider then has to substitute the caller's default on the strength of that signal. A + # backend that instead serves the flag's configured value leaves nothing to detect, and a + # provider that does not substitute cannot pass however clear the signal was. + # + # Where evaluation happens is not the axis. flagd's RPC resolver is remote and passes, by + # substituting locally when the reason is DISABLED and no variant came back; an OFREP + # provider can and does pass on exactly the same reasoning. # # Nothing in the specification says what a provider owes a disabled flag. Requirement # 1.4.7 is about the SDK propagating whatever reason arrived, and 2.2.5 only lists From ee9ae9282cd047b1113fc5faeec0483f22d8a7c2 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 13:41:16 +0200 Subject: [PATCH 32/63] feat: settle what a known deviation means, and who owns the container stack Two things four implementations answered differently, which is one more answer than a portable conformance claim can carry. A knownDeviations entry is legitimate in two shapes -- declare the capability and let the scenario fail, or withhold it and explain the absence -- and the appendix said nothing about which. Java and Go documented the second, JS the first and called the second discouraged, Python required an issue link and had no untracked form at all. A consumer comparing four reports would read one field three ways. The first shape is now preferred, in prose, with the reason: withdrawing a capability to turn a failure into a skip hides a defect behind something that looks deliberate, which is the outcome the field exists to prevent rather than one of its uses. summary is required, issue is not. "Own the lifecycle" already covered the container stack, and three of four implementations shipped only the control-API client anyway -- so every flagd adoption hand-rolled an orchestrator, between 130 and 460 lines of it. That satisfies the letter of the item and not its point, so the item now says what an adopter supplies (a Compose file, which service and ports, a factory taking an endpoint) and what the suite owes. The start-once rule and the ask-do-not-sleep rule are stated with their reasons, because both were learned by getting them wrong. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 03a51fa84..24a1eca83 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -245,6 +245,30 @@ exist. A false failure is the mirror image of a vacuous pass, and a reader canno from the outside. When a scenario fails, find the numbered requirement before concluding anything: check whether it is a `MUST`, a `SHOULD`, or explicitly optional. +**A deviation is legitimate in two shapes, and the results already tell them apart.** Which one an +adopter reaches for is the single most consequential thing about this field, so it is stated here +rather than left to each implementation's documentation — four implementations left to themselves +produced three different answers, and a consumer comparing their reports would read one field three +ways. + +1. **The capability is declared, the scenario runs, and it fails.** The failure stays in the + results and the deviation says it is known, what it is, and where it is tracked. +2. **The capability is withheld, and the scenarios it gates are skipped.** The deviation explains + the absence, so that a reader can tell a defect from a design decision. Both look identical + otherwise: scenarios skipped, reason recoverable from the declaration. + +**Prefer the first.** The second is honest only when the provider cannot attempt the behaviour at +all, so that running the scenario would establish nothing. Where the provider does attempt it and +gets it wrong, withdrawing the capability replaces a failing scenario with a skip and hides a defect +behind something that looks deliberate — which is the outcome this field exists to prevent, not one +of its uses. A conformance report is not improved by having fewer failures in it. + +**`summary` is required; `issue` is not.** A deviation whose summary is empty records that something +is wrong without saying what, which leaves a reader worse off than the bare skip or failure it +accompanies. An untracked deviation is worth declaring even so: naming the defect is what separates +it from a withheld capability, and a declaration that merely omits the tag cannot say which of the +two happened. Prefer a tracked one as soon as there is somewhere to point at. + **The `reason` field is the deliberate exception, and it is stated here so it is a decision rather than an oversight.** [Requirement 2.2.5](./sections/02-providers.md#requirement-225) is also a `SHOULD`, and it goes further than 2.2.4 does: it lets a provider populate the field with one of the @@ -352,6 +376,26 @@ A TCK implementation is the language-specific harness around these three artifac 3. **Own the lifecycle** — start the backend stack once, register the provider under test with the SDK, await events, tear down — so that an adopting provider writes no test infrastructure. If a provider author finds themselves adding lifecycle code, that is a defect in the TCK. + + **The container stack is part of that, and it is the part implementations get wrong.** An + adopter names a Docker Compose file, says which service and which container-internal ports the + provider connects to, and supplies a factory that builds a provider from a discovered endpoint. + Everything else — starting the stack, discovering the dynamically mapped host ports, building + the control client, waiting until the control API accepts commands, tearing down after the last + scenario — belongs to the TCK. Shipping only the control-API client and leaving orchestration to + the adopter satisfies the letter of this item and not its point: the orchestration is then + rewritten by every adopting provider, and it is the largest single piece of test infrastructure + in an adoption. + + Start the stack **once** per suite and never restart it. Container runtimes assign host ports + dynamically and do not reliably preserve them across a restart, so a restart silently + invalidates every provider already pointed at the old port. Backend unavailability is simulated + inside the running stack through the control API, which is what `@stale` and `@unavailable` + already require. + + Wait for readiness by **asking the control API**, not by sleeping. A fixed delay after a control + call is a way of not noticing when the backend's own readiness contract breaks, and the suite + exists to notice. 4. **Drive the backend only through the control API.** This is the part that makes the conformance claim portable: another language's TCK drives the same endpoints against the same stack and must get the same answers. From 640beb2b305c4f61e5b8ec3d9eab416623d1c1c1 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 14:00:23 +0200 Subject: [PATCH 33/63] feat: the control states which path a run used, and the harness must not guess Appendix F described the in-process allowance and the rule that a provider with an external backend must use the control API, but said nothing about how a reader finds out which one a given run actually used. Four implementations filled that silence four different ways, one of them by inferring it from the control's concrete type -- which is correct for the two controls the suite ships and silently wrong for an adopter's own, the only case where the question is open. So: the control states it, it is required, and an omitted value is not neutral. Every run is one or the other, which makes silence an unfalsifiable claim rather than an abstention. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 24a1eca83..e8b73d20d 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -421,6 +421,16 @@ control path should **fail loudly** if a connection operation is reached anyway capability was declared that the harness cannot back up, which is a test-configuration bug rather than a provider defect. +**Which of the two paths a run used is stated by the control, not inferred by the harness, and it is +not optional.** It is the one fact that decides what everything else in a result is worth: the same +scenarios passing over the normative control API and passing through in-process manipulation of a +provider that has a real backend are not the same claim. Nothing outside the control can tell the +two apart — a harness that infers it from the control's concrete type is right about its own two +built-in controls and silently wrong about an adopter's custom one, which is the case where the +answer actually matters. Nor is an absent value neutral: every run is one or the other, so an +omitted value is not "no claim made", it is an unfalsifiable one. A control that cannot say which +path it used is not finished. + ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, From c8c50f502e85870f3f950e2a9fc6cd3c69883b9a Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 14:43:18 +0200 Subject: [PATCH 34/63] fix: require every state-changing control endpoint to serve before it returns Only /start carried "MUST NOT return until the seeded flag state is actually being served". /change and /reset change flag state too and said nothing, so the promise a caller can rely on depended on which endpoint it called -- and /reset is the one a TCK calls before every scenario, which is the worst place for the gap to be. A window there is re-rolled per scenario instead of once per suite, so a small per-call probability becomes a near-certainty over a few hundred scenarios and the failures land on scenarios chosen at random. Two runs then disagree about which scenarios failed, which reads as a flaky provider and is the hardest shape of this to diagnose. For /change the promise is spelled out against the backend rather than the provider: a fresh evaluation resolves the new value once the call returns, while how long the provider takes to notice is its transport's business and is what the event timeout measures. Conflating the two makes detection latency unmeasurable, because the clock would start before there is anything to detect. The appendix gains the invariant in prose beside the two it already lists, plus where a wait belongs when an adopter is stuck with a backend that breaks it: in that adoption, named and citing the defect, never as a constant in the shared harness where every later adopter inherits it without knowing why. The implementer checklist's readiness note is corrected to match -- it said to ask the control API rather than sleep, which is right for the stack coming up and wrong as a general instruction, since after that the endpoints owe the caller the wait themselves. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 27 ++++++++++++++++--- .../provider-tck/openapi/control-api.yaml | 22 ++++++++++++++- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index e8b73d20d..705c73b4a 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -138,6 +138,24 @@ Two invariants are worth stating here because they are the ones a TCK implementa already pointed at the old port. The failure looks like a flaky provider. - **Scenario isolation comes from the control API**, not from cycling the stack. The backend is started once per suite and reset before each scenario. +- **An endpoint that changes flag state must not return success until that state is being served.** + Returning when the change has been *accepted* rather than *applied* pushes a race onto every + caller, and the caller cannot close it: a suite has no way to distinguish "the backend has not + caught up yet" from "the provider resolved the wrong value", which is the exact question it + exists to answer. Readiness of the process is not readiness of the state — a backend whose health + probe reports ready as soon as a configuration has been *handed over* to be parsed will answer + `FLAG_NOT_FOUND` for flags the configuration plainly defines, and a provider that happens to + block during its own initialisation absorbs the window while a stateless one races it. The result + is a suite that flaps per provider rather than per backend, which is the most misleading shape a + conformance failure can take. + + A suite must not paper over a backend that breaks this. A fixed delay after every control call + buys silence, not correctness: it hides the defect from the one consumer positioned to notice, + and it is un-tunable, because the window it covers is a property of the backend and not of the + suite. Where an adopter is stuck with such a backend, the wait belongs in **that adoption**, set + explicitly and citing the defect, so that it reads as a named workaround for a specific backend + and disappears when the backend is fixed — not as a constant buried in the shared harness where + every future adopter inherits it without knowing why. ## Capabilities: how a provider says what it cannot do @@ -393,9 +411,12 @@ A TCK implementation is the language-specific harness around these three artifac inside the running stack through the control API, which is what `@stale` and `@unavailable` already require. - Wait for readiness by **asking the control API**, not by sleeping. A fixed delay after a control - call is a way of not noticing when the backend's own readiness contract breaks, and the suite - exists to notice. + Wait for the stack by **asking the control API** whether it is ready, bounded by the startup + timeout. After that, do not wait at all: a control endpoint that changes flag state owes the + caller that the state is being served before it returns, so a suite that adds a delay of its own + is covering for a backend that broke its side of the contract — see the control API's invariants. + If an adopter's backend does break it, the wait belongs in that adoption, named and with the + defect cited, and not in the shared harness. 4. **Drive the backend only through the control API.** This is the part that makes the conformance claim portable: another language's TCK drives the same endpoints against the same stack and must get the same answers. diff --git a/specification/assets/provider-tck/openapi/control-api.yaml b/specification/assets/provider-tck/openapi/control-api.yaml index d21191067..91e6935f7 100644 --- a/specification/assets/provider-tck/openapi/control-api.yaml +++ b/specification/assets/provider-tck/openapi/control-api.yaml @@ -286,7 +286,18 @@ paths: * change the resolved value of the flag with key `changing-flag`; * do so without restarting the backend process, so that a provider sees a configuration-change signal rather than a reconnect; - * make the change durable until the next `/start` or `/reset`. + * make the change durable until the next `/start` or `/reset`; + * **not return until the new value is actually being served.** + + That last point is the same promise `/start` makes, and it means the + backend, not the provider. A fresh evaluation against the backend MUST + resolve the new value once this returns; how long the *provider under + test* takes to notice is a property of its transport — streaming sees it + in milliseconds, a poller may need most of an interval — and that is what + the TCK's event timeout is for. The two must not be confused: a backend + that returns before it is serving the new value makes the provider's + detection latency unmeasurable, because the clock starts before there is + anything to detect. The implementation SHOULD toggle between exactly two known values so that repeated calls are meaningful and the test remains deterministic @@ -315,6 +326,15 @@ paths: causes no availability blip, so it cannot inject spurious lifecycle events into the next scenario. + **MUST NOT return until the restored baseline is actually being served**, + exactly as `/start` must not. This is worth stating separately because + `/reset` is the endpoint a TCK calls before *every* scenario: a window + here is re-rolled per scenario rather than once per suite, so a small + probability becomes a near-certainty over a few hundred scenarios, and + the failures land on scenarios chosen at random. Two runs of the same + suite then disagree about which scenarios failed, which reads as a flaky + provider and is the hardest shape of this defect to diagnose. + **Scope.** This operation resets flag state only. It MUST NOT be expected to start a backend that is currently stopped — that is what `/start` is for. A TCK therefore uses `/reset` only when the backend is From 537da09f2a9bf653177a9ffea613102e69fc8ea8 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 15:53:10 +0200 Subject: [PATCH 35/63] fix: /restart is optional, and the TCK never called it The endpoint was marked [REQUIRED] on the strength of a claim in its own description -- "the TCK uses this for the disconnect/reconnect scenarios" -- that was not true. The disconnect/reconnect scenario is written as an unbounded outage, "the connection is lost" then "the connection is restored", which an implementation serves with /stop followed by /start. Nothing in the shipped Gherkin reaches /restart in any of the four languages: Go dropped its binding deliberately, and the Java and Python clients kept one that no step can call. So the contract was asking every backend author to implement an endpoint nothing invoked. Demoted rather than deleted, with the condition that would bring it back written down. /start on reconnect resets flag state where /restart would have preserved it, and that is only acceptable because the one scenario involved asserts events and client state and never a resolved value. A scenario that asserts what a stale provider serves during an outage -- last-known-value caching, held behind @caching and not yet written -- needs exactly this endpoint's preservation. A backend author who wants that capability testable later should implement it now. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 2 +- .../provider-tck/openapi/control-api.yaml | 22 ++++++++++++++++--- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 705c73b4a..8185df162 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -123,7 +123,7 @@ vendors. The control API is a small HTTP surface the backend under test exposes | --- | --- | --- | | `POST /start?config=` | yes | start the backend, seeding flags to a named baseline | | `POST /stop` | yes | make the backend unreachable | -| `POST /restart?seconds=` | yes | simulate an outage of a bounded duration | +| `POST /restart?seconds=` | no | simulate a *bounded* outage, preserving flag state | | `POST /change` | yes | mutate flag configuration so the provider observes a change | | `POST /reset` | no | restore the baseline without an availability blip | | `GET /healthz` | no | readiness of the control API | diff --git a/specification/assets/provider-tck/openapi/control-api.yaml b/specification/assets/provider-tck/openapi/control-api.yaml index 91e6935f7..316772ccb 100644 --- a/specification/assets/provider-tck/openapi/control-api.yaml +++ b/specification/assets/provider-tck/openapi/control-api.yaml @@ -233,7 +233,7 @@ paths: post: tags: [availability] operationId: restart - summary: "[REQUIRED] Simulate an outage of a bounded duration" + summary: "[OPTIONAL] Simulate an outage of a bounded duration" description: | Makes the backend unreachable, waits `seconds`, then starts it again with the configuration currently in effect. @@ -247,8 +247,24 @@ paths: blocking for the full duration; the TCK does not rely on the response being delayed. It awaits provider events instead. - The TCK uses this for the disconnect/reconnect scenarios: `STALE` → - `PROVIDER_STALE`, then back to `READY` → `PROVIDER_READY`. + **No shipped scenario reaches this endpoint**, and it is optional for + that reason. The disconnect/reconnect scenario — `PROVIDER_STALE`, then + back to `PROVIDER_READY` — is written as an *unbounded* outage (`the + connection is lost`, then `the connection is restored`), which a TCK + implements with `/stop` followed by `/start`. An earlier version of this + description claimed the TCK used `/restart` for those scenarios; it did + not, and requiring the endpoint on that basis asked every backend author + to implement something nothing called. + + `/start` on reconnect also resets flag state, which `/restart` would + have preserved. That is acceptable today because the only scenario + involved asserts events and client state, never a resolved value. It + stops being acceptable the moment a scenario asserts what a stale + provider serves *during* an outage — last-known-value caching, which is + held behind the `@caching` capability and not yet written. Such a + scenario needs exactly this endpoint's state preservation, which is why + it remains specified rather than deleted: implement it if you want that + capability testable against your backend later. parameters: - name: seconds in: query From 67a564bbf3dcc04557a63aeb1fe522c1942cdd40 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 23:14:27 +0200 Subject: [PATCH 36/63] feat: say how an adoption suite relates to CI, and warn about cached reasons Two cross-language lessons that were living in four READMEs, which is where the known-deviation and control-path decisions had already drifted into three different answers. An adoption suite is not a required gate while real gaps remain: its honest output is red, because it fails on filed provider defects, missing backend fixtures and undeclared capabilities. Making it block a merge forces someone to silence it, and the cheapest way to silence a conformance suite is to stop asking the question -- withdraw a capability, delete an assertion, pin an older backend. So exclude it and say so, with two observed mistakes named: an exclusion that a profile, target or build tag undoes, and an exclusion nobody wrote down. All four implementations believed their suites were excluded; all four were running them, red and unwatched, each defeated by a different mechanism. Keep the suite compiling in the default build even when it does not run, because a suite that has silently stopped building against its harness is worse than one that runs and fails. The caching gap gains the constraint a scenario author needs. A provider may cache client-side and rewrite the reason when it does -- flagd's RPC resolver runs an LRU cache by default and reports CACHED on a repeat evaluation -- so any scenario evaluating the same flag twice sees a different reason the second time from a provider that is behaving correctly. Exactly one current scenario evaluates twice, and it changes the configuration in between. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 8185df162..21eaf5f6b 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -452,6 +452,33 @@ answer actually matters. Nor is an absent value neutral: every run is one or the omitted value is not "no claim made", it is an unfalsifiable one. A control that cannot say which path it used is not finished. +### Running the suite in CI + +Guidance rather than a rule — a repository's pipeline is its own business — but the reasoning is the +same in every language, and leaving it unwritten produced four mechanisms and one unnoticed +consequence. + +**An adoption suite is not a required gate while real gaps remain.** Its honest output is red: it +fails on provider defects that are filed and unfixed, on backend fixtures that do not exist yet, and +on capabilities the provider has not implemented. A red result is the suite working. Making it +block a merge forces someone to silence it, and the cheapest way to silence a conformance suite is +to stop asking the question — withdraw a capability, delete an assertion, or pin an older backend. + +So **exclude it from the default build, and make the exclusion explicit.** Two mistakes to avoid, +both observed: + +- **An exclusion that something else undoes.** The question is not whether an exclusion exists but + whether any profile, target or job re-enables it. In one language a CI profile cleared the + adopter's own exclusion property; in another the suite ran under a build tag applied to every + module; in a third the default test task simply collected it. All four languages believed their + suites were excluded and all four were running them, red, unwatched. +- **An exclusion nobody wrote down.** It is then indistinguishable from an oversight, and the next + person to touch the pipeline removes it or duplicates it. State it where an adopter will read it. + +Provide a single documented command that runs the suite deliberately, and keep the suite +*compiling* in the default build even when it does not execute — a conformance suite that has +quietly stopped building against its own harness is a worse failure than one that runs and fails. + ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, @@ -534,6 +561,14 @@ belong here rather than in any one implementation: designated flag. Finer-grained flag manipulation would need new endpoints. - **Caching.** Whether a stale provider keeps serving last-known values during an outage depends on whether it holds a local copy of the ruleset. The `@caching` tag is reserved; no scenarios yet. + + Worth knowing before writing them: a provider may cache on the client side and *rewrite the + reason* when it does. flagd's RPC resolver runs an LRU cache by default and reports `CACHED` on a + repeat evaluation of an unchanged flag. So any scenario that evaluates the same flag twice in one + scenario — the obvious shape for a caching test, and equally for a "value is stable" test — will + see a different reason the second time from a provider that is behaving correctly. This is why no + existing scenario evaluates a flag twice without a configuration change in between, and it is a + constraint on new scenarios rather than a defect in any provider. - **Coverage of the numbered requirements.** Mapped against [the provider requirements](./sections/02-providers.md), leaving out 2.8.5.1 (it constrains the SDK) and 2.2.8.1 (a language-binding property, not observable at runtime), the suite covers 10 of the From 9071e13756230caa642494a1e130dedfa7ac92a9 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 09:34:19 +0200 Subject: [PATCH 37/63] feat: make the resolution reason a claim a provider declares, not a house rule The suite asserted an exact reason in thirteen places across three feature files, which narrowed 2.2.5 into a MUST for every adopter. 2.2.5 is a SHOULD that goes further than the others: a provider may use one of the listed values "or some other string indicating the semantic reason for the returned flag value". A provider whose backend reports vendor-specific reasons was therefore being failed for something the specification permits, and the appendix recorded that as a deliberate exception. It bought very little. Every canonical flag resolves to a value distinct from the caller's default, so a provider that silently falls back is already caught by the value assertion -- the reason only said why it failed rather than that it did. And of the thirteen assertions, five sat beside an error-code assertion that already carries the MUST, while the remaining eight asserted STATIC, which is the one reason the specification genuinely leaves open: types.md types DEFAULT as "no dynamic evaluation occurred or dynamic evaluation yielded no result", and a rule-less flag satisfies that as readily as STATIC does. So the reasons move to reason.feature, gated as a whole on @standard-reasons. The gate is a claim rather than an exemption: a provider declares that it uses the standard vocabulary with the standard meanings, and that file checks it. One that does not declare it loses nothing, since values, variants and error codes are asserted everywhere else on MUSTs. What the declaration buys is something a report's reader can act on -- anyone building telemetry or debugging on reason can see the vocabulary was verified rather than assumed. This also lets the appendix state what each reason means without asking the specification to close the set, because the mapping is the content of an opt-in claim and constrains nobody who does not make it. TARGETING_MATCH and DISABLED compose with @targeting and @disabled-flags, since neither reason can be observed without the behaviour that produces it. CACHED and STALE are left out: both need a scenario shape the suite does not have yet. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 90 ++++++++++----- .../provider-tck/gherkin/errors.feature | 6 - .../provider-tck/gherkin/evaluation.feature | 51 ++++----- .../provider-tck/gherkin/lifecycle.feature | 2 - .../provider-tck/gherkin/reason.feature | 108 ++++++++++++++++++ 5 files changed, 195 insertions(+), 62 deletions(-) create mode 100644 specification/assets/provider-tck/gherkin/reason.feature diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 21eaf5f6b..1f89163f4 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -39,8 +39,9 @@ claims, and a language needs both suites to make both. **In scope — the provider contract:** -- mapping backend responses onto typed resolution details: value, variant, reason, error code, and - no error message on a normal evaluation +- mapping backend responses onto typed resolution details: value, variant, error code, and no error + message on a normal evaluation; the resolution reason where a provider claims the standard + vocabulary - the values most often mistaken for an absence — `false`, `0` and `""` — resolving as values - integer precision: a 32-bit maximum for every language, and 2^53 − 1 where the accessor allows it - keeping the integer and float types distinct rather than coercing between them @@ -78,10 +79,12 @@ others breaks the suite in every language at once. ### Gherkin scenarios -Five feature files: +Six feature files: - [`evaluation.feature`](./assets/provider-tck/gherkin/evaluation.feature) — resolving each type with - the right value, variant and reason; falsy values; integer precision + the right value and variant; falsy values; integer precision +- [`reason.feature`](./assets/provider-tck/gherkin/reason.feature) — the standard resolution reasons, + gated as a whole on `@standard-reasons` - [`errors.feature`](./assets/provider-tck/gherkin/errors.feature) — the type-mismatch matrix, numeric coercion and the unknown-flag case - [`events.feature`](./assets/provider-tck/gherkin/events.feature) — configuration change, and the @@ -107,9 +110,9 @@ Two properties are load-bearing and easy to break by accident: - **`missing-flag` must not exist.** Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. - **Only `targeting-key-flag` has a targeting rule.** Every other enabled flag resolves to its - default variant whatever the evaluation context, which is what lets the untargeted scenarios - expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at - once. + default variant whatever the evaluation context, which is what lets a provider declaring + `@standard-reasons` expect `STATIC` rather than `TARGETING_MATCH` for them. Seeding targeting onto + any other flag breaks them in every language at once. - **The four `disabled-*` flags are the only ones whose state is not `ENABLED`.** They resolve to nothing: the caller's default stands in. Every other scenario assumes a flag serves its own value, so enabling one of these, or disabling anything else, breaks that assumption silently. @@ -182,6 +185,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](./sections/02-providers.md#requirement-252) permits rather than requires | | `@targeting` | resolves a flag differently for a matching evaluation context | +| `@standard-reasons` | reports the standard resolution reasons, with the meanings given below | | `@caching` | reserved; **not declarable** -- no scenarios yet | Untagged scenarios are mandatory and always run. @@ -287,24 +291,55 @@ accompanies. An untracked deviation is worth declaring even so: naming the defec it from a withheld capability, and a declaration that merely omits the tag cannot say which of the two happened. Prefer a tracked one as soon as there is somewhere to point at. -**The `reason` field is the deliberate exception, and it is stated here so it is a decision rather -than an oversight.** [Requirement 2.2.5](./sections/02-providers.md#requirement-225) is also a -`SHOULD`, and it goes further than 2.2.4 does: it lets a provider populate the field with one of the -listed values *"or some other string indicating the semantic reason for the returned flag value"*. -The suite nonetheless requires a reason, and requires a specific one, in every scenario that asserts -it. A provider whose backend reports vendor-specific reason strings will fail those scenarios. - -That is a narrowing of the specification, and it is accepted for now because the reason is the -suite's cheapest diagnosis of a whole class of silent failure: a provider that quietly falls back to -the code default reports a different reason, and the assertion names the problem where a value -assertion alone only says the number was wrong. Gating it would mean a second capability, a second -set of scenarios to keep in step, and a declaration nearly every provider would make anyway. - -A reader comparing reports should therefore treat a reason failure differently from a value failure: -the value assertions rest on `MUST` requirements, the reason assertions rest on a house rule. If a -conformant provider is failed by one, that is this suite's narrowing and not that provider's defect --- and the right response is to revisit this decision, not to record a deviation against the -provider. +### `@standard-reasons`: a claim, not an exemption + +[Requirement 2.2.5](./sections/02-providers.md#requirement-225) is a `SHOULD`, and it goes further +than 2.2.4 does: it lets a provider populate `reason` with one of the listed values *"or some other +string indicating the semantic reason for the returned flag value"*. A provider whose backend +reports vendor-specific reasons is therefore conformant, and asserting an exact reason against it +would fail it for something the specification permits. + +An earlier revision of this suite did exactly that, in thirteen places across three feature files, +and recorded the narrowing here as a deliberate exception. It is not one any more, for two reasons. +It bought very little: every canonical flag resolves to a value distinct from the caller's default, +so a provider that silently falls back is already caught by the value assertion, and the reason only +said *why* it failed. And of the thirteen, five sat beside an error-code assertion that already +carries the `MUST`, while the other eight asserted `STATIC` -- the one reason the specification +genuinely leaves open. + +So the reasons now live in `reason.feature`, gated as a whole. **Declaring `@standard-reasons` is a +provider saying "I use the standard vocabulary with the standard meanings", and that file is what +checks the claim.** A provider that does not declare it loses nothing: its values, variants and error +codes are asserted everywhere else, on `MUST` requirements. What the declaration adds is something a +report's reader can act on -- anyone building telemetry, dashboards or debugging on `reason` can see +that the vocabulary was verified rather than assumed. + +This also settles a question the specification does not, without asking it to. The meanings below are +the content of an opt-in claim; they constrain nobody who does not make it. + +| Situation | Reason | +| --- | --- | +| The flag was resolved from configuration and carries no targeting rule | `STATIC` | +| A targeting rule matched the evaluation context | `TARGETING_MATCH` | +| A targeting rule exists and did not match | `DEFAULT` | +| The flag is disabled in the management system | `DISABLED` | +| The evaluation failed, and an error code is reported with it | `ERROR` | + +`STATIC` for the first row is the call worth flagging. `types.md` types `DEFAULT` as *"no dynamic +evaluation occurred **or** dynamic evaluation yielded no result"*, which a rule-less flag satisfies +as readily as `STATIC` does -- two providers can disagree here and both conform. A provider that +answers `DEFAULT` for a rule-less flag is not defective; it does not use the standard meanings, and +should not declare the tag. + +`SPLIT`, `UNKNOWN`, `CACHED` and `STALE` are not asserted. The first two have no scenario that +produces them. `CACHED` needs a repeat evaluation, which nothing here performs without a +configuration change in between -- see the caching entry under known gaps. `STALE` needs a scenario +asserting what a provider serves *during* an outage, which is the same gap. + +**Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without +targeting, and `DISABLED` cannot be observed unless the backend distinguishes a disabled flag, so +those scenarios carry `@targeting` and `@disabled-flags` as well. A provider declaring +`@standard-reasons` alone runs the rest and skips those two with their reason. `@variants` is the clearest case, and it was found the hard way. Every evaluation scenario asserted a variant, which reads as obviously correct until a backend with no variant concept for a plain flag @@ -572,8 +607,9 @@ belong here rather than in any one implementation: - **Coverage of the numbered requirements.** Mapped against [the provider requirements](./sections/02-providers.md), leaving out 2.8.5.1 (it constrains the SDK) and 2.2.8.1 (a language-binding property, not observable at runtime), the suite covers 10 of the - 14 `MUST` requirements in scope, all 5 `SHOULD` — 2.2.4 only for a provider declaring `@variants` - — and 1 of 6 `MAY`. The `MUST` gaps are 2.3.1 (the provider hook mechanism, a compile-time + 14 `MUST` requirements in scope, all 5 `SHOULD` — 2.2.4 only for a provider declaring `@variants`, + and 2.2.5 only for one declaring `@standard-reasons` — and 1 of 6 `MAY`. The `MUST` gaps are + 2.3.1 (the provider hook mechanism, a compile-time property in typed languages with little to observe at runtime), 2.2.10 (flag metadata structure, blocked with 2.2.9 on the canonical flag set defining none), 2.4.4 (a domain-scoped provider accepts its bound domain, which is as much SDK as provider diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature index fd696092d..6fb191fcc 100644 --- a/specification/assets/provider-tck/gherkin/errors.feature +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -19,7 +19,6 @@ Feature: Provider error handling Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" - And the reason should be "ERROR" And the error-code should be "TYPE_MISMATCH" And no exception should have been thrown @@ -48,7 +47,6 @@ Feature: Provider error handling Given a -flag with key "object-flag" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" - And the reason should be "ERROR" And the error-code should be "TYPE_MISMATCH" And no exception should have been thrown @@ -71,7 +69,6 @@ Feature: Provider error handling Given a Integer-flag with key "float-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "1" - And the reason should be "ERROR" And the error-code should be "TYPE_MISMATCH" And no exception should have been thrown @@ -82,7 +79,6 @@ Feature: Provider error handling Given a Integer-flag with key "integral-float-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "10" - And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown @@ -93,7 +89,6 @@ Feature: Provider error handling Given a Float-flag with key "integer-flag" and a default value "0.1" When the flag was evaluated with details Then the resolved details value should be "10" - And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown @@ -102,6 +97,5 @@ Feature: Provider error handling Given a String-flag with key "missing-flag" and a default value "fallback" When the flag was evaluated with details Then the resolved details value should be "fallback" - And the reason should be "ERROR" And the error-code should be "FLAG_NOT_FOUND" And no exception should have been thrown diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature index ee3bba5d0..6d1a0b599 100644 --- a/specification/assets/provider-tck/gherkin/evaluation.feature +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -4,9 +4,11 @@ Feature: Provider flag evaluation # # This does NOT test the backend's evaluation logic. Every flag in the canonical set except # targeting-key-flag resolves to its default variant whatever the context, so what is under - # test is purely the provider's mapping of a backend response to a value, a variant and a - # reason. targeting-key-flag carries the one rule, only to show the context reached the - # backend. + # test is purely the provider's mapping of a backend response to a value and a variant. + # targeting-key-flag carries the one rule, only to show the context reached the backend. + # + # No scenario here asserts a resolution reason. The reasons are a provider's claim to use the + # standard vocabulary, checked in reason.feature behind @standard-reasons. # # Every success path also asserts that no error message was set (requirement 2.3.2). A # provider that reports a value AND an error message is sending two contradictory signals, @@ -17,31 +19,29 @@ Feature: Provider flag evaluation Background: Given a stable provider - Scenario Outline: Resolve values with reason + Scenario Outline: Resolve values Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" - And the reason should be "" And the error-code should be "" And the error message should be empty And no exception should have been thrown Examples: - | key | type | default | value | reason | - | boolean-flag | Boolean | false | true | STATIC | - | string-flag | String | bye | hi | STATIC | - | integer-flag | Integer | 1 | 10 | STATIC | - | float-flag | Float | 0.1 | 0.5 | STATIC | + | key | type | default | value | + | boolean-flag | Boolean | false | true | + | string-flag | String | bye | hi | + | integer-flag | Integer | 1 | 10 | + | float-flag | Float | 0.1 | 0.5 | Scenario Outline: A falsy value is a value, not an absence # false, 0 and "" are the values most likely to be mistaken for "nothing came back": a # `value || default` in JavaScript, a zero-value check in Go, an `if not value` in Python. # Each row's default differs from its resolved value, so a provider that falls back on a - # falsy result returns the wrong value AND the wrong reason, and is caught twice over. + # falsy result returns the wrong value and is caught by the value assertion alone. Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" - And the reason should be "STATIC" And the error-code should be "" And the error message should be empty And no exception should have been thrown @@ -68,8 +68,8 @@ Feature: Provider flag evaluation # # A provider whose backend names its variants declares this tag and these rows run. One # whose backend does not leaves it undeclared, and they are skipped with that reason rather - # than passed. Either way the value and reason assertions above are unaffected: they are - # untagged, and 2.2.3 makes the value a MUST. + # than passed. Either way the value assertions above are unaffected: they are untagged, and + # 2.2.3 makes the value a MUST. Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the variant should be "" @@ -95,7 +95,6 @@ Feature: Provider flag evaluation Given a Integer-flag with key "large-integer-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "2147483647" - And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown @@ -113,7 +112,6 @@ Feature: Provider flag evaluation Given a Integer-flag with key "huge-integer-flag" and a default value "1" When the flag was evaluated with details Then the resolved details value should be "9007199254740991" - And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown @@ -157,8 +155,9 @@ Feature: Provider flag evaluation # Asserts the value and the absence of an error, not the reason. Each row's caller default # differs from the flag's configured value, so a provider that ignores the state returns # the configured value and fails on the value alone — which rests on 2.2.3, a MUST. - # Pinning reason "DISABLED" would rest on 2.2.5, a SHOULD that permits "some other - # string". + # Pinning reason "DISABLED" here would rest on 2.2.5, a SHOULD that permits "some other + # string"; it is pinned in reason.feature instead, for providers that declare + # @standard-reasons and so opt into the standard meanings. # # No variant is asserted. A disabled flag has resolved no variant, so there is none to # name; this scenario and @variants deliberately do not compose. @@ -180,8 +179,7 @@ Feature: Provider flag evaluation Scenario: Resolve a structured value Given a Object-flag with key "object-flag" and a default value "{}" When the flag was evaluated with details - Then the reason should be "STATIC" - And the error-code should be "" + Then the error-code should be "" And the error message should be empty And no exception should have been thrown And the resolved object value should contain @@ -204,9 +202,8 @@ Feature: Provider flag evaluation # # Deliberately asserts the value and the absence of an error rather than the reason. # 2.2.3 makes the value a MUST and 2.2.6 forbids an error code in normal execution, while - # the reason is a SHOULD that 2.2.5 lets a provider populate with "some other string" — - # and with a context supplied and nothing matching, both "STATIC" and "DEFAULT" are - # defensible readings. + # the reason is a SHOULD that 2.2.5 lets a provider populate with "some other string". + # Reasons are asserted only in reason.feature, behind @standard-reasons. Given a String-flag with key "string-flag" and a default value "bye" And a context containing a targeting key with value "f20bd32d-703b-48b6-bc8e-79d53c85134a" When the flag was evaluated with details @@ -227,10 +224,10 @@ Feature: Provider flag evaluation # expresses targeting. The flag, its variants and the uuid are flagd-testbed's own, so a # backend serving that harness already serves this one. # - # Asserts the value, not the reason. Elsewhere the suite does pin the reason, deliberately - # — see Appendix F. Here it cannot: flagd reports TARGETING_MATCH for this hit and DEFAULT - # for the miss below, and with a rule present but unmatched both readings are right, so - # there is no single value to pin. The resolved value carries the whole signal anyway. + # Asserts the value, not the reason: the resolved value carries the whole signal, and a + # provider that reports vendor-specific reasons is conformant. TARGETING_MATCH for this hit + # and DEFAULT for the miss below are asserted in reason.feature, which composes + # @standard-reasons with @targeting so that both must be declared. Given a String-flag with key "targeting-key-flag" and a default value "fallback" And a context containing a targeting key with value "5c3d8535-f81a-4478-a6d3-afaa4d51199e" When the flag was evaluated with details diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature index 635e5fd95..16033456c 100644 --- a/specification/assets/provider-tck/gherkin/lifecycle.feature +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -42,7 +42,6 @@ Feature: Provider lifecycle Then the error event handler should have been executed within 10000ms When the flag was evaluated with details Then the resolved details value should be "false" - And the reason should be "ERROR" And no exception should have been thrown Scenario: Shutting down a provider twice has no further effect @@ -76,7 +75,6 @@ Feature: Provider lifecycle And the provider is initialized again And the flag was evaluated with details Then the resolved details value should be "true" - And the reason should be "STATIC" And the error-code should be "" And no exception should have been thrown diff --git a/specification/assets/provider-tck/gherkin/reason.feature b/specification/assets/provider-tck/gherkin/reason.feature new file mode 100644 index 000000000..9c7bcc06b --- /dev/null +++ b/specification/assets/provider-tck/gherkin/reason.feature @@ -0,0 +1,108 @@ +@standard-reasons +Feature: Provider resolution reasons + + # Verifies that a provider reports the standard resolution reasons, with the meanings Appendix F + # gives them. + # + # THE WHOLE FILE IS GATED, and the gate is a claim rather than an excuse. Requirement 2.2.5 is a + # SHOULD, and it goes further than the other SHOULDs: a provider may populate `reason` with one of + # the listed values "or some other string indicating the semantic reason for the returned flag + # value". A provider whose backend reports vendor-specific reasons is therefore conformant, and + # asserting an exact reason against it would fail it for something the specification permits. + # + # So `@standard-reasons` is a provider saying "I use the standard vocabulary, with the standard + # meanings" — and this file is what checks that claim. A provider that does not say it leaves these + # scenarios skipped with their reason and loses nothing; its values and error codes are still + # asserted everywhere else, because those rest on MUSTs. The declaration is what a consumer reads + # when it wants to build telemetry, dashboards or debugging on `reason`: the claim was verified, + # not assumed. + # + # No other scenario in this suite asserts a reason. That is deliberate — an earlier revision + # asserted one in thirteen places across three files, which narrowed a SHOULD into a MUST for every + # adopter, and bought very little: every canonical flag resolves to a value distinct from the + # caller's default, so a provider that silently falls back is already caught by the value. + # + # TAGS COMPOSE. A scenario carrying `@targeting` or `@disabled-flags` needs that capability + # declared as well, because the reason cannot be observed without the behaviour that produces it. + # + # Two reasons are deliberately absent. CACHED belongs behind the reserved `@caching` tag and needs + # a repeat evaluation, which no scenario in this suite performs without a configuration change in + # between — see the caching note in Appendix F. STALE is reported during an outage, and what a + # provider serves while stale is itself uncovered, so there is nothing to attach it to yet. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: A flag with no targeting rules resolves statically + # This is the assertion the specification leaves genuinely open, and the reason this capability + # has to define its terms rather than merely name them. `types.md` types DEFAULT as "no dynamic + # evaluation occurred OR dynamic evaluation yielded no result", which a rule-less flag satisfies + # as readily as STATIC does. Appendix F picks STATIC for this capability — see its reason + # mapping — and a provider that answers DEFAULT here is not thereby defective, it simply does + # not use the standard meanings and should not declare the tag. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + + Examples: + | key | type | default | + | boolean-flag | Boolean | false | + | string-flag | String | bye | + | integer-flag | Integer | 1 | + | float-flag | Float | 0.1 | + + Scenario: An unknown flag reports an error + # Paired with the error-code assertion in errors.feature, which is the MUST. This adds only that + # the reason agrees with the error code, which is what a consumer keying on reason alone needs. + Given a String-flag with key "missing-flag" and a default value "fallback" + When the flag was evaluated with details + Then the reason should be "ERROR" + And the error-code should be "FLAG_NOT_FOUND" + And no exception should have been thrown + + Scenario: A type mismatch reports an error + Given a String-flag with key "boolean-flag" and a default value "fallback" + When the flag was evaluated with details + Then the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + @targeting + Scenario: A matching targeting rule reports a targeting match + # Needs @targeting as well: a provider with no targeting has no rule to match, so there is no + # TARGETING_MATCH for it to report and the scenario would fail it for an absence rather than a + # defect. + Given a String-flag with key "targeting-key-flag" and a default value "fallback" + And a context containing a targeting key with value "5c3d8535-f81a-4478-a6d3-afaa4d51199e" + When the flag was evaluated with details + Then the reason should be "TARGETING_MATCH" + And the error-code should be "" + And no exception should have been thrown + + @targeting + Scenario: A targeting rule that does not match reports the default + # The other half, and the one that distinguishes the two reasons rather than merely observing + # one of them. A provider that reports TARGETING_MATCH whenever a rule exists, matched or not, + # passes the scenario above and fails this one. + Given a String-flag with key "targeting-key-flag" and a default value "fallback" + And a context containing a targeting key with value "f20bd32d-703b-48b6-bc8e-79d53c85134a" + When the flag was evaluated with details + Then the reason should be "DEFAULT" + And the error-code should be "" + And no exception should have been thrown + + @disabled-flags + Scenario: A disabled flag reports that it is disabled + # Needs @disabled-flags as well, for the same reason: a backend that cannot distinguish a + # disabled flag produces no DISABLED signal for the provider to pass on. The @disabled-flags + # scenarios themselves assert only the value, because pinning the reason there would have + # narrowed 2.2.5 for every adopter — this is where that narrowing is opted into instead. + Given a Boolean-flag with key "disabled-boolean-flag" and a default value "false" + When the flag was evaluated with details + Then the reason should be "DISABLED" + And the error-code should be "" + And no exception should have been thrown From b902949ba6c5f884636ea69ad4f4b7fb78e2584e Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 09:40:34 +0200 Subject: [PATCH 38/63] docs: say what an ERROR reason assertion does and does not establish The four reasons a provider produces in normal execution rest on Requirement 1.4.7, which makes the SDK propagate the provider's reason -- but 1.4.7 is scoped to "cases of normal execution". Abnormal execution is 1.4.9, a SHOULD on the SDK to "indicate an error", and nothing requires the provider's reason to survive it. So an ERROR scenario in a provider suite can pass on a value the SDK wrote, which is the vacuous shape this appendix warns about everywhere else. Kept rather than dropped, because the assertion is the pair and not the field. The error code on its own is already covered for every provider by 2.2.7 and 1.4.8, both MUSTs on a closed enumeration, asserted ungated in errors.feature. The reason on its own could be the SDK's. Their agreement is what neither can satisfy alone, and an evaluation reporting FLAG_NOT_FOUND with reason STATIC is incoherent whoever wrote it. Recorded in both places a reader might look: beside the scenarios, and in the appendix's reason mapping. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 14 ++++++++++++++ .../assets/provider-tck/gherkin/reason.feature | 16 ++++++++++++++-- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 1f89163f4..e2313defb 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -331,6 +331,20 @@ as readily as `STATIC` does -- two providers can disagree here and both conform. answers `DEFAULT` for a rule-less flag is not defective; it does not use the standard meanings, and should not declare the tag. +`ERROR` is the row where this suite's subject is blurred, and it is asserted anyway. The other four +rest on [Requirement 1.4.7](./sections/01-flag-evaluation.md#requirement-147), which makes the SDK +propagate the provider's reason — but only *"in cases of normal execution"*. Abnormal execution is +[1.4.9](./sections/01-flag-evaluation.md#requirement-149), a `SHOULD` on the **SDK** to "indicate an +error", and nothing requires the provider's reason to survive. So a passing `ERROR` scenario +establishes that the value reaching the application is coherent, not that the provider produced it. + +That is still worth asserting, because it is the pair that carries the meaning. The error code alone +is already covered for every provider — 2.2.7 and +[1.4.8](./sections/01-flag-evaluation.md#requirement-148) make it a `MUST`, it is a closed +enumeration, and `errors.feature` asserts it ungated. The reason alone could have been written by the +SDK. An evaluation reporting `FLAG_NOT_FOUND` with reason `STATIC` is incoherent whoever wrote it, +and that is what the pairing catches. + `SPLIT`, `UNKNOWN`, `CACHED` and `STALE` are not asserted. The first two have no scenario that produces them. `CACHED` needs a repeat evaluation, which nothing here performs without a configuration change in between -- see the caching entry under known gaps. `STALE` needs a scenario diff --git a/specification/assets/provider-tck/gherkin/reason.feature b/specification/assets/provider-tck/gherkin/reason.feature index 9c7bcc06b..97bbc7da5 100644 --- a/specification/assets/provider-tck/gherkin/reason.feature +++ b/specification/assets/provider-tck/gherkin/reason.feature @@ -55,9 +55,21 @@ Feature: Provider resolution reasons | integer-flag | Integer | 1 | | float-flag | Float | 0.1 | + # THE TWO ERROR SCENARIOS BELOW ASSERT AGREEMENT, not authorship, and the distinction is worth + # stating because it is the one place this file's subject is blurred. + # + # Every other scenario here rests on [Requirement 1.4.7](../../../sections/01-flag-evaluation.md), + # which makes the SDK propagate the provider's reason — but only "in cases of normal execution". + # Abnormal execution is 1.4.9, and that is a SHOULD on the *SDK* to "indicate an error"; nothing + # requires the provider's reason to survive. So a passing ERROR scenario does not establish that + # the provider set the reason, only that whatever reached the application is consistent. + # + # They are still worth running. The error code alone is already asserted in errors.feature, on a + # MUST, for every provider; the reason alone could be written by the SDK. Asserting the pair is + # the part neither field can satisfy on its own, and an evaluation that reports FLAG_NOT_FOUND + # with reason STATIC is incoherent whoever wrote it. + Scenario: An unknown flag reports an error - # Paired with the error-code assertion in errors.feature, which is the MUST. This adds only that - # the reason agrees with the error code, which is what a consumer keying on reason alone needs. Given a String-flag with key "missing-flag" and a default value "fallback" When the flag was evaluated with details Then the reason should be "ERROR" From 6b20110a71dd09bf52182c8a8fa8f2e8579d6ed6 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 10:41:27 +0200 Subject: [PATCH 39/63] feat: a capability the SDK cannot express is refused by the implementation Two capabilities say nothing about any provider: @large-integers where the integer accessor is 32-bit, and @numeric-coercion where the language has a single numeric type. No provider in those languages can satisfy them and none ever will until the SDK changes. Until now that was documentation, and adopters were expected to act on it. They did, repeatedly: in one implementation three separate suites each left the same capability undeclared, each with its own comment restating the same property of the language. A fact about a language was being remembered in three places per language and in every future adoption, and a single wrong one would put a claim in a report that no scenario could have verified -- the failure the reserved-capability rules exist to prevent, reached by another route. So the implementation refuses it at configuration time, using the machinery that already refuses a reserved capability. The two refusals are deliberately kept distinguishable in the skip reason, because they are different facts: a reserved capability is global and expires when the specification adds scenarios, while an inexpressible one is a single language's and permanent. A reader seeing a capability absent from a report has to be able to tell "this provider declined" from "no provider in this language can be asked", since only the first describes the provider. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index e2313defb..2c6f086af 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -241,9 +241,29 @@ to decline. ### Rules for declaring -The four rules below are stated rather than implied because each was discovered by four +The five rules below are stated rather than implied because each was discovered by four implementations answering the same question differently. They are what makes two reports comparable. +**A capability the language's SDK cannot express is refused by the implementation, not left to +adopters.** Two exist today: `@large-integers` where the integer accessor is 32-bit, and +`@numeric-coercion` where the language has a single numeric type. Neither says anything about a +provider -- no provider in that language can satisfy them, and none ever will until the SDK changes. + +Leaving it to adopters means every adopter in that language has to know a fact about their language +and remember to act on it. That is not hypothetical: in one implementation three separate suites +each left the same capability undeclared, each with its own comment explaining the same property of +the language. Three places to get it right, and a single wrong one puts a claim in a report that no +scenario could have verified -- the exact failure the reserved-capability rules prevent, reached by +another route. + +So the implementation refuses it at configuration time, as it refuses a reserved capability. **The +two refusals are not the same thing and their skip reasons must differ.** A reserved capability is +global and temporary: no scenario anywhere carries the tag, and the reservation expires the moment +the specification adds one. An inexpressible capability is one language's and permanent: the +scenarios exist and pass elsewhere. A reader who sees a capability absent from a report needs to know +which of *"this provider declined"* and *"no provider in this language can be asked"* they are +looking at, because only the first says anything about the provider. + **A scenario is gated by every capability tag that applies to it, including tags inherited from its feature.** Tags compose: a tag on a `Feature` applies to every scenario in it, and a scenario carrying its own tag is gated by both. A scenario runs only when **all** of its capabilities are From 155e8960a9ff51313d71405e222cb1eea09ceeaf Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 11:11:10 +0200 Subject: [PATCH 40/63] fix: the canonical flag set still told adopters every scenario expects STATIC Two $comment blocks survived the reason change: "Every scenario expects reason STATIC" and "which is what lets every other scenario expect reason STATIC". The appendix was corrected in c342461a and this asset was not, which matters more than the usual stale comment -- the flag set is copied verbatim into four languages' packaged artifacts, so it is what a backend author reads when seeding a harness, and it now described a suite that no longer exists. Both now say what is actually true: only targeting-key-flag has a rule, and the absence of rules elsewhere is what lets a provider declaring @standard-reasons expect STATIC there rather than TARGETING_MATCH. The load-bearing property is unchanged -- it was never really about the reason, it was about not seeding targeting onto anything else. Found by the Java implementation while re-pinning, which is the argument for four implementations in one sentence: the asset is consumed by all of them and was wrong in a way no single language's tests could notice. Signed-off-by: Simon Schrottner --- .../assets/provider-tck/flags/canonical-flags.json | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json index 9876547d0..fec6e6ecd 100644 --- a/specification/assets/provider-tck/flags/canonical-flags.json +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -9,8 +9,10 @@ "", "Two things are load-bearing and easy to get wrong:", " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", - " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", - " tests the provider's mapping of a response, not the backend's evaluation logic.", + " * Only 'targeting-key-flag' has a targeting rule. Everything else resolves the same way", + " whatever the context, because the TCK tests the provider's mapping of a response, not", + " the backend's evaluation logic. That is also what lets a provider declaring", + " @standard-reasons expect STATIC rather than TARGETING_MATCH for those flags.", "", "Three more are easy to lose in translation, because a seeding step that 'cleans up' values", "destroys exactly what they test:", @@ -180,8 +182,8 @@ "targeting-key-flag": { "$comment": [ "The one flag in this set with a targeting rule. Everything else resolves to its", - "defaultVariant whatever the context, which is what lets every other scenario expect", - "reason STATIC.", + "defaultVariant whatever the context, which is what lets a provider declaring", + "@standard-reasons expect STATIC for those flags and TARGETING_MATCH or DEFAULT here.", "", "The rule is specified by its behaviour, not by this encoding: resolve variant 'hit'", "when the evaluation context's targeting key is exactly the uuid below, and 'miss'", From 537bf87fdef14f0decc706f628d28968b3fecca2 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 17:33:52 +0200 Subject: [PATCH 41/63] fix: the numeric-coercion note taught the shape the deviation rule forbids "A provider withholding this capability should say which it is -- a deliberate choice, or a tracked defect -- and a conformance report has knownDeviations for the second." That is withhold-plus-deviate, and a few sections above, the known-deviation rule says to prefer declaring the capability and letting the scenario fail, because a withheld capability plus a deviation asserts a defect at something nobody asked. Two parts of one document disagreeing is bad enough; this was also the part an implementer would read to learn the shape, and two of the four adoptions followed it there. Measurement settles which is right: flagd widens 10 to 10.0 correctly and narrows 0.5 to 0 with no error code, in both resolvers and in more than one language. It does coerce and gets one direction wrong, which is exactly what a skip cannot express. Withholding is still right for a provider that cannot attempt the behaviour at all -- a single numeric type, where the distinction does not exist to get wrong. Also adds the carve-out a TCK implementation's own self-tests need, which the rule did not cover. Those suites run the scenarios against an SDK's in-memory provider as a fixture, produce no report about a third party, and run in the ordinary build where a permanently failing scenario is a broken build rather than a finding -- nobody downstream can act on it, because the fix is an SDK release away. They may withhold, on the condition that the defect is pinned by a test of its own so the skip is not the only record. An adoption has no such licence. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 27 ++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 2c6f086af..c1e56eafd 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -305,6 +305,19 @@ gets it wrong, withdrawing the capability replaces a failing scenario with a ski behind something that looks deliberate — which is the outcome this field exists to prevent, not one of its uses. A conformance report is not improved by having fewer failures in it. +**A TCK implementation's own self-tests are the one place where withholding to stay green is +acceptable**, and it is worth saying so because the rule above otherwise forbids it. Those suites run +the conformance scenarios against an SDK's in-memory provider as a fixture for the harness; they are +not an adoption and they produce no report about a third party. They also run in the implementation's +ordinary build, where a permanently failing scenario is a broken build rather than a finding — and +nobody downstream can act on it, because the defect belongs to the SDK and the fix is a release away. + +So a self-test may leave a capability undeclared for a defect it has identified, on one condition: +**the defect is pinned by a test of its own**, so that the behaviour is still asserted and the skip +is not the only record. Go's in-memory self-test does this for `@disabled-flags` against a +`memprovider` defect fixed upstream but unreleased. An adoption has no such licence: it exists to +report on a provider, and a skip there is a claim about that provider. + **`summary` is required; `issue` is not.** A deviation whose summary is empty records that something is wrong without saying what, which leaves a reader worse off than the bare skip or failure it accompanies. An untracked deviation is worth declaring even so: naming the defect is what separates @@ -418,9 +431,17 @@ therefore on the second. What remains true is that the observed behaviour is bad for users: flagd narrows `0.5` to `0` with no error code at all, in Go and in Java, in both resolvers, so an application receives a plausible value and no signal. That is being fixed in -[open-feature/flagd#1996](https://github.com/open-feature/flagd/issues/1996). A provider withholding -this capability should say which it is — a deliberate choice, or a tracked defect — and a conformance -report has `knownDeviations` for the second. +[open-feature/flagd#1996](https://github.com/open-feature/flagd/issues/1996). **A provider in that +position should declare the capability and let the scenario fail**, with a `knownDeviations` entry +beside it: it does coerce, and gets one direction wrong, which is precisely what a skip cannot +express. An earlier revision of this paragraph said such a provider should withhold the capability +and explain which it was, and that contradicted the known-deviation rule a few sections above — the +one place a reader would look to learn the shape. Two of the four implementations followed it into +exactly the combination that rule exists to discourage, so the wording is corrected here rather than +worked around there. + +Withholding remains right for a provider that **cannot attempt** the behaviour — one whose SDK has a +single numeric type, where the distinction does not exist to get wrong. Both halves of the rule have scenarios. The lossy half asks for `float-flag` (`0.5`) as an integer and expects `TYPE_MISMATCH`; the lossless half asks for `integral-float-flag` (`10.0`) as an integer From 7587a1bb9524dfbb4aee7fdbb37b4cbfccdd9bf3 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 18:09:27 +0200 Subject: [PATCH 42/63] feat: the unit of a declaration decision is the scenario, not the tag Three implementations answered one question three ways, and the question was never written down: what do you declare when the backend under test cannot serve a flag some scenario needs? Go and JavaScript withheld @large-integers; Python declared it and reported a failure that was the backend's. Python also withheld @numeric-coercion on the same reasoning, and Go withheld it while recording a deviation against it. The rule that resolves all of them: declare when at least one scenario gating the tag can actually be put to the provider, withhold only when none can. @large-integers has one scenario and the reference backend serves no flag for it, so nothing can be established. @numeric-coercion has three and a backend missing one flag can still be asked the other two. That difference is not academic. Declaring @numeric-coercion is how one provider's two resolvers were found to disagree with each other -- one coercing correctly, one silently narrowing 0.5 to 0 -- which six passes of reading the source had not revealed. Withholding it to avoid one fixture failure would have cost both answers. Two consequences are stated with it. A scenario failing because its fixture is absent is not a provider defect and must not be recorded as one, or the report accuses the provider of the backend's gap. And a capability withheld for a backend gap is temporary in a way one withheld by choice is not -- it needs a note saying why, or it outlives its reason. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index c1e56eafd..57e02356a 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -241,9 +241,28 @@ to decline. ### Rules for declaring -The five rules below are stated rather than implied because each was discovered by four +The six rules below are stated rather than implied because each was discovered by four implementations answering the same question differently. They are what makes two reports comparable. +**Declare a capability when at least one scenario gating it can actually be put to the provider; +withhold it only when none can.** The unit of this decision is the *scenario*, not the tag — which is +the part that is easy to miss, and the part three implementations got wrong in three different +directions. + +The case that forces it is a backend that does not serve a flag some scenario needs. `@large-integers` +has exactly one scenario, and it asks for a flag the reference backend does not serve, so nothing +about that capability can be established and withholding is right. `@numeric-coercion` has three, and +a backend missing one flag can still be asked the other two — so withholding it hides two answers to +save one failure. That is not hypothetical: declaring it is how one provider's two resolvers were +found to *disagree with each other*, one coercing correctly and one not, which no amount of reading +the source had revealed. + +Two consequences worth stating. A scenario that fails because the backend cannot serve its fixture is +**not** a provider defect and must not be recorded as one — say so in the deviation's summary, or the +report accuses the provider of the backend's gap. And a capability withheld for a backend gap is +**temporary** in a way one withheld by choice is not: it should be revisited when the backend gains +the fixture, so note why, or it will outlive its reason. + **A capability the language's SDK cannot express is refused by the implementation, not left to adopters.** Two exist today: `@large-integers` where the integer accessor is 32-bit, and `@numeric-coercion` where the language has a single numeric type. Neither says anything about a From 9632f49b269ea0be13862685d4b7366a871ad9d2 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 19:01:51 +0200 Subject: [PATCH 43/63] fix: rule six decides whether a question is askable, not whether it is owed As written the rule said "declare when at least one scenario gating it can be put to the provider", with no condition on it. Read literally that forces a declaration wherever the scenarios are reachable -- including where the specification permits a provider to decline outright. @numeric-coercion is defined by no requirement, so a provider that simply does not coerce is entitled to withhold it, and the rule as written would have turned that permitted choice into a manufactured failure. That is the mirror image of the mistake the rule exists to prevent. Found by the Go implementation, whose self-tests withhold @numeric-coercion for exactly that reason while its flagd adoption declares it: memprovider does not coerce by design, flagd does coerce and gets one direction wrong. Two bullets four lines apart, resting on different grounds, and only one of them is rule six's business. So the rule now opens with the condition it always had implicitly: it applies once a provider is attempting the capability. Whether the provider owes an answer at all is Decision 4's question and comes first. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 57e02356a..91005bf5b 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -244,10 +244,17 @@ to decline. The six rules below are stated rather than implied because each was discovered by four implementations answering the same question differently. They are what makes two reports comparable. -**Declare a capability when at least one scenario gating it can actually be put to the provider; -withhold it only when none can.** The unit of this decision is the *scenario*, not the tag — which is -the part that is easy to miss, and the part three implementations got wrong in three different -directions. +**Once a provider is attempting a capability, declare it when at least one scenario gating it can +actually be put to the provider, and withhold only when none can.** The unit of this decision is the +*scenario*, not the tag — which is the part that is easy to miss, and the part three implementations +got wrong in three different directions. + +The opening clause is a real condition and not throat-clearing. This rule decides *whether the +question is askable*; it does not decide whether the provider owes an answer, and that question comes +first. Where the specification permits declining — `@numeric-coercion` is not defined by any +requirement, so a provider may simply not coerce — withholding is the honest report however askable +the scenarios are. Applying this rule to that case turns a permitted choice into a manufactured +failure, which is the mirror image of the mistake it exists to prevent. The case that forces it is a backend that does not serve a flag some scenario needs. `@large-integers` has exactly one scenario, and it asks for a flag the reference backend does not serve, so nothing From 1ebbce2aba09fa3d234a7ddb336033fbd697ba05 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 20:20:36 +0200 Subject: [PATCH 44/63] docs: give the conformance suite its own step, and mark the gating advice provisional Two changes to the CI section, deliberately separated because one is settled and the other is not. Settled: a conformance suite gets a step of its own rather than being folded into a provider's existing end-to-end suite. The reason is what a failure says, not how long the run takes. A dedicated step going red reports that conformance failed; the same scenarios inside an e2e suite report that a test failed, and a reader has to go and find out which kind. The two also mean different things by failure -- an e2e suite is expected green, so a failure is a regression, while a conformance suite fails scenarios by design wherever a knownDeviation is declared, and that failure is correct output until the defect ships a fix. Sharing one signal between "you broke something" and "this is the known state" ends with somebody silencing the informative half. Four implementations answered this differently and none of them decided it; each landed where its build system pushed it. Not settled: whether the suite can be a required gate. That advice rests on the premise that a conformance run is unavoidably red, which is only true because we have been judging it by "zero failures". Judged instead by whether its results match its declaration, a healthy adoption is green in its steady state with its deviations intact. That is under discussion in open-feature/spec#417 and the paragraph now says so, so nobody builds on it before it is decided. The separation advice holds either way. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 91005bf5b..e7b5ca80d 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -595,6 +595,24 @@ Provide a single documented command that runs the suite deliberately, and keep t *compiling* in the default build even when it does not execute — a conformance suite that has quietly stopped building against its own harness is a worse failure than one that runs and fails. +**Give it a step of its own, rather than folding it into an existing end-to-end suite.** This holds +whatever is decided about gating, and the reason is what the failure *says* rather than how long it +takes. A dedicated step that goes red reports that **conformance** failed; the same scenarios inside +a provider's own e2e suite report that a test failed, and a reader has to go and find out which kind. +The two also differ in what a failure means: an e2e suite is expected green, so a failure is a +regression, while a conformance suite carries failures by design — a declared `knownDeviation` fails +its scenario deliberately, and that failure is correct output until the defect is fixed upstream. +Sharing one signal between "you broke something" and "this is the known state" reliably ends with +somebody silencing the informative half. + +> **The gating half of this section is provisional.** It rests on the premise that a conformance +> run's output is unavoidably red, and therefore cannot be a required gate. That premise is under +> discussion in [open-feature/spec#417](https://github.com/open-feature/spec/issues/417): if a run is +> judged by whether its **results match its declaration** — every failure covered by a declared +> deviation, every skip gated by an undeclared capability, and every declared deviation still +> failing something — then a healthy adoption is green in its steady state, deviations included, and +> the suite can be a required gate after all. The separation advice above is unaffected either way. + ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, From b761af77922e35784993a475cf6bf7dfa2655d4f Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 20:46:49 +0200 Subject: [PATCH 45/63] fix: the separation argument rested on the step running, which it does not The paragraph justified a dedicated step with "a dedicated step that goes red reports that conformance failed" -- a property that only exists if something invokes the step. The same section says, provisionally, that nothing should: the suite is excluded from the default build pending the gating discussion. So the argument depended on a condition the surrounding advice forbids, and an audit of the two implementations that already have a separate target found exactly that hole -- both satisfy "do not fold it into the e2e suite", neither has a conformance job, so nothing goes red because nothing runs. Rewritten to argue from what a result means rather than from a failure being observed, and to say plainly that this is about a build target rather than a CI job. The two are easy to conflate. A step no pipeline invokes still earns its place: a maintainer running it by hand gets an unambiguous answer instead of a mixed one, and if the gating question resolves towards running it, the thing to gate on already exists and is already scoped to the right scenarios. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index e7b5ca80d..585107944 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -596,9 +596,15 @@ Provide a single documented command that runs the suite deliberately, and keep t quietly stopped building against its own harness is a worse failure than one that runs and fails. **Give it a step of its own, rather than folding it into an existing end-to-end suite.** This holds -whatever is decided about gating, and the reason is what the failure *says* rather than how long it -takes. A dedicated step that goes red reports that **conformance** failed; the same scenarios inside -a provider's own e2e suite report that a test failed, and a reader has to go and find out which kind. +whatever is decided about gating, and the reason is what a result *means* rather than how long it +takes. A dedicated step reports on **conformance**; the same scenarios inside a provider's own e2e +suite report that a test passed or failed, and a reader has to go and find out which kind. + +Note that this is about a build target, not a CI job, and the two are easy to conflate — a suite can +have a step of its own that no pipeline invokes, which is the position every implementation is in +while the gating question below is open. The step still earns its place: a maintainer running it by +hand gets an unambiguous answer rather than a mixed one, and if gating is resolved in favour of +running it, the thing to gate on already exists and is already scoped correctly. The two also differ in what a failure means: an e2e suite is expected green, so a failure is a regression, while a conformance suite carries failures by design — a declared `knownDeviation` fails its scenario deliberately, and that failure is correct output until the defect is fixed upstream. From 00cd16e7f02d32a365dcbf3418c9098c1d9c1112 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 21:07:18 +0200 Subject: [PATCH 46/63] feat: a conformance suite must be separable without a naming convention The appendix asked for a step of its own and stopped there, so four implementations put the files in four places and three of them selected the suite by test name: a skip filter on "Conformance", a Surefire pattern on *TckTest.java, a nested directory under the end-to-end suite. Only one filed it as a sibling. Stated as a property rather than a layout, because where files live is idiomatic per language and per repository, while the thing that goes wrong is not. A naming convention works until a test is renamed, and then it fails in the direction that hides the problem: the suite stops being selected, the step goes green having run nothing, and a pull request keeps quoting a tally from the last run that happened. That is the same shape as the CI exclusions every implementation believed it had and did not. The evidence is what each approach needed to be trustworthy. Selecting by name required an AST parser asserting that the set of tests calling the runner equals the set matching the pattern, mutation-tested in both directions -- and a third mutation, because such a guard otherwise passes vacuously the day it goes blind. Selecting by directory required nothing. Nesting inside the end-to-end directory gets its own sentence because it is the same mistake one level down: it says the conformance suite is a kind of end-to-end test, which is precisely what a separate step exists to deny, and where the end-to-end suite is its own module it also drags container libraries into tests that never use them. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 585107944..3607d7cc6 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -605,6 +605,23 @@ have a step of its own that no pipeline invokes, which is the position every imp while the gating question below is open. The step still earns its place: a maintainer running it by hand gets an unambiguous answer rather than a mixed one, and if gating is resolved in favour of running it, the thing to gate on already exists and is already scoped correctly. + +**Keep the conformance suite separable from the provider's other suites, and do not select it by +naming convention.** The unit does not matter — a directory, a module, a project, whatever the +language's tooling selects on — but it should be possible to run the conformance suite, and only it, +without enumerating or pattern-matching individual test names. + +Naming conventions work until a test is renamed, and then they fail in the direction that hides the +problem: the suite stops being selected, the step goes green having run nothing, and the tally in a +pull request keeps quoting numbers from the last time it did run. Every language that selected by +name had to build something to defend the convention — an AST parser asserting that the set of tests +calling the runner equals the set matching the name pattern, mutation-tested in both directions. The +languages that selected by directory needed nothing, because a file is in it or it is not. + +Filing the conformance suite *inside* the provider's end-to-end directory is the same mistake one +level down. It says the suite is a kind of end-to-end test, which is what a step of its own exists to +deny — and where the end-to-end suite is its own module, it also drags the conformance suite's +dependencies, container libraries and all, into tests that never use them. The two also differ in what a failure means: an e2e suite is expected green, so a failure is a regression, while a conformance suite carries failures by design — a declared `knownDeviation` fails its scenario deliberately, and that failure is correct output until the defect is fixed upstream. From 1b32a00cd4419f48d783f9541d40647746f75323 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 22:01:34 +0200 Subject: [PATCH 47/63] feat: a suite must not be able to run against assets it did not fetch Three of the four implementations could, by three different routes, and one of them did: a full adoption suite ran against the previous pin's feature files and reported a tally byte-identical to the run before it. Nothing failed, nothing warned, and it was caught only because someone compared two numbers that should have differed. The cause is the same wherever a copy is involved. Moving a pin updates the recorded revision, not the working tree the build copies from, so the two disagree silently and the copy wins. A rebase does the same thing: it moves the gitlink and leaves the checkout behind. Recorded here because it was recorded nowhere durable -- it surfaced in two languages' working notes during a rewrite and would have been lost with them, which is a poor fate for the only evidence that the failure mode is real. It is also the same shape as the CI exclusions every implementation believed it had and did not: a structural property everyone assumed held, in four places, none of them checked. Says to wire the fetch into the build rather than trusting whoever moves the pin to run a second command, notes that an immutable checksummed dependency avoids the problem outright, and warns that a guard catching one symptom is not a substitute -- a pin that changes only a scenario's content passes every such guard and still tests the wrong thing. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 3607d7cc6..4dc074c75 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -527,6 +527,21 @@ A TCK implementation is the language-specific harness around these three artifac inside the running stack through the control API, which is what `@stale` and `@unavailable` already require. + **Make it impossible to run the suite against assets you did not just fetch.** Three of four + implementations could, by three different routes, and one of them did: a full adoption suite ran + against the *previous* pin's feature files and reported a tally byte-identical to the run before + it — nothing failed, nothing warned, and it was caught only by someone comparing two numbers that + should have differed. The cause is the same everywhere a copy is involved: moving a pin updates + the recorded revision, not the working tree the build copies from, so the two disagree silently + and the copy wins. + + Wire the fetch into the build so the suite cannot run without it, rather than relying on whoever + moves the pin to remember a second command. Where the assets arrive as an immutable, checksummed + dependency the problem does not arise at all, and that is worth preferring. A guard that catches + one symptom — a declared capability no scenario carries, say — is worth having and is not a + substitute: a pin that changes only the *content* of a scenario passes every such guard and still + tests the wrong thing. + Wait for the stack by **asking the control API** whether it is ready, bounded by the startup timeout. After that, do not wait at all: a control endpoint that changes flag state owes the caller that the state is being served before it returns, so a suite that adds a delay of its own From 6a13f22c49f3c1753306980e4f49e3a37ccb3fbc Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 22:06:44 +0200 Subject: [PATCH 48/63] docs: record that @stale has no Docker-free coverage in any implementation Every other capability is covered somewhere without containers, by an in-memory or controllable provider standing in for the backend, so a regression in the step definitions or the capability gate shows up in an ordinary build. @stale is not, in any of the four implementations -- faking a provider that loses its backend and regains it needs a control that can disconnect, and the in-process control paths refuse connection operations by design. The consequence is the part worth recording: the stale/ready transition is exercised only by a containerised adoption, which is also the suite most likely to be excluded from a default build. So the one capability with no cheap coverage is the one whose only coverage is the easiest to switch off. Surfaced by the JavaScript implementation while rewriting its README, which had the observation in a code comment and nowhere else -- exactly the case this appendix asked implementers to report rather than let vanish. Naming a way to close it, since a gap with no route out reads as a permanent limitation: a controllable provider that can fake a disconnect, written carefully enough not to become a mock that passes whatever the provider does. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 4dc074c75..b587ae7ff 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -731,6 +731,16 @@ belong here rather than in any one implementation: The latter is cheaper and worth doing first, since attributes are where dropping is most likely. - **Setting and removing individual flags.** The control API can reset to a baseline and mutate one designated flag. Finer-grained flag manipulation would need new endpoints. +- **`@stale` is exercised only against a real backend.** Every other capability has at least one + implementation covering it without containers — an in-memory or controllable provider standing in + for the backend — so a regression in the step definitions or the gate is caught in an ordinary + build. `@stale` is not covered that way anywhere, in any of the four implementations, because + simulating a provider that loses its backend and regains it needs a control that can disconnect, + and the in-process control paths deliberately refuse connection operations. So the stale/ready + transition is only ever exercised by a containerised adoption, which is also the suite most likely + to be excluded from a default build. A controllable provider able to fake a disconnect would close + it, and would need care not to become a mock that passes whatever the provider does. + - **Caching.** Whether a stale provider keeps serving last-known values during an outage depends on whether it holds a local copy of the ruleset. The `@caching` tag is reserved; no scenarios yet. From ac4edbc9f3e2ae4f606c850501d91fa1da80ec4c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 23:08:45 +0200 Subject: [PATCH 49/63] fix: keep the adoption typechecked, not "compiling in the default build" The requirement said the suite must keep compiling in the default build. In at least one language that is impossible and always will be: where the ordinary build compiles what gets published, it is configured for library code -- no test globals, a different module target -- and an adoption that calls the test framework's own functions can never join it. Stated that way the requirement was unmeetable, which makes it a permanent known gap rather than a rule. What was actually wanted is that the adoption is typechecked against the harness by something that runs ordinarily, so a signature change in the harness cannot rot an adoption unnoticed. A typecheck scoped to the adoption satisfies that, and is a further argument for the adoption having a directory of its own -- a directory nothing else occupies is something a typecheck can be pointed at. Both ways of failing it are now named, because they are opposites and both were observed within an hour of each other. Excluding the adoption by path can remove it from the build as well as from the run, silently, since nothing fails when nothing is compiled. Removing an exclusion can pull the adoption into a build that cannot compile it, which at least fails loudly. The cheap check is the same either way and is worth stating: introduce a deliberate compile error in the adoption and confirm the ordinary build rejects it. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index b587ae7ff..43b19306c 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -606,9 +606,25 @@ both observed: - **An exclusion nobody wrote down.** It is then indistinguishable from an oversight, and the next person to touch the pipeline removes it or duplicates it. State it where an adopter will read it. -Provide a single documented command that runs the suite deliberately, and keep the suite -*compiling* in the default build even when it does not execute — a conformance suite that has -quietly stopped building against its own harness is a worse failure than one that runs and fails. +Provide a single documented command that runs the suite deliberately, and **keep the adoption +typechecked by something that runs ordinarily**, even though it does not execute — a conformance +suite that has quietly stopped building against its own harness is a worse failure than one that +runs and fails. + +"Something that runs ordinarily" rather than "the default build", because in at least one language +the default build cannot do it and never will. Where the ordinary build compiles what gets +*published*, it is configured for library code — no test globals, a different module target — and a +conformance adoption calling the test framework's own functions can never join it. There the +requirement is met by a typecheck scoped to the adoption instead, which is a further argument for +giving it a directory of its own: a directory nothing else occupies is something a typecheck can be +pointed at. + +Two ways to fail this, both observed, and they are opposites. Excluding the adoption by path can +remove it from the build as well as from the run — silently, because nothing fails when nothing is +compiled. Or the reverse: removing an exclusion can pull the adoption *into* a build that cannot +compile it, which at least fails loudly. Whichever shape applies, assert it rather than assume it — +the cheap check is to introduce a deliberate compile error in the adoption and confirm the ordinary +build rejects it. **Give it a step of its own, rather than folding it into an existing end-to-end suite.** This holds whatever is decided about gating, and the reason is what a result *means* rather than how long it From 783e0ff8e4a1a0dedc4e236e3d17d8656d04e197 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 07:10:57 +0200 Subject: [PATCH 50/63] docs: name the two steps the inherited vocabulary had to rename The appendix said the step vocabulary was inherited from the flagd test harness "wherever it was already provider-neutral", which implies exceptions and named none. There are two, and they are the entire diff: a stable provider and an unavailable one both carried "flagd" in the step text. Recording them here is what lets four language READMEs stop recording them separately -- it was the one piece of provenance the Java implementation could not find an upstream home for -- and it turns an unfalsifiable claim into a checkable one. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 43b19306c..bae0acc4a 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -98,6 +98,12 @@ The step vocabulary is inherited from the [flagd test harness](https://github.com/open-feature/test-harness) wherever it was already provider-neutral, so an existing suite ports with a near-zero diff. +Two steps were not neutral and were renamed, which is the whole of the diff: `Given a stable flagd +provider` became `Given a stable provider`, and its unavailable counterpart the same. They are named +here so that an implementer porting from the flagd harness knows the two places their existing step +definitions will not match, and so that "wherever it was already provider-neutral" can be checked +rather than taken on trust. + ### The canonical flag set A backend under test must serve an equivalent set under the configuration named `default`. The file From b7a0e74a79eec0b21277677c4ea0aa42139a7f7b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 07:16:24 +0200 Subject: [PATCH 51/63] docs: say that the configuration-change scenarios rest on cache invalidation The appendix already said CACHED is unasserted because no scenario repeats an evaluation without a configuration change in between. That clause is load-bearing in a second way it did not state: against a provider with a client-side cache -- flagd's RPC resolver runs an LRU by default -- the evaluation after the change answers from the cache unless invalidation works, so the configuration-change scenarios depend on it and nothing tests it directly. An adoption that does not disable the cache is therefore resting on an untested dependency, and a failure there reads as a configuration-change defect rather than an invalidation one. Not requiring adoptions to disable the cache, because a provider measured as it ships is the more useful measurement -- but the dependency should be visible to whoever reads the failure. Found by the Go flagd adoption, which does not disable the cache. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index bae0acc4a..6840db5da 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -415,6 +415,15 @@ produces them. `CACHED` needs a repeat evaluation, which nothing here performs w configuration change in between -- see the caching entry under known gaps. `STALE` needs a scenario asserting what a provider serves *during* an outage, which is the same gap. +**The "without a configuration change in between" is doing more work than it looks.** It is what +keeps the reason assertions correct against a provider that caches, and it means the +configuration-change scenarios silently depend on that provider's cache invalidation working: if it +did not, the evaluation after the change would answer from the cache and the scenario would fail +somewhere that says nothing about configuration change. No scenario tests invalidation directly, so +an adoption against a caching provider is resting on it untested. Adoptions are not required to +disable a client-side cache -- a provider evaluated as it ships is the more useful measurement -- +but an implementer should know the dependency is there before reading such a failure. + **Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without targeting, and `DISABLED` cannot be observed unless the backend distinguishes a disabled flag, so those scenarios carry `@targeting` and `@disabled-flags` as well. A provider declaring From 041e50efec3748eb2e4814dba60ab4e06c06423a Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 07:19:03 +0200 Subject: [PATCH 52/63] docs: the load-bearing properties are five, and the heading said four A property was added without the heading following it. Noticed independently by the Go and Java implementations, both of which read this file while deciding what their READMEs could stop restating -- which is the use the count matters for: an implementer checking they have covered all of them counts the heading, not the bullets. Signed-off-by: Simon Schrottner --- specification/assets/provider-tck/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 85ba14ee1..0d2174c4b 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -20,7 +20,7 @@ These validate a **provider** against a real backend. For assets that validate a A feature file that evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is meaningless without the control endpoint that produces the disconnect. Changing one without the others breaks the suite in every language at once. -## Four properties that are load-bearing +## Five properties that are load-bearing - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. - **Only `targeting-key-flag` has a targeting rule.** Every other enabled flag resolves to its default variant whatever the evaluation context, which is what lets the untargeted scenarios expect reason `STATIC`. Seeding targeting onto any other flag breaks them in every language at once. Its rule is specified by behaviour — resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`, `miss` otherwise — so express it however your backend expresses targeting. The flag, its variants and the uuid are the ones [flagd-testbed's `targeting.feature`](https://github.com/open-feature/flagd-testbed) already uses, on the same reasoning as the zero flags: a backend serving that harness already serves this. From 321e56ca556c5608208cd2f169170a3b15d434f3 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 08:34:56 +0200 Subject: [PATCH 53/63] docs: an absent @stale is a permitted choice, and the appendix never said so Requirement 5.1.1 offers two responses to a provider losing its backend, in consecutive sentences with the same modal: one unable to evaluate flags "can signal this by emitting a PROVIDER_ERROR event", and one that caches "can signal this by emitting a PROVIDER_STALE event". Can, twice. Neither is required. The appendix gated @stale without recording why, so the only thing telling an adopter that withholding it is legitimate was the tag's absence from a list -- and the tag's name reads like an obligation. @reinitialization already has this treatment for the same shape of requirement; @stale did not, and the omission had a cost: a provider going straight to ERROR was on its way to being written down as deviating from a contract that explicitly permits it. flagd is the worked example, and it differs by transport rather than by defect, which makes it a portability finding rather than a conformance failure. Found by the Go implementation while cutting restatements out of its adoption -- this was the one fact it could not find an owner for upstream. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 6840db5da..2a76d6497 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -238,6 +238,24 @@ requirement admits. It is worth keeping for the providers that do offer reuse, b client on shutdown while leaving an initialised flag set is easy to write and leaves the provider evaluating against a closed connection rather than failing outright. +`@stale` is gated for the same reason, and it is worth stating because the tag's name reads like an +obligation. [Requirement 5.1.1](./sections/05-events.md#requirement-511) offers two responses to a +provider losing its backend, in consecutive sentences and with the same modal: one unable to evaluate +flags *"can signal this by emitting a `PROVIDER_ERROR` event"*, and one that caches rule-sets or +previously evaluated flags *"can signal this by emitting a `PROVIDER_STALE` event"*. **Can, twice.** +A provider that goes straight to `ERROR` on connection loss is exercising the first option, not +failing the second — and it is the safer of the two, since it is not quietly serving cached values +while disconnected. + +So an absent `@stale` is a design choice and **must not be recorded as a known deviation**. The flagd +provider is the worked example and it differs by transport: its in-process resolver emits +`PROVIDER_STALE` on connection loss and escalates to `PROVIDER_ERROR` when a retry grace period +expires, while at least one language's RPC resolver emits `PROVIDER_ERROR` directly and never +`PROVIDER_STALE`. Both are conformant. That two transports of one provider answer differently is +worth knowing — an application switching resolver stops receiving stale events — but it is a +portability finding, not a conformance failure, and the report should carry it as an undeclared +capability rather than a defect. + The design rule behind this: **a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all** — and, learned later and at some cost, one that reports a permitted choice as a failure is not much better. A TCK implementation must report unsupported From daf1b233fc53de84d82f136f0e3165cae7bb22e5 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 12:54:42 +0200 Subject: [PATCH 54/63] fix: a backend that returns before it serves is the backend's defect, not the adoption's This appendix said a state-changing control endpoint must not return until the state is being served, and then, twice, told an adopter stuck with a backend that breaks it to put the wait in the adoption. Stating a rule and licensing a local workaround for violations of it undercuts the rule. It is also measurably the wrong instrument. One backend defect becomes every adoption's problem, solved once per language; and the adoptions that implement the workaround report stable results while the ones that do not report flapping ones against the same backend -- so the suite reports a difference in harness behaviour as a difference in provider behaviour. Observed directly: of the adoptions run against one testbed today, the one carrying a settle-wait read a clean floor while the others bounced between 2 and 28 failures, and the difference was the wait rather than the provider. The wait belongs in the backend. Until it is fixed there the suite fails, the failures are read against a documented floor, and a run is repeated before an extra failure is attributed to the provider -- a race moves between scenarios, a defect does not. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 29 +++++++++++++------ 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 2a76d6497..f03282ec3 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -158,13 +158,23 @@ Two invariants are worth stating here because they are the ones a TCK implementa is a suite that flaps per provider rather than per backend, which is the most misleading shape a conformance failure can take. - A suite must not paper over a backend that breaks this. A fixed delay after every control call - buys silence, not correctness: it hides the defect from the one consumer positioned to notice, - and it is un-tunable, because the window it covers is a property of the backend and not of the - suite. Where an adopter is stuck with such a backend, the wait belongs in **that adoption**, set - explicitly and citing the defect, so that it reads as a named workaround for a specific backend - and disappears when the backend is fixed — not as a constant buried in the shared harness where - every future adopter inherits it without knowing why. + A suite must not paper over a backend that breaks this, and **that includes an adoption**. A delay + buys silence, not correctness: it hides the defect from the one consumer positioned to notice, and + it is un-tunable, because the window it covers is a property of the backend and not of the suite. + + An earlier revision of this appendix said the wait belonged in the adoption, named and citing the + defect. That was wrong, and the reason is measurable. One backend's defect then becomes every + adoption's problem, solved N times in N languages; and the adoptions that implement the workaround + report stable results while the ones that do not report flapping ones, **against the same backend**. + The suite then reports a difference in harness behaviour as though it were a difference in provider + behaviour, which is the one thing a conformance suite must never do. It was observed exactly that + way: of two adoptions against one testbed, the one carrying a settle-wait looked clean and the ones + without it bounced between 2 and 28 failures. + + **So the wait belongs in the backend.** A backend that returns before it serves has a defect to be + filed and fixed where the backend lives. Until it is, the suite fails, the failures are read + against a documented floor, and a run is repeated before an extra failure is attributed to the + provider — a race hits a different scenario each time, a defect hits the same one. ## Capabilities: how a provider says what it cannot do @@ -579,8 +589,9 @@ A TCK implementation is the language-specific harness around these three artifac timeout. After that, do not wait at all: a control endpoint that changes flag state owes the caller that the state is being served before it returns, so a suite that adds a delay of its own is covering for a backend that broke its side of the contract — see the control API's invariants. - If an adopter's backend does break it, the wait belongs in that adoption, named and with the - defect cited, and not in the shared harness. + That holds for an adoption as much as for the shared harness: a backend that returns before it + serves is a defect to fix in the backend, and compensating for it anywhere in the suite makes that + adoption's results incomparable with every other adoption run against the same backend. 4. **Drive the backend only through the control API.** This is the part that makes the conformance claim portable: another language's TCK drives the same endpoints against the same stack and must get the same answers. From e5732f3876c4fee455ac4e09955eca18e4197c6a Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 13:01:16 +0200 Subject: [PATCH 55/63] fix: correct the evidence for why a compensating wait is the wrong shape The previous commit argued the case from an observation it did not check: that the adoption carrying a settle-wait read a clean floor while the others bounced, and that the wait was the difference. It was not. Removing that wait left its suite on exactly the same tally over four consecutive runs, and faster, because it had stopped polling for a condition that was already true. The correlation was real and the causal claim was mine to verify before writing it into a normative document. The principled argument is unchanged and does not need it -- and the measurement, correctly stated, is the stronger case: a compensating wait is hard to show is load-bearing, easy to leave in long after its defect is fixed, and certain to make two adoptions incommensurable meanwhile. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index f03282ec3..33c1eb436 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -163,13 +163,16 @@ Two invariants are worth stating here because they are the ones a TCK implementa it is un-tunable, because the window it covers is a property of the backend and not of the suite. An earlier revision of this appendix said the wait belonged in the adoption, named and citing the - defect. That was wrong, and the reason is measurable. One backend's defect then becomes every - adoption's problem, solved N times in N languages; and the adoptions that implement the workaround - report stable results while the ones that do not report flapping ones, **against the same backend**. - The suite then reports a difference in harness behaviour as though it were a difference in provider - behaviour, which is the one thing a conformance suite must never do. It was observed exactly that - way: of two adoptions against one testbed, the one carrying a settle-wait looked clean and the ones - without it bounced between 2 and 28 failures. + defect. That was wrong. One backend's defect becomes every adoption's problem, solved once per + language; and an adoption that compensates cannot be compared with one that does not, against the + same backend — so the suite risks reporting a difference in harness behaviour as though it were a + difference in provider behaviour, which is the one thing a conformance suite must never do. + + The one such workaround written under the old guidance turned out to buy nothing: removing it left + its suite on the same tally, four runs running, and faster for no longer polling. That is the + clearest argument against the shape. A compensating wait is hard to show is load-bearing, easy to + leave in place long after the defect it named is fixed, and certain to make two adoptions' + results incommensurable in the meantime. **So the wait belongs in the backend.** A backend that returns before it serves has a defect to be filed and fixed where the backend lives. Until it is, the suite fails, the failures are read From 8595a57dc764488f16e62cbf3777bf985fe3f17b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 13:38:10 +0200 Subject: [PATCH 56/63] feat: add "The runner", the contract four implementations kept and none wrote down The vocabulary and the artifacts decide what is asked. Nothing decided whether two languages' answers mean the same thing, and three things do: what the steps this suite adds actually assert, how long it waits, and how it registers the provider. All four implementations agree on every one of them -- because one author wrote them in parallel, which is not a mechanism a fifth can rely on. Each fact here was read out of the four branches rather than asserted: - Seven steps are this suite's own and four assert something other than their wording suggests. "no exception should have been thrown" is about a failure escaping to the caller, not about the evaluation succeeding. "the resolved details value should have changed" asserts a delta, because POST /change promises only that the value differs and not what it becomes. The split connection-lost/restored steps exist because the inherited self-healing form cannot express "assert stale, then reconnect" -- which is also why no scenario reaches POST /restart. The shutdown bound asserts that shutdown returns at all, not that it is fast. - Event 12s, ready 30s, startup 60s, identical in Go, Java, JavaScript and Python, and an explicit `within {int}ms` in a step always wins over the event default. The appendix previously mentioned timeouts once, in passing, and gave no values -- so "promptly" and "timed out" meant whatever each implementation chose. - A fresh provider per scenario under a domain derived from the suite name. Both halves matter: registering once per suite changes what the lifecycle scenarios establish, and a fresh domain per scenario leaks a provider each time, since registering into a domain replaces whatever was there. Also requires the extension point to expose the client under test -- without it an extension step can only build a second client, which resolves against a different provider and reports success having asked the provider under test nothing -- and requires the two run-integrity checks all four already implement. Alongside, three stale facts: the assets README listed five feature files and there are six (reason.feature), the reference implementation is tools/tck rather than tools/provider-tck, and the control API's known-gap paragraph said context passthrough was out of scope when the @targeting scenarios now cover the targeting key. What remains uncovered is every other attribute, which is what it now says. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 66 ++++++++++++++++++- specification/assets/provider-tck/README.md | 1 + .../provider-tck/openapi/control-api.yaml | 15 +++-- 3 files changed, 76 insertions(+), 6 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 33c1eb436..a1063cdbc 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -714,6 +714,70 @@ somebody silencing the informative half. > failing something — then a healthy adoption is green in its steady state, deviations included, and > the suite can be a required gate after all. The separation advice above is unaffected either way. +## The runner + +The vocabulary and the artifacts decide what is asked. This section decides whether two languages' +answers mean the same thing. It is here because four implementations agreed on all of it and none of +it was written down — they agreed because one author wrote them in parallel, which is not a mechanism +a fifth implementation can rely on. + +### Steps the suite adds + +Most of the step vocabulary is inherited (see above). Seven steps are this suite's own, and four of +them assert something other than what their wording first suggests. An implementation that binds them +literally will produce results that are not comparable with anyone else's. + +| Step | What it asserts | +| --- | --- | +| `no exception should have been thrown` | That no failure **escaped** the evaluation to the caller — a thrown exception, or a panic in a language without them. Not that the evaluation succeeded: typed evaluation must absorb every error into the returned details, so an error scenario proves both halves, the right error code *and* nothing escaping. The lifecycle scenarios reuse it for a repeated `shutdown()` and for `initialize()` against a reachable backend. | +| `the resolved value is remembered` | Records the current value for a later delta. | +| `the resolved details value should have changed` | That re-evaluation differs from the remembered value — **a delta, not a value**. `POST /change` promises only that `changing-flag` resolves differently; which value it changes to is vendor-defined, so asserting an absolute would bind the scenario to one backend and to how many times it had run. | +| `the connection is lost` / `the connection is restored` | An outage with an explicit start and end. The inherited harness has a self-healing `the connection is lost for {int}s`, which cannot express "assert the provider is stale, *then* reconnect" — the reconnect races the assertion. This is also why no shipped scenario reaches `POST /restart`. | +| `the provider is shut down` / `the provider is initialized again` | Call the provider **directly**, not through the SDK. | +| `the shutdown should have completed within {int}ms` | That shutdown **returned at all** rather than blocking on a backend that will never answer, which would hang the host application's own shutdown. The bound is deliberately generous; it is not a performance assertion. | + +### Timeouts + +These decide what "promptly" and "timed out" mean, so they decide comparability. An implementation +**MUST** use these defaults and **MUST** let an adopter override each one. + +| | default | bounds | +| --- | --- | --- | +| event | **12 s** | waiting for a lifecycle event a scenario asserts | +| ready | **30 s** | waiting for the provider to reach ready after registration | +| startup | **60 s** | bringing the whole backend stack up, before any scenario runs | + +**An explicit `within {int}ms` in a step always wins over the event default.** The step states a bound +the scenario is about; the default is only for steps that state none. + +### Registering the provider + +A **fresh provider per scenario**, registered under a **domain derived from the suite name** that the +adopter never names, and replaced at the end of the scenario so the previous one is shut down and its +connections released. + +Both halves matter. Registering once per suite would change what the lifecycle and reinitialisation +scenarios establish, since they assert against a provider whose state they control. A fresh *domain* +per scenario would leak: registering into a domain replaces and shuts down whatever was there, so a +new domain each time leaves every previous provider registered and running — for a provider holding a +network connection, one leaked connection per scenario. + +### Reaching the provider from an extension step + +An implementation that offers the extension point **MUST** also expose the client and provider under +test to an adopter's steps. Without it the only way to evaluate a flag from an extension step is to +build a second client, which resolves against a different provider — so the step tests the wiring and +reports success having asked the provider under test nothing. + +### Run-integrity checks + +Two failures are invisible from the results alone, so an implementation **MUST** fail the run on each: + +- **A reserved tag reached a collected scenario.** The tag is reserved because nothing carries it; if + something now does, the reservation has expired and the vocabulary is stale. +- **A declarable capability gates no collected scenario.** Either the assets are not the ones the + implementation thinks it shipped, or a capability has outlived its scenarios — and in both cases a + provider can declare it and be told nothing. ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, @@ -775,7 +839,7 @@ Two details of that check are worth recording, because both are easy to get wron ## Reference implementation -The first implementation is `tools/provider-tck` in +The first implementation is `tools/tck` in [open-feature/java-sdk-contrib](https://github.com/open-feature/java-sdk-contrib), adopted by the flagd provider for both its RPC and in-process resolvers. It is in review alongside this appendix. diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 0d2174c4b..4c06f2e8d 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -13,6 +13,7 @@ These validate a **provider** against a real backend. For assets that validate a | [`gherkin/events.feature`](./gherkin/events.feature) | configuration change, and the stale/ready transition across an outage | | [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one; shutdown | | [`gherkin/metadata.feature`](./gherkin/metadata.feature) | the provider identifies itself by name | +| [`gherkin/reason.feature`](./gherkin/reason.feature) | the standard resolution reasons, gated behind `@standard-reasons` | | [`flags/canonical-flags.json`](./flags/canonical-flags.json) | the flag set every scenario assumes | | [`openapi/control-api.yaml`](./openapi/control-api.yaml) | the HTTP surface a backend under test must expose | diff --git a/specification/assets/provider-tck/openapi/control-api.yaml b/specification/assets/provider-tck/openapi/control-api.yaml index 316772ccb..e293876ea 100644 --- a/specification/assets/provider-tck/openapi/control-api.yaml +++ b/specification/assets/provider-tck/openapi/control-api.yaml @@ -111,11 +111,16 @@ info: ## Known gap — evaluation context passthrough - There is currently no operation for asserting that an evaluation context sent - by the provider actually reached the backend intact. Verifying that requires - an echo mechanism (e.g. `GET /last-evaluation` returning the most recent - request the backend received). Until such an operation exists, context - passthrough is out of scope for the TCK. + The **targeting key** is covered: the `@targeting` scenarios resolve + `targeting-key-flag` with a matching and a non-matching context and get + different variants back, so a provider that drops the context entirely fails + them. + + What is still uncovered is **every other attribute**. A provider that forwards + the targeting key and silently discards the rest of the evaluation context + passes every scenario here. Closing that needs either an echo operation on + this API (e.g. `GET /last-evaluation`, returning the most recent request the + backend received) or a canonical flag whose rule keys on a custom attribute. license: name: Apache 2.0 From e0ac6f586046ec3766c3dc7c0656226091eb4224 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 13:43:03 +0200 Subject: [PATCH 57/63] feat: require the extension point rather than merely permitting it Asked for directly on spec#423: an explicit Appendix F requirement that the single-init lifecycle be open to an adopter's own scenarios and steps running in the same phase, so vendors extend the suite rather than maintaining a parallel harness for provider-specific features. The section already described the mechanism and the four properties that keep an extension from becoming a conformance claim, but only as something an implementation MAY offer. That is the wrong strength for exactly the reason given: it is nearly free in a runner that resolves steps dynamically and not free in one driven by declarative suite annotations, so the implementations that would skip it are precisely the ones whose adopters most need it. Adopters who cannot extend do not go without; they build the parallel harness this section exists to prevent. Also states the lifecycle requirement outright -- one backend start and teardown covering canonical and extension scenarios alike -- which was implied by "the same lifecycle" and is the part an implementation could satisfy in letter while missing in substance. All four implementations already offer one, so this records what they do rather than asking for new work. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index a1063cdbc..d6c060016 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -783,9 +783,18 @@ Two failures are invisible from the results alone, so an implementation **MUST** A provider often has behaviour this specification does not describe — flagd's fractional targeting, a vendor's own segment rules — and no way to test it inside this suite. The alternative an adopter reaches for is a parallel harness that reimplements provider registration, the readiness wait and -the per-scenario backend reset, and then drifts from the one here. So a TCK implementation **MAY** +the per-scenario backend reset, and then drifts from the one here. So a TCK implementation **MUST** offer an extension point: the adopter supplies feature files and step definitions, and they run -inside the same suite, against the same backend, in the same lifecycle. +inside the same suite, against the same backend, and **in the same lifecycle phase** — one backend +start and teardown covering canonical and extension scenarios alike. + +This is a requirement rather than a suggestion, and the reason is what happens when it is not. In a +runner that resolves steps dynamically the extension point is nearly free; in one driven by +declarative suite annotations it is not, and an implementation that skipped it would leave adopters +unable to add a feature file without redeclaring the whole set. Those adopters do not then go +without — they build the parallel harness, which is the outcome this section exists to prevent, and +by the time it exists the cost of retrofitting the extension point is paid by someone else. Requiring +it makes the cheap thing happen while it is still cheap. The mechanism is the implementation's own — a classpath scan, a `conftest.py`, two configuration fields — and this appendix does not prescribe one. What it does prescribe is the four properties that From bfd0db24fbccfa8491b9211a9c6ac5254c238ac5 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 13:49:17 +0200 Subject: [PATCH 58/63] docs: state the appendix's obligations normatively, in Appendix A's style Appendix F had grown real obligations on a TCK implementation -- offer an extension point, use these timeouts, fail the run on a reserved tag -- and stated every one of them inside rationale. A fifth implementer could not enumerate what they had to do without reading the whole document, which is the substance of the review feedback that it "took a long time to get through". Twenty obligations are now blockquoted statements with the RFC 2119 keyword bolded, each followed by the reasoning it used to be buried in. Nothing changed about what is required; the statements were already there as prose. Deliberately not numbered, and the reason is worth recording since numbering was the first instinct. No appendix in this specification uses numbered requirements -- A states five as blockquotes, B, D and E have none -- and specification.json builds its 135 rules from the numbered sections only, referencing no appendix at all. Numbers here would be inert: nothing extracts them and no SDK suite cites them. Worse, this appendix *tests* numbered requirements (2.2.1, 2.4.1, 5.1.1), so a second scheme beside them would put provider obligations and harness obligations in one namespace, which is a trap for exactly the implementer the numbering would be for. Adds "What an implementation must do" near the top: every obligation in one place, grouped and linked to the section that explains it, with a note saying why it carries no numbers. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 249 ++++++++++++------ 1 file changed, 169 insertions(+), 80 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index d6c060016..905032f5c 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -64,6 +64,44 @@ claims, and a language needs both suites to make both. - **The provider↔backend wire protocol.** How a provider talks to its backend is its own business. - **SDK behaviour.** That is Appendix B. +## What an implementation must do + +Every obligation this appendix places on a TCK implementation, in one place. Each links to the +section that explains it; the explanations are why, and none of them is a requirement. + +Nothing here is numbered. The numbered requirements in this specification are the provider contract — +`2.2.1`, `2.4.1`, `5.1.1` — and this suite exists to test them. These are obligations on the *harness*, +and giving them numbers beside the ones they test would put two different kinds of claim in one +namespace. + +**Artifacts and lifecycle** — [Implementing the suite in a language](#implementing-the-suite-in-a-language) + +- Package the Gherkin, the flag set and the control API document with the library. +- Make it impossible to run against assets the build did not fetch. +- Own the lifecycle and the container stack, so an adopter writes no test infrastructure. +- Start the stack once per suite; never restart it. +- Wait for readiness by asking the control API, and never wait after a control call. +- Drive the backend only through the control API. +- Run scenarios serially. + +**Capabilities** — [Capabilities](#capabilities-how-a-provider-says-what-it-cannot-do) + +- Report a scenario gated on an undeclared capability as skipped with its reason, never as passed. +- Refuse a reserved capability, and refuse one the language's SDK cannot express. +- Fail the run when a reserved tag reaches a collected scenario, or when a declarable capability gates + none. + +**The runner** — [The runner](#the-runner) + +- Use the timeout defaults, and let an adopter override each. +- Let an explicit `within {int}ms` take precedence over the event default. +- Register a fresh provider per scenario under a suite-derived domain, and replace it at the end. + +**Extension** — [Extending the suite](#extending-the-suite) + +- Offer an extension point running in the same lifecycle phase. +- Expose the client and provider under test to extension steps. +- Keep extension scenarios distinguishable from canonical ones. ## The three artifacts Conformance rests on three files, and they **travel together by necessity**. A feature file that @@ -210,7 +248,12 @@ declares which capabilities it supports. Scenarios whose tag is not declared are Untagged scenarios are mandatory and always run. A reserved tag is documented so the vocabulary has a place for the capability when scenarios exist, -but it **must not be declared** and must not appear in a conformance report's declaration. No +but it must not be declared. + +> A reserved capability **MUST NOT** be declared, and **MUST NOT** appear in a conformance report's +> declaration. A TCK implementation **MUST** refuse one an adopter names. + +No scenario carries it, so declaring it cannot be verified, cannot produce a skip, and tells a reader only that something was claimed and nothing examined -- the vacuous conformance claim this whole vocabulary exists to prevent. `@caching` is the only reserved tag left. @@ -319,6 +362,9 @@ the language. Three places to get it right, and a single wrong one puts a claim scenario could have verified -- the exact failure the reserved-capability rules prevent, reached by another route. +> A TCK implementation **MUST** refuse a capability its language's SDK cannot express, rather than +> leaving each adopter to withhold it. + So the implementation refuses it at configuration time, as it refuses a reserved capability. **The two refusals are not the same thing and their skip reasons must differ.** A reserved capability is global and temporary: no scenario anywhere carries the tag, and the reservation expires the moment @@ -547,60 +593,90 @@ the open question in [open-feature/spec#430](https://github.com/open-feature/spe ## Implementing the suite in a language -A TCK implementation is the language-specific harness around these three artifacts. What it owns: - -1. **Ship the artifacts.** Package the Gherkin, the flag set and the control API document with the - library so that adopting providers need no submodule of their own. -2. **Implement the step definitions** against the language's OpenFeature SDK, using its Cucumber (or - equivalent) runner. -3. **Own the lifecycle** — start the backend stack once, register the provider under test with the - SDK, await events, tear down — so that an adopting provider writes no test infrastructure. If a - provider author finds themselves adding lifecycle code, that is a defect in the TCK. - - **The container stack is part of that, and it is the part implementations get wrong.** An - adopter names a Docker Compose file, says which service and which container-internal ports the - provider connects to, and supplies a factory that builds a provider from a discovered endpoint. - Everything else — starting the stack, discovering the dynamically mapped host ports, building - the control client, waiting until the control API accepts commands, tearing down after the last - scenario — belongs to the TCK. Shipping only the control-API client and leaving orchestration to - the adopter satisfies the letter of this item and not its point: the orchestration is then - rewritten by every adopting provider, and it is the largest single piece of test infrastructure - in an adoption. - - Start the stack **once** per suite and never restart it. Container runtimes assign host ports - dynamically and do not reliably preserve them across a restart, so a restart silently - invalidates every provider already pointed at the old port. Backend unavailability is simulated - inside the running stack through the control API, which is what `@stale` and `@unavailable` - already require. - - **Make it impossible to run the suite against assets you did not just fetch.** Three of four - implementations could, by three different routes, and one of them did: a full adoption suite ran - against the *previous* pin's feature files and reported a tally byte-identical to the run before - it — nothing failed, nothing warned, and it was caught only by someone comparing two numbers that - should have differed. The cause is the same everywhere a copy is involved: moving a pin updates - the recorded revision, not the working tree the build copies from, so the two disagree silently - and the copy wins. - - Wire the fetch into the build so the suite cannot run without it, rather than relying on whoever - moves the pin to remember a second command. Where the assets arrive as an immutable, checksummed - dependency the problem does not arise at all, and that is worth preferring. A guard that catches - one symptom — a declared capability no scenario carries, say — is worth having and is not a - substitute: a pin that changes only the *content* of a scenario passes every such guard and still - tests the wrong thing. - - Wait for the stack by **asking the control API** whether it is ready, bounded by the startup - timeout. After that, do not wait at all: a control endpoint that changes flag state owes the - caller that the state is being served before it returns, so a suite that adds a delay of its own - is covering for a backend that broke its side of the contract — see the control API's invariants. - That holds for an adoption as much as for the shared harness: a backend that returns before it - serves is a defect to fix in the backend, and compensating for it anywhere in the suite makes that - adoption's results incomparable with every other adoption run against the same backend. -4. **Drive the backend only through the control API.** This is the part that makes the conformance - claim portable: another language's TCK drives the same endpoints against the same stack and must - get the same answers. -5. **Gate on capabilities** and report undeclared ones as skipped with a reason. -6. **Run scenarios serially.** Backend state is global to the suite; concurrent scenarios corrupt - each other, and the symptom looks like a flaky provider rather than a broken test. +A TCK implementation is the language-specific harness around these three artifacts. Each obligation +below is stated first and explained after; the explanations are why, not what. + +### Ship the artifacts + +> A TCK implementation **MUST** package the Gherkin scenarios, the canonical flag set and the control +> API document with the library, so that an adopting provider needs no submodule of its own. + +> A TCK implementation **MUST** make it impossible to run the suite against assets the build did not +> fetch. + +Three of four implementations could run against stale assets, by three different routes, and one did: +a full adoption suite ran against the *previous* pin's feature files and reported a tally +byte-identical to the run before it — nothing failed, nothing warned, and it was caught only by +someone comparing two numbers that should have differed. The cause is the same wherever a copy is +involved: moving a pin updates the recorded revision, not the working tree the build copies from, so +the two disagree silently and the copy wins. + +Wire the fetch into the build rather than relying on whoever moves the pin to remember a second +command. Where the assets arrive as an immutable, checksummed dependency the problem does not arise, +and that is worth preferring. A guard that catches one symptom — a declared capability no scenario +carries, say — is worth having and is not a substitute: a pin that changes only the *content* of a +scenario passes every such guard and still tests the wrong thing. + +### Implement the step definitions + +> A TCK implementation **MUST** implement the step definitions against its language's OpenFeature +> SDK, using that language's Cucumber or equivalent runner. + +### Own the lifecycle + +> A TCK implementation **MUST** own the suite lifecycle — starting the backend stack, registering the +> provider under test, awaiting events and tearing down — so that an adopting provider writes no test +> infrastructure. + +If a provider author finds themselves adding lifecycle code, that is a defect in the TCK. + +> A TCK implementation **MUST** own the container stack: given a Compose file, a service name and the +> container-internal ports the provider connects to, it starts the stack, discovers the dynamically +> mapped host ports, builds the control client, waits until the control API accepts commands, and +> tears down after the last scenario. + +This is the part implementations get wrong. Shipping only the control-API client and leaving +orchestration to the adopter satisfies the letter of the previous requirement and not its point: the +orchestration is then rewritten by every adopting provider, and it is the largest single piece of test +infrastructure in an adoption. + +> A TCK implementation **MUST** start the backend stack once per suite and **MUST NOT** restart it. + +Container runtimes assign host ports dynamically and do not reliably preserve them across a restart, +so a restart silently invalidates every provider already pointed at the old port. Backend +unavailability is simulated inside the running stack through the control API, which is what `@stale` +and `@unavailable` already require. + +> A TCK implementation **MUST** wait for the stack by asking the control API whether it is ready, +> bounded by the startup timeout, and **MUST NOT** wait after any control call. + +> A TCK implementation **MUST NOT** add a delay to compensate for a backend that +> returns before it serves, and an adoption **MUST NOT** either. + +A control endpoint that changes flag state owes the caller that the state is being served before it +returns — see the control API's invariants. A backend that breaks that has a defect to fix in the +backend, and compensating for it anywhere in the suite makes that adoption's results incomparable +with every other adoption run against the same backend. + +### Drive the backend only through the control API + +> A TCK implementation **MUST** drive the backend only through the control API. + +This is what makes the conformance claim portable: another language's TCK drives the same endpoints +against the same stack and must get the same answers. + +### Gate on capabilities + +> A TCK implementation **MUST** report a scenario gated on an undeclared capability as skipped with +> its reason, and **MUST NOT** report it as passed. + +### Run scenarios serially + +> A TCK implementation **MUST** run scenarios serially. + +Backend state is global to the suite; concurrent scenarios corrupt each other, and the symptom looks +like a flaky provider rather than a broken test. + ### Providers with no backend @@ -738,8 +814,10 @@ literally will produce results that are not comparable with anyone else's. ### Timeouts -These decide what "promptly" and "timed out" mean, so they decide comparability. An implementation -**MUST** use these defaults and **MUST** let an adopter override each one. +These decide what "promptly" and "timed out" mean, so they decide comparability. + +> A TCK implementation **MUST** use these timeout defaults, and **MUST** let an adopter override each +> one. | | default | bounds | | --- | --- | --- | @@ -747,14 +825,15 @@ These decide what "promptly" and "timed out" mean, so they decide comparability. | ready | **30 s** | waiting for the provider to reach ready after registration | | startup | **60 s** | bringing the whole backend stack up, before any scenario runs | -**An explicit `within {int}ms` in a step always wins over the event default.** The step states a bound -the scenario is about; the default is only for steps that state none. +> An explicit `within {int}ms` in a step **MUST** take precedence over the event timeout default. + +The step states a bound the scenario is about; the default is only for steps that state none. ### Registering the provider -A **fresh provider per scenario**, registered under a **domain derived from the suite name** that the -adopter never names, and replaced at the end of the scenario so the previous one is shut down and its -connections released. +> A TCK implementation **MUST** register a fresh provider for each scenario, under a domain derived +> from the suite name, and **MUST** replace it at the end of the scenario so the previous provider is +> shut down and its connections released. Both halves matter. Registering once per suite would change what the lifecycle and reinitialisation scenarios establish, since they assert against a provider whose state they control. A fresh *domain* @@ -764,29 +843,38 @@ network connection, one leaked connection per scenario. ### Reaching the provider from an extension step -An implementation that offers the extension point **MUST** also expose the client and provider under -test to an adopter's steps. Without it the only way to evaluate a flag from an extension step is to -build a second client, which resolves against a different provider — so the step tests the wiring and -reports success having asked the provider under test nothing. +> A TCK implementation **MUST** expose the client and the provider under test to an adopter's +> extension step definitions. + +Without it the only way to evaluate a flag from an extension step is to build a second client, which +resolves against a different provider — so the step tests the wiring and reports success having asked +the provider under test nothing. ### Run-integrity checks -Two failures are invisible from the results alone, so an implementation **MUST** fail the run on each: +Two failures are invisible from the results alone. -- **A reserved tag reached a collected scenario.** The tag is reserved because nothing carries it; if - something now does, the reservation has expired and the vocabulary is stale. -- **A declarable capability gates no collected scenario.** Either the assets are not the ones the - implementation thinks it shipped, or a capability has outlived its scenarios — and in both cases a - provider can declare it and be told nothing. +> A TCK implementation **MUST** fail the run when a reserved capability tag is carried by a collected +> scenario. + +The tag is reserved because nothing carries it; if something now does, the reservation has expired and +the vocabulary is stale. + +> A TCK implementation **MUST** fail the run when a declarable capability gates no collected scenario. + +Either the assets are not the ones the implementation thinks it shipped, or a capability has outlived +its scenarios — and in both cases a provider can declare it and be told nothing. ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, a vendor's own segment rules — and no way to test it inside this suite. The alternative an adopter reaches for is a parallel harness that reimplements provider registration, the readiness wait and -the per-scenario backend reset, and then drifts from the one here. So a TCK implementation **MUST** -offer an extension point: the adopter supplies feature files and step definitions, and they run -inside the same suite, against the same backend, and **in the same lifecycle phase** — one backend -start and teardown covering canonical and extension scenarios alike. +the per-scenario backend reset, and then drifts from the one here. + +> A TCK implementation **MUST** offer an extension point, through which an adopter supplies feature +> files and step definitions that run inside the same suite, against the same backend, and in the +> same lifecycle phase — one backend start and teardown covering canonical and extension scenarios +> alike. This is a requirement rather than a suggestion, and the reason is what happens when it is not. In a runner that resolves steps dynamically the extension point is nearly free; in one driven by @@ -800,10 +888,11 @@ The mechanism is the implementation's own — a classpath scan, a `conftest.py`, fields — and this appendix does not prescribe one. What it does prescribe is the four properties that keep an extension from quietly becoming a conformance claim. -**Extension scenarios must be distinguishable from canonical ones.** A results payload that mixes -them with no way to tell which is which lets an adopter's own passing scenarios flatter the -conformance result. Partitioning by path is enough, and it is what a consumer reads to separate the -two: +> Extension scenarios **MUST** be distinguishable from canonical ones in the results. + +A payload that mixes them with no way to tell which is which lets an adopter's own passing scenarios +flatter the conformance result. Partitioning by path is enough, and it is what a consumer reads to +separate the two: - a canonical feature is identified by its path **relative to this asset directory** — `gherkin/errors.feature`, not a path relative to the repository root; From 40d24c1e542561f13af434f8fef6154c44b7023d Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 13:56:31 +0200 Subject: [PATCH 59/63] docs: split the capability sections apart, and compress The @standard-reasons section had grown to 158 lines and half of it was about other capabilities: @numeric-coercion, the SDK-inexpressible rule and accessor width had all accumulated underneath it, where a reader looking for the coercion rule would never find them. They are now three sections with their own headings, and a misfiled paragraph about empty knownDeviations arrays has moved beside the other deviation rules. Compression alongside, mostly in @standard-reasons and the CI guidance: the thirteen-place provenance for why reasons became a capability is one parenthesis rather than a paragraph, the four-mechanism CI story is two clauses rather than five lines, and the compile-exclusion discussion loses a restatement. Also corrects the normative-status open question, which said nothing here is expressed normatively. That stopped being true in the previous commit. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 190 ++++++++---------- 1 file changed, 87 insertions(+), 103 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 905032f5c..e56d622a3 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -433,31 +433,28 @@ accompanies. An untracked deviation is worth declaring even so: naming the defec it from a withheld capability, and a declaration that merely omits the tag cannot say which of the two happened. Prefer a tracked one as soon as there is somewhere to point at. +**Emit `knownDeviations` only when there is at least one.** An empty array and an absent field are +not the same claim: stating none asserts that deviations were considered and none found, which no +suite can know on the adopter's behalf. Omit the field when the list is empty, and never synthesise +an empty one. + ### `@standard-reasons`: a claim, not an exemption -[Requirement 2.2.5](./sections/02-providers.md#requirement-225) is a `SHOULD`, and it goes further -than 2.2.4 does: it lets a provider populate `reason` with one of the listed values *"or some other -string indicating the semantic reason for the returned flag value"*. A provider whose backend -reports vendor-specific reasons is therefore conformant, and asserting an exact reason against it -would fail it for something the specification permits. - -An earlier revision of this suite did exactly that, in thirteen places across three feature files, -and recorded the narrowing here as a deliberate exception. It is not one any more, for two reasons. -It bought very little: every canonical flag resolves to a value distinct from the caller's default, -so a provider that silently falls back is already caught by the value assertion, and the reason only -said *why* it failed. And of the thirteen, five sat beside an error-code assertion that already -carries the `MUST`, while the other eight asserted `STATIC` -- the one reason the specification -genuinely leaves open. - -So the reasons now live in `reason.feature`, gated as a whole. **Declaring `@standard-reasons` is a -provider saying "I use the standard vocabulary with the standard meanings", and that file is what -checks the claim.** A provider that does not declare it loses nothing: its values, variants and error -codes are asserted everywhere else, on `MUST` requirements. What the declaration adds is something a -report's reader can act on -- anyone building telemetry, dashboards or debugging on `reason` can see -that the vocabulary was verified rather than assumed. - -This also settles a question the specification does not, without asking it to. The meanings below are -the content of an opt-in claim; they constrain nobody who does not make it. +[Requirement 2.2.5](./sections/02-providers.md#requirement-225) is a `SHOULD` that lets a provider +populate `reason` with a listed value *"or some other string indicating the semantic reason"*. A +provider reporting vendor-specific reasons is therefore conformant, and asserting an exact reason +against it would fail it for something the specification permits. So the reasons live in +`reason.feature`, gated as a whole, rather than asserted throughout. (An earlier revision asserted +them in thirteen places and recorded the narrowing as a deliberate exception; it bought little, since +every canonical flag resolves to a value distinct from the caller's default and the value assertion +already catches a silent fallback.) + +**Declaring `@standard-reasons` says "I use the standard vocabulary with the standard meanings", and +`reason.feature` checks the claim.** A provider that does not declare it loses nothing — its values, +variants and error codes are asserted elsewhere, on `MUST` requirements. What the declaration adds is +something a report's reader can act on: anyone building telemetry or debugging on `reason` can see the +vocabulary was verified rather than assumed. The meanings below are the content of an opt-in claim and +constrain nobody who does not make it. | Situation | Reason | | --- | --- | @@ -467,57 +464,47 @@ the content of an opt-in claim; they constrain nobody who does not make it. | The flag is disabled in the management system | `DISABLED` | | The evaluation failed, and an error code is reported with it | `ERROR` | -`STATIC` for the first row is the call worth flagging. `types.md` types `DEFAULT` as *"no dynamic -evaluation occurred **or** dynamic evaluation yielded no result"*, which a rule-less flag satisfies -as readily as `STATIC` does -- two providers can disagree here and both conform. A provider that -answers `DEFAULT` for a rule-less flag is not defective; it does not use the standard meanings, and -should not declare the tag. - -`ERROR` is the row where this suite's subject is blurred, and it is asserted anyway. The other four -rest on [Requirement 1.4.7](./sections/01-flag-evaluation.md#requirement-147), which makes the SDK -propagate the provider's reason — but only *"in cases of normal execution"*. Abnormal execution is -[1.4.9](./sections/01-flag-evaluation.md#requirement-149), a `SHOULD` on the **SDK** to "indicate an -error", and nothing requires the provider's reason to survive. So a passing `ERROR` scenario -establishes that the value reaching the application is coherent, not that the provider produced it. - -That is still worth asserting, because it is the pair that carries the meaning. The error code alone -is already covered for every provider — 2.2.7 and -[1.4.8](./sections/01-flag-evaluation.md#requirement-148) make it a `MUST`, it is a closed -enumeration, and `errors.feature` asserts it ungated. The reason alone could have been written by the -SDK. An evaluation reporting `FLAG_NOT_FOUND` with reason `STATIC` is incoherent whoever wrote it, -and that is what the pairing catches. - -`SPLIT`, `UNKNOWN`, `CACHED` and `STALE` are not asserted. The first two have no scenario that -produces them. `CACHED` needs a repeat evaluation, which nothing here performs without a -configuration change in between -- see the caching entry under known gaps. `STALE` needs a scenario -asserting what a provider serves *during* an outage, which is the same gap. - -**The "without a configuration change in between" is doing more work than it looks.** It is what -keeps the reason assertions correct against a provider that caches, and it means the -configuration-change scenarios silently depend on that provider's cache invalidation working: if it -did not, the evaluation after the change would answer from the cache and the scenario would fail -somewhere that says nothing about configuration change. No scenario tests invalidation directly, so -an adoption against a caching provider is resting on it untested. Adoptions are not required to -disable a client-side cache -- a provider evaluated as it ships is the more useful measurement -- -but an implementer should know the dependency is there before reading such a failure. +`STATIC` for the first row is the call worth flagging: `types.md` types `DEFAULT` as *"no dynamic +evaluation occurred **or** dynamic evaluation yielded no result"*, which a rule-less flag satisfies as +readily. Two providers can disagree here and both conform. One answering `DEFAULT` is not defective — +it does not use the standard meanings, and should not declare the tag. + +**`ERROR` asserts agreement, not authorship.** The other four rows rest on +[1.4.7](./sections/01-flag-evaluation.md#requirement-147), which makes the SDK propagate the +provider's reason — but only *"in cases of normal execution"*. Abnormal execution is +[1.4.9](./sections/01-flag-evaluation.md#requirement-149), a `SHOULD` on the **SDK**, and nothing +requires the provider's reason to survive. So a passing `ERROR` scenario establishes that the value +reaching the application is coherent, not that the provider produced it — which is still worth +asserting, because the pair carries the meaning. The error code alone is already a `MUST` for every +provider and asserted ungated; the reason alone could have been written by the SDK. `FLAG_NOT_FOUND` +with reason `STATIC` is incoherent whoever wrote it, and the pairing is what catches it. + +`SPLIT`, `UNKNOWN`, `CACHED` and `STALE` are not asserted: the first two have no scenario producing +them, and the last two need a repeat evaluation and an assertion about what a provider serves *during* +an outage — both under known gaps. + +**That "repeat evaluation" exclusion is load-bearing.** It is what keeps these assertions correct +against a provider that caches, and it means the configuration-change scenarios silently depend on +cache invalidation working: if it did not, the evaluation after the change would answer from the cache +and fail somewhere that says nothing about configuration change. Nothing tests invalidation directly, +so an adoption against a caching provider rests on it untested. Adoptions need not disable a +client-side cache — a provider measured as it ships is more useful — but an implementer should know +the dependency is there before reading such a failure. **Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without -targeting, and `DISABLED` cannot be observed unless the backend distinguishes a disabled flag, so -those scenarios carry `@targeting` and `@disabled-flags` as well. A provider declaring -`@standard-reasons` alone runs the rest and skips those two with their reason. - -`@variants` is the clearest case, and it was found the hard way. Every evaluation scenario asserted -a variant, which reads as obviously correct until a backend with no variant concept for a plain flag -is put under test: its evaluation response carries no such key, the provider never receives one, and -no seeding can produce one. Ten scenarios failed a conformant provider for something its author -could not fix, and nothing could be recorded as a known deviation because there was no capability to -hang one on. [Requirement 2.2.4](./sections/02-providers.md#requirement-224) is a `SHOULD` and -`types.md` types the field as optional; the suite was asserting a `MUST` neither of them states. +targeting and `DISABLED` cannot be observed unless the backend distinguishes a disabled flag, so those +scenarios also carry `@targeting` and `@disabled-flags`. A provider declaring `@standard-reasons` +alone runs the rest and skips those two with their reason. + +`@variants` is the same shape, found the hard way: every evaluation scenario once asserted a variant, +which reads as obviously correct until a backend with no variant concept is put under test — its +response carries no such key, no seeding can produce one, and ten scenarios failed a conformant +provider for something its author could not fix. +[2.2.4](./sections/02-providers.md#requirement-224) is a `SHOULD` and `types.md` types the field +optional; the suite was asserting a `MUST` neither states. -**Emit `knownDeviations` only when there is at least one.** An empty array and an absent field are -not the same claim: stating none asserts that deviations were considered and none found, which no -suite can know on the adopter's behalf. Omit the field when the list is empty, and never synthesise -an empty one. + +### `@numeric-coercion`: a borrowed rule, not a specified one `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**, and readers should not mistake it for one that does. @@ -569,6 +556,8 @@ are what stop it. A provider whose SDK has a single numeric type, such as JavaSc distinguish the cases at all, so it leaves the tag undeclared and the scenarios are reported as skipped with that reason. +### Capabilities a language cannot express + **Where a capability cannot hold in a language, this appendix is where that is recorded** — not a field in every report. `@numeric-coercion` in a single-numeric-type language and `@large-integers` on a 32-bit accessor are properties of the SDK, true of every provider written against it and for as @@ -708,46 +697,38 @@ path it used is not finished. ### Running the suite in CI -Guidance rather than a rule — a repository's pipeline is its own business — but the reasoning is the -same in every language, and leaving it unwritten produced four mechanisms and one unnoticed -consequence. +Guidance rather than a rule — a pipeline is a repository's own business — but leaving it unwritten +produced four different mechanisms and one unnoticed consequence. -**An adoption suite is not a required gate while real gaps remain.** Its honest output is red: it -fails on provider defects that are filed and unfixed, on backend fixtures that do not exist yet, and -on capabilities the provider has not implemented. A red result is the suite working. Making it -block a merge forces someone to silence it, and the cheapest way to silence a conformance suite is -to stop asking the question — withdraw a capability, delete an assertion, or pin an older backend. +**An adoption suite is not a required gate while real gaps remain.** Its honest output is red — filed +provider defects, missing backend fixtures, unimplemented capabilities — and a red result is the suite +working. Making it block a merge forces someone to silence it, and the cheapest way to silence a +conformance suite is to stop asking the question. So **exclude it from the default build, and make the exclusion explicit.** Two mistakes to avoid, both observed: -- **An exclusion that something else undoes.** The question is not whether an exclusion exists but - whether any profile, target or job re-enables it. In one language a CI profile cleared the - adopter's own exclusion property; in another the suite ran under a build tag applied to every - module; in a third the default test task simply collected it. All four languages believed their - suites were excluded and all four were running them, red, unwatched. -- **An exclusion nobody wrote down.** It is then indistinguishable from an oversight, and the next - person to touch the pipeline removes it or duplicates it. State it where an adopter will read it. +- **An exclusion something else undoes.** The question is not whether one exists but whether any + profile, target or job re-enables it. All four languages believed their suites were excluded; all + four were running them, red and unwatched, by four different routes. +- **An exclusion nobody wrote down**, which is indistinguishable from an oversight. State it where an + adopter will read it. Provide a single documented command that runs the suite deliberately, and **keep the adoption typechecked by something that runs ordinarily**, even though it does not execute — a conformance suite that has quietly stopped building against its own harness is a worse failure than one that runs and fails. -"Something that runs ordinarily" rather than "the default build", because in at least one language -the default build cannot do it and never will. Where the ordinary build compiles what gets -*published*, it is configured for library code — no test globals, a different module target — and a -conformance adoption calling the test framework's own functions can never join it. There the -requirement is met by a typecheck scoped to the adoption instead, which is a further argument for -giving it a directory of its own: a directory nothing else occupies is something a typecheck can be -pointed at. - -Two ways to fail this, both observed, and they are opposites. Excluding the adoption by path can -remove it from the build as well as from the run — silently, because nothing fails when nothing is -compiled. Or the reverse: removing an exclusion can pull the adoption *into* a build that cannot -compile it, which at least fails loudly. Whichever shape applies, assert it rather than assume it — -the cheap check is to introduce a deliberate compile error in the adoption and confirm the ordinary -build rejects it. +"Something that runs ordinarily" rather than "the default build", because where that build compiles +what gets *published* it is configured for library code — no test globals, a different module target — +and an adoption calling the test framework's own functions can never join it. A typecheck scoped to +the adoption meets the requirement instead, which is a further argument for the suite having a +directory of its own: a directory nothing else occupies is something a typecheck can be pointed at. + +Two opposite failures, both observed: excluding by path can remove the adoption from the build as well +as the run, silently, because nothing fails when nothing compiles; removing an exclusion can pull it +*into* a build that cannot compile it, which at least fails loudly. Assert it rather than assume it — +introduce a deliberate compile error and confirm the ordinary build rejects it. **Give it a step of its own, rather than folding it into an existing end-to-end suite.** This holds whatever is decided about gating, and the reason is what a result *means* rather than how long it @@ -990,6 +971,9 @@ belong here rather than in any one implementation: context, and the failure is silent. It wants its own capability tag, and until the control API has an echo operation a scenario can show only that reconciliation was signalled, not that the values that follow are the new context's. -- **Normative status.** Nothing in this appendix is currently expressed as a numbered requirement. - Whether the control API contract and the capability vocabulary should become normative sections is - a decision for the TSC. +- **Normative status.** The obligations on a TCK implementation are stated as normative blockquotes, + in the style Appendix A uses, and deliberately carry no numbers: `specification.json` builds its + rules from the numbered sections only, so numbers here would be inert, and this appendix *tests* + numbered requirements — a second scheme beside them would put provider obligations and harness + obligations in one namespace. Whether the control API contract and the capability vocabulary should + instead be promoted to a numbered section remains a decision for the TSC. From ba882dfb6e8e1b854becdf8973f35348055bc485 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Tue, 15 Sep 2026 21:53:41 +0200 Subject: [PATCH 60/63] style: the blank lines markdownlint wants in the appendix The compression pass left five violations, which is what the json-lint job was failing on: two headings with no blank line above (MD022), the list that runs straight into the first of them (MD032), and two double blanks where sections were taken out (MD012). Worth recording why only this file was reported, since it is not obvious: the job runs markdownlint over `**/*.md` from a shell with globstar off, so the pattern matches one directory level -- eight files, this appendix among them. Signed-off-by: Simon Schrottner --- specification/appendix-f-provider-conformance.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index e56d622a3..7e66c4d2c 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -102,6 +102,7 @@ namespace. - Offer an extension point running in the same lifecycle phase. - Expose the client and provider under test to extension steps. - Keep extension scenarios distinguishable from canonical ones. + ## The three artifacts Conformance rests on three files, and they **travel together by necessity**. A feature file that @@ -503,7 +504,6 @@ provider for something its author could not fix. [2.2.4](./sections/02-providers.md#requirement-224) is a `SHOULD` and `types.md` types the field optional; the suite was asserting a `MUST` neither states. - ### `@numeric-coercion`: a borrowed rule, not a specified one `@numeric-coercion` deserves a note, because it is the one capability here that **the specification @@ -666,7 +666,6 @@ against the same stack and must get the same answers. Backend state is global to the suite; concurrent scenarios corrupt each other, and the symptom looks like a flaky provider rather than a broken test. - ### Providers with no backend An in-memory, environment-variable or file-based provider has nothing to connect to and no control @@ -845,6 +844,7 @@ the vocabulary is stale. Either the assets are not the ones the implementation thinks it shipped, or a capability has outlived its scenarios — and in both cases a provider can declare it and be told nothing. + ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, From eeca96cbc5a1fa0f5756ae26517ce0dc7a39dd86 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Tue, 15 Sep 2026 22:04:37 +0200 Subject: [PATCH 61/63] feat: hold string representation out of the mandatory mismatch matrix A backend that stores flag values as strings satisfies the string accessor for every flag and has no mismatch to report, so four scenarios in the mandatory matrix could not be satisfied by a provider that was behaving reasonably. They move behind a new `@string-typing` capability: `boolean-flag`, `integer-flag` and `float-flag` requested as strings, and `object-flag` requested as a string -- that one tagged `@object` too, since a provider with no structured values cannot be asked at all. The matrix had justified keeping them mandatory on the grounds that "is a string a boolean?" has no defensible wrong answer. That holds for parsing a string into another type, which a provider chooses to do; it does not hold for rendering another type as a string, which an untyped backend does whether anyone chose it or not. The asymmetry was real and pointed the wrong way. The specification cannot settle it either way today. TYPE_MISMATCH appears exactly once, as a row in the error table, and no requirement obliges anyone to raise it; the only normative statement about value type is Requirement 1.3.4, which is a SHOULD and is on the client rather than the provider. So the rule is borrowed exactly as @numeric-coercion's is, and withholding the tag is not non-conformance. The question sits upstream of open-feature/spec#430: that one asks which accessor a number may satisfy, this one asks what a flag's type is when the backend has none. Found by a Flipt provider implementation written against this suite. The scenario count is unchanged at 19 -- four move from mandatory to gated. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 37 ++++++++++++++++ specification/assets/provider-tck/README.md | 2 +- .../provider-tck/gherkin/errors.feature | 43 ++++++++++++++++--- 3 files changed, 74 insertions(+), 8 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 7e66c4d2c..38ac73f85 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -240,6 +240,7 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@disabled-flags` | resolves a flag disabled in the management system to the code default | | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | +| `@string-typing` | reports `TYPE_MISMATCH` for a non-string flag requested as a string, rather than its string representation | | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](./sections/02-providers.md#requirement-252) permits rather than requires | | `@targeting` | resolves a flag differently for a matching evaluation context | @@ -556,6 +557,42 @@ are what stop it. A provider whose SDK has a single numeric type, such as JavaSc distinguish the cases at all, so it leaves the tag undeclared and the scenarios are reported as skipped with that reason. +### `@string-typing`: the same gap, one type further out + +`@string-typing` is the same question reached from the other direction, and it needs stating +separately because it breaks the reasoning that had kept four scenarios mandatory. + +Every value has a string representation. A backend that stores flag values as strings — Flipt does, +and so do some Flagsmith configurations — therefore satisfies the string accessor for **every** flag +and has no mismatch to report. Its flags are strings; +[Requirement 2.2.3](./sections/02-providers.md#requirement-223) asks it to populate `value` with the +resolved flag value, and it did. + +Nothing in the specification contradicts that, because the specification never says what the type of +a flag value **is**. `TYPE_MISMATCH` appears exactly once, as a row in the +[error code table](./types.md), and no requirement obliges anyone to raise it. The only normative +statement about value type is [Requirement 1.3.4](./sections/01-flag-evaluation.md#requirement-134) — +a `SHOULD`, and on the **client** rather than the provider. The provider requirements do not mention +type at all. + +So this tag sits upstream of [open-feature/spec#430](https://github.com/open-feature/spec/issues/430): +that issue asks which accessor a number may satisfy, this one asks what a flag's type is when the +backend has none. Until the specification answers it the rule is borrowed, exactly as the numeric one +is, and **a provider that withholds the tag is not violating the specification**. + +Four scenarios moved out of the mandatory matrix to make that honest: `boolean-flag`, `integer-flag` +and `float-flag` requested as strings, and `object-flag` requested as a string — the last tagged +`@object` too, since a provider with no structured values cannot be asked at all. The matrix had +justified keeping them on the grounds that *"is a string a boolean?"* has no defensible wrong answer. +That holds for parsing a string into another type, which a provider chooses to do; it does not hold +for rendering another type as a string, which an untyped backend does whether anyone chose it or not. +The asymmetry was real and pointed the wrong way. The case came from a Flipt provider written against +this suite, where those rows failed for a provider behaving reasonably. + +Withholding rather than a deviation is the right instrument, for the reason given under +[Rules for declaring](#rules-for-declaring): a deviation records a required behaviour the provider +lacks, and this behaviour is not required. + ### Capabilities a language cannot express **Where a capability cannot hold in a language, this appendix is where that is recorded** — not a diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 4c06f2e8d..2ce9b2463 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -9,7 +9,7 @@ These validate a **provider** against a real backend. For assets that validate a | Path | What it is | | --- | --- | | [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value and reason; the variant where the backend names one; falsy values; integer precision | -| [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix, numeric coercion and the unknown-flag case | +| [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix, numeric coercion, string typing and the unknown-flag case | | [`gherkin/events.feature`](./gherkin/events.feature) | configuration change, and the stale/ready transition across an outage | | [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one; shutdown | | [`gherkin/metadata.feature`](./gherkin/metadata.feature) | the provider identifies itself by name | diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature index 6fb191fcc..6d8552c65 100644 --- a/specification/assets/provider-tck/gherkin/errors.feature +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -13,9 +13,11 @@ Feature: Provider error handling Given a stable provider Scenario Outline: Requesting the wrong type returns the code default - # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered - # by the @numeric-coercion scenarios below, because "is 0.5 an integer?" has a defensible - # wrong answer whereas "is a string a boolean?" does not. + # The requests no representation can satisfy honestly. Two questions are held out of this + # matrix because they have defensible answers rather than obvious ones: whether a number fits + # a narrower accessor (@numeric-coercion) and whether any flag may be returned through the + # string accessor (@string-typing). What is left is the set no backend can satisfy -- "hello" + # is not a boolean, and true is not a number, however the backend stores them. Given a -flag with key "" and a default value "" When the flag was evaluated with details Then the resolved details value should be "" @@ -31,16 +33,13 @@ Feature: Provider error handling Examples: a boolean flag requested as something else | key | requested | default | - | boolean-flag | String | fallback | | boolean-flag | Integer | 1 | | boolean-flag | Float | 0.1 | Examples: a numeric flag requested as a non-numeric type | key | requested | default | | integer-flag | Boolean | false | - | integer-flag | String | fallback | | float-flag | Boolean | false | - | float-flag | String | fallback | @object Scenario Outline: Requesting a structured flag as a scalar returns the code default @@ -53,7 +52,6 @@ Feature: Provider error handling Examples: | requested | default | | Boolean | false | - | String | fallback | | Integer | 1 | | Float | 0.1 | @@ -92,6 +90,37 @@ Feature: Provider error handling And the error-code should be "" And no exception should have been thrown + @string-typing + Scenario Outline: A non-string flag is not returned as its string representation + # Every value has a string representation, so a backend that stores flag values as strings + # satisfies the string accessor for every flag and has no mismatch to report. Its flags are + # strings, and Requirement 2.2.3 asks it for the resolved flag value, which it returned. + # + # Whether that is wrong is not something this suite can assert -- see the appendix. A provider + # over an untyped backend withholds this tag and is not thereby non-conformant. + Given a String-flag with key "" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: + | key | + | boolean-flag | + | integer-flag | + | float-flag | + + @object @string-typing + Scenario: A structured flag is not returned as its JSON text + # The same property one type further out: a structure serialises to a string as readily as a + # scalar does. Tagged with both, because a provider with no structured values cannot be asked + # the question at all. + Given a String-flag with key "object-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + Scenario: An unknown flag key returns the code default # 'missing-flag' is deliberately absent from the canonical flag set. Given a String-flag with key "missing-flag" and a default value "fallback" From bd73a96eb9967e3476db817a4e802dcf635248b0 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Wed, 16 Sep 2026 08:30:14 +0200 Subject: [PATCH 62/63] feat: split string typing by what the backend can type, and close two integrity holes Three changes, all of them things running the suite found rather than things reading it suggested. `@string-typing` covered four cases and measurement showed one tag over all four hides defects inside a permitted absence. Three providers over one Flagsmith backend: Go and Java report TYPE_MISMATCH for a boolean and an integer requested as strings, JavaScript returns "true" and "10", and all three stringify the float and the structure. The bottom two are the backend, which records no native float or structure type, so the absence is permitted. The top two are not, and under one tag the provider that fails them withholds and its bug is reported as a permitted absence. `@fully-typed-values` now gates the float and structured cases; `@string-typing` keeps the two a partially typed store can still answer. The run-integrity section claimed two invisible failures and had only the vocabulary-to-assets direction of the tag check. The reverse is the one that is easy to leave out and all four reference implementations left it out: an unknown tag gates nothing, so its scenarios stay mandatory for every adopter and a suite that has not learned a capability silently keeps demanding the old behaviour. Two rules about the checks rather than about providers. A check that cannot run must fail rather than skip -- the revision check can only compare against a readable pin, and an unpacked distribution or a linked worktree is exactly where assets go stale. And the revision check must be in force where the scenarios execute: one adoption ran a full suite against the previous revision's scenarios and produced plausible numbers, because the check lived in the implementation's own tests and not in the run that published results. Also links spec#433, which tracks the underlying question of what fixes a flag value's type when the backend has none. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 61 ++++++++++++++++++- .../provider-tck/gherkin/errors.feature | 27 +++++--- 2 files changed, 78 insertions(+), 10 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 38ac73f85..8ee0845a6 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -240,7 +240,8 @@ declares which capabilities it supports. Scenarios whose tag is not declared are | `@disabled-flags` | resolves a flag disabled in the management system to the code default | | `@unavailable` | reports an error state instead of hanging against a dead backend | | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | -| `@string-typing` | reports `TYPE_MISMATCH` for a non-string flag requested as a string, rather than its string representation | +| `@string-typing` | reports `TYPE_MISMATCH` for a boolean or integer flag requested as a string, rather than its string representation | +| `@fully-typed-values` | records a native type for float and structured values too, so the same question can be asked of them | | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](./sections/02-providers.md#requirement-252) permits rather than requires | | `@targeting` | resolves a flag differently for a matching evaluation context | @@ -575,7 +576,7 @@ statement about value type is [Requirement 1.3.4](./sections/01-flag-evaluation. a `SHOULD`, and on the **client** rather than the provider. The provider requirements do not mention type at all. -So this tag sits upstream of [open-feature/spec#430](https://github.com/open-feature/spec/issues/430): +So this tag sits upstream of [open-feature/spec#430](https://github.com/open-feature/spec/issues/430), and is tracked as [open-feature/spec#433](https://github.com/open-feature/spec/issues/433): that issue asks which accessor a number may satisfy, this one asks what a flag's type is when the backend has none. Until the specification answers it the rule is borrowed, exactly as the numeric one is, and **a provider that withholds the tag is not violating the specification**. @@ -593,6 +594,31 @@ Withholding rather than a deviation is the right instrument, for the reason give [Rules for declaring](#rules-for-declaring): a deviation records a required behaviour the provider lacks, and this behaviour is not required. +#### Why this is two capabilities + +The first draft of this made it one tag over all four cases, and measurement showed that a single +tag **hides defects inside a permitted absence**. Three providers were run against one Flagsmith +backend: + +| requested as a string | Go | Java | JavaScript | +| --- | --- | --- | --- | +| `boolean-flag` | `TYPE_MISMATCH` | `TYPE_MISMATCH` | `"true"` | +| `integer-flag` | `TYPE_MISMATCH` | `TYPE_MISMATCH` | `"10"` | +| `float-flag` | stringified | stringified | stringified | +| `object-flag` | stringified | stringified | stringified | + +The bottom two rows are the backend: Flagsmith records no native float or structure type, so no +provider over it can report a mismatch, and the absence is permitted. The top two are not — that +store does record booleans and integers, two providers answer them, and the third fails them +because of its own code rather than the backend's shape. Under one tag that provider withholds and +its defect is reported as a permitted absence, which is the worst of the available outcomes: the +suite goes quiet on a real bug. Two tags separate the question the backend cannot answer from the +question a provider got wrong. + +The general rule this is an instance of: **a capability coarser than the variation providers +actually show will hide defects inside permitted absences.** When one tag would gate scenarios +that fail for different reasons, it is the wrong unit of declaration. + ### Capabilities a language cannot express **Where a capability cannot hold in a language, this appendix is where that is recorded** — not a @@ -869,7 +895,8 @@ the provider under test nothing. ### Run-integrity checks -Two failures are invisible from the results alone. +Three failures are invisible from the results alone, and two rules below are about the checks +themselves rather than about a provider. > A TCK implementation **MUST** fail the run when a reserved capability tag is carried by a collected > scenario. @@ -882,6 +909,34 @@ the vocabulary is stale. Either the assets are not the ones the implementation thinks it shipped, or a capability has outlived its scenarios — and in both cases a provider can declare it and be told nothing. +> A TCK implementation **MUST** fail the run when a collected scenario carries a capability tag the +> implementation's vocabulary does not know. + +This is the previous check's own direction reversed, and it is the one that is easy to leave out. An +unknown tag gates nothing, so its scenarios stay **mandatory for every adopter** — a suite that has +not learned a new capability does not report a new capability, it silently keeps demanding the old +behaviour. All four reference implementations ignored an unknown tag rather than failing, and the +symptom is a provider that legitimately withholds the capability showing unexplained failures while +every other provider stays green. Nothing in the results says why. + +> A TCK implementation **MUST** fail the run when a run-integrity check cannot be performed, rather +> than skipping it. + +A check that reports nothing when it cannot do its job is absent exactly where it is needed. The +revision check is the case in point: it can only compare the assets on disk against the pin when the +pin is readable, and in an unpacked distribution or a linked worktree it is not — which is also +where stale assets are most likely. One adoption ran a full suite against the previous revision's +scenarios and produced entirely plausible numbers, because the check that would have caught it was +skipping and the suite that runs it was not the suite that ran. + +That last point generalises past the check itself: + +> The revision check **MUST** be in force where the scenarios execute, not only in the TCK +> implementation's own tests. + +An adopter runs the canonical scenarios from its own build. A guarantee that holds only in the +implementation's test suite does not cover the run whose results are being published. + ## Extending the suite A provider often has behaviour this specification does not describe — flagd's fractional targeting, diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature index 6d8552c65..6d184d86b 100644 --- a/specification/assets/provider-tck/gherkin/errors.feature +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -95,9 +95,13 @@ Feature: Provider error handling # Every value has a string representation, so a backend that stores flag values as strings # satisfies the string accessor for every flag and has no mismatch to report. Its flags are # strings, and Requirement 2.2.3 asks it for the resolved flag value, which it returned. + # Whether that is wrong is not something this suite can assert -- see the appendix. # - # Whether that is wrong is not something this suite can assert -- see the appendix. A provider - # over an untyped backend withholds this tag and is not thereby non-conformant. + # These two rows are the ones a partially typed backend can still answer: a boolean and an + # integer are types such a store records natively. The float and structured cases are held + # separately behind @fully-typed-values, because a backend can lack a type for those while + # having one for these -- and one tag covering both would report a provider that fails these + # as merely untyped. Given a String-flag with key "" and a default value "fallback" When the flag was evaluated with details Then the resolved details value should be "fallback" @@ -108,13 +112,22 @@ Feature: Provider error handling | key | | boolean-flag | | integer-flag | - | float-flag | - @object @string-typing + @string-typing @fully-typed-values + Scenario: A float flag is not returned as its string representation + # Held apart from the two rows above because a store can record booleans and integers + # natively and still keep floats as text, which is what @fully-typed-values asks about. + Given a String-flag with key "float-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + @object @string-typing @fully-typed-values Scenario: A structured flag is not returned as its JSON text - # The same property one type further out: a structure serialises to a string as readily as a - # scalar does. Tagged with both, because a provider with no structured values cannot be asked - # the question at all. + # The same property one type further out. Three tags: a provider with no structured values + # cannot be asked at all (@object), and a store that keeps structures as text has nothing + # to report a mismatch about (@fully-typed-values). Given a String-flag with key "object-flag" and a default value "fallback" When the flag was evaluated with details Then the resolved details value should be "fallback" From ff68adb4c7617ad2d980988241e92603bc247926 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Wed, 16 Sep 2026 12:44:48 +0200 Subject: [PATCH 63/63] fix: a check with nothing to check is not a check that cannot run The fail-rather-than-skip rule named an unpacked distribution and a linked worktree in one breath. They are alike only in that git says nothing. A worktree has a pin and the implementation was not reading it; a distribution has no pin, no repository and no working tree that could have drifted, because its assets were produced by a sync that ran this check at build time. As written the rule required failing whoever unpacked it, for a defect they cannot hold or fix. Argued back from an implementation rather than reasoned out here, and the same reading produced the better fix for the worktree case: a worktree records its git directory as an absolute path while a submodule's is relative, so the pin is readable from the submodule side and the condition disappears rather than being reported. Signed-off-by: Simon Schrottner --- .../appendix-f-provider-conformance.md | 25 +++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 8ee0845a6..25c9edd41 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -923,11 +923,26 @@ every other provider stays green. Nothing in the results says why. > than skipping it. A check that reports nothing when it cannot do its job is absent exactly where it is needed. The -revision check is the case in point: it can only compare the assets on disk against the pin when the -pin is readable, and in an unpacked distribution or a linked worktree it is not — which is also -where stale assets are most likely. One adoption ran a full suite against the previous revision's -scenarios and produced entirely plausible numbers, because the check that would have caught it was -skipping and the suite that runs it was not the suite that ran. +revision check is the case in point: it compares the assets on disk against the pin, and where the +pin could not be read it skipped — which was also where stale assets were most likely. One adoption +ran a full suite against the previous revision's scenarios and produced entirely plausible numbers, +because the check that would have caught it was skipping and the suite that runs it was not the +suite that ran. + +**A check with nothing to check is not the same as a check that cannot be performed**, and this rule +does not reach the first. An unpacked distribution has no pin, no repository, and no working tree +that could have drifted from one: its assets are distribution content, produced by a sync that ran +this check when the distribution was built. Failing there would accuse whoever unpacked it of a +defect they cannot hold or fix. A linked worktree is the opposite case and looks similar only +because git is silent in both — there the pin exists and the implementation was not reading it +correctly, which is a check that must be made to work rather than allowed to skip. An implementation +has to tell the two apart before it applies this rule. + +That distinction was argued back from an implementation rather than reasoned out here: an earlier +revision of this section named the two cases in one breath, which would have required failing a +downstream packager. It also produced the better fix for the second case — a worktree records its +git directory as an absolute path, while a submodule's is relative, so the pin is readable from the +submodule side and the condition disappears instead of being reported. That last point generalises past the check itself: