feat(provider-tck): in-process control path and in-memory/multi-provider self-tests

[aepfli]

Part of open-feature/spec#417 (cross-language tracking) via #1829 (Java implementation issue). Language-agnostic artifacts: spec#423.
Stacked on #1830. Base is feat/provider-tck, so this diff shows only the new work. Review #1830 first.

What

Makes the TCK runnable without Docker, a compose stack, or HTTP, so providers with no external backend (in-memory, environment-variable, file-based) can adopt it — and uses that to give the TCK a self-test.

Three commits, deliberately separable, each building standalone:

1. refactor — extract BackendControl, split the base class. No behaviour change.
2. feat — the in-process control path and the in-memory self-test.
3. test — a second self-test against MultiProvider, which found a real gap.

1. The refactoring

Step definitions reached the Compose stack and the HTTP control API directly through TckRuntime. That made the suite unrunnable for any provider without a containerised backend, and put transport knowledge in the one layer that should have none.

BackendControl is now the single seam between step definitions and backend manipulation:

void prepareScenario();                  // reset to the canonical baseline
void changeFlag();                       // mutate changing-flag
default void disconnect();               // throws unless implemented
default void reconnect();                // throws unless implemented
default void disconnectFor(Duration);    // throws unless implemented
default String description();

All nine places where a step touched HTTP or container state now go through it. ControlApiClient became HttpBackendControl, one implementation of that seam. Nothing about the HTTP control API spec changes — it remains the normative contract for external backends.

The base class splits along the same line:

• ProviderTckTest (renamed from AbstractProviderTckTest) — capability declaration, timeouts, awaiting, step wiring
• ContainerizedProviderTckTest extends ProviderTckTest — Compose lifecycle, port discovery, HttpBackendControl construction, and the compose-specific config that used to sit on ProviderTckHarness

flagd needed exactly the superclass rename — one import, one extends, nothing else.

Where the abstraction leaked

Two places beyond the step layer had to change, both unavoidable and both the point of the split:

• ProviderTckHarness mixed core config (capabilities, eventTimeout, readyTimeout) with Compose config (composeFile, backendPorts, controlPort, defaultConfig, startupTimeout, settleTime) in one SPI
• TckRuntime hard-coded ComposeContainer, and BackendEndpoint wraps it directly

Nothing else moved. FlagSteps, EventSteps and ContextSteps were already clean.

2. The feature

InProcessBackendControl manipulates the SDK's InMemoryProvider directly. Flag operations are map updates; changeFlag() is updateFlag(), so the event the suite awaits is the provider's own PROVIDER_CONFIGURATION_CHANGED carrying changing-flag in flagsChanged — not something the TCK synthesised.

InMemoryProviderTckTest runs the full applicable suite:

29 scenarios (26 passed, 3 skipped)
295 steps (270 passed, 25 skipped)
0m 0.622s

No Docker, no network, under a second. It doubles as the reference adoption for a backend-less provider (three methods) and as a Docker-free CI canary.

Connection control is modelled by capability, not by no-op stubs

disconnect(), reconnect() and disconnectFor() are left at their throwing defaults, and the harness leaves STALE and UNAVAILABLE_INIT undeclared. The three skipped scenarios are exactly those, each reported with its reason:

Losing the backend makes the provider stale, regaining it makes it ready again
  Skipped: provider does not declare capability STALE (tag @stale).
  Declared capabilities: [EVENTS, CONFIGURATION_CHANGE, OBJECT, STRICT_NUMERIC_TYPING]

I verified this cannot go silently green. Temporarily declaring STALE makes the scenario fail, with a message naming the fix:

UnsupportedOperationException: in-process control of InMemoryProvider does not support
'disconnect'. This is a test-configuration bug rather than a provider defect: a scenario
needing connection control ran, so the harness declared Capability.STALE or
Capability.UNAVAILABLE_INIT for a backend that cannot simulate an outage.

InProcessBackendControlTest pins that permanently, along with two things the Gherkin cannot assert about itself: that changeFlag() actually changes the resolved value, and that it does not leak into the next scenario.

InMemoryProvider does declare STRICT_NUMERIC_TYPING — its isAssignableTo only widens Integer→Long, so float-flag (0.5) requested as an integer is a TYPE_MISMATCH rather than a silent 0. It is the reference behaviour that capability describes.

3. A second self-test: MultiProvider

MultiProviderTckTest runs the same suite against the SDK's MultiProvider wrapping exactly one InMemoryProvider. One child is the interesting configuration rather than a degenerate one: the correct answer is then precisely what InMemoryProviderTckTest already asserts, so any difference between the two suites is attributable to MultiProvider and nothing else. This is not a test of aggregation across backends — it is a test that delegation is transparent, which is where the contract is easiest to drop (a variant that does not survive the hop, a reason rewritten, an error code flattened, an event that never arrives).

It found something on the first run:

29 scenarios (25 passed, 4 skipped)

MultiProvider extends EventProvider but never subscribes to its children, so a child's PROVIDER_CONFIGURATION_CHANGED — along with PROVIDER_ERROR and PROVIDER_STALE — is swallowed and never reaches the client. Wrapping a provider in a multi-provider silently costs you those events, with nothing in the API to hint at it.

This is a known gap: open-feature/java-sdk#1882, gap 1 ("child provider event aggregation and status tracking", High), which states verbatim that it "does not listen to or forward events from child providers". That gap was originally found by hand-comparing implementations against the js-sdk reference; the TCK reproduced it from the outside without knowing it was there, which is a fair advertisement for the whole exercise.

CONFIGURATION_CHANGE is therefore left undeclared, so the scenario is reported as skipped-with-reason rather than passing on a provider that cannot satisfy it — the same treatment flagd's STRICT_NUMERIC_TYPING gets. Delete the omission once #1882 is fixed. Everything else survives delegation unchanged: values, variants, reasons, the full type-mismatch matrix, FLAG_NOT_FOUND, structured values, strict numeric typing and reaching READY.

CI

New provider-tck job running both in-process suites with no Docker and no e2e profile. It runs in parallel with, not as a gate on, the existing matrix, so a green run is not delayed. The containerised flagd suite is unchanged.

Docs

New README section on which base class to extend, and an explicit statement that in-process control is for backend-less providers only — an external backend driven through a custom in-JVM BackendControl bypasses the control API and proves nothing. Same statement in the BackendControl javadoc, where someone is more likely to hit it.

Scope decision

The design sketch listed setFlag / removeFlag on BackendControl. No Gherkin step needs them and the control API has no endpoints for them, so adding them would create methods HttpBackendControl cannot implement. Left out, and recorded in the README's known gaps as needing a control-API revision first.

Verification

• in-memory suite: 29 scenarios, 26 passed, 3 skipped-by-capability, 0 failures
• multi-provider suite: 29 scenarios, 25 passed, 4 skipped-by-capability, 0 failures
• InProcessBackendControlTest: 4 passed
• whole module: 62 tests, 0 failures, in ~1s
• mvn -Pcodequality,deploy verify green — checkstyle, PMD, SpotBugs, javadoc, spotless
• flagd test-compiles against the refactored TCK
• all three commits build standalone

Not verified: the containerised flagd suite has not been run — no Docker on the machine this was developed on. It compiles, and every control-API call is the same endpoint in the same order as before, but CI is the first real execution.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}