test(flagsmith): run the provider conformance suite against the testbed

[aepfli]

Runs the OpenFeature Provider Conformance Suite against the Flagsmith Java provider.

Stacked on feat/provider-tck. Companion to go-sdk-contrib#959 and js-sdk-contrib#1623 — same container, same scenarios.

Result: 35 pass, 11 fail, 19 skipped of 65

	Go	Python	Java	JS
passed	45	29	35	31
failed	2	11	11	13
skipped	18	25	19	21
of	65	65	65	65

@standard-reasons is what changed this run. It was 24 pass / 12 fail before that capability existed; eight of those twelve failures were this provider not populating the resolution reason, and they are now skips carrying a recorded reason rather than bare failures.

@standard-reasons — declared, and failing

This provider never populates the resolution reason. Every evaluation returns null, so not one of the reasons the capability claims is reported. The resolved values are correct throughout; only the reason is missing.

An earlier pass withheld this capability, which turned eight failures into skips — the exact move KnownDeviation exists to discourage, and the base has since corrected the worked example that taught it. The provider does build a resolution and simply leaves the reason out, so running those scenarios establishes something and the failures belong in the results with the deviation attached.

What makes it a defect rather than a permitted absence is that the Go Flagsmith provider reports STATIC, DISABLED and TARGETING_MATCH against the identical backend.

The 5 remaining failures

• Two are unreadable floats. Flagsmith has no float type — feature_state_value is natively boolean, integer or string — so every float is stored as a string. Go's provider parses it back ("Because We store floats as string"); this one type-checks and falls through to the caller default, so getFloatValue never works against Flagsmith. Python has the same gap; the @numeric-coercion deviation now records this as measured rather than predicted.
• Two are the shared type-system pair. float-flag and object-flag requested as a String succeed rather than reporting TYPE_MISMATCH, because on this backend both genuinely are strings. Not a provider bug in any of the four languages.
• One is object-flag not resolving as a structure.

@large-integers — refused by the TCK, and correctly

The base now refuses this capability in Java rather than leaving it to the adopter: Java's integer accessor is a 32-bit Integer, so 2^53−1 cannot be asked for at all. That is a property of the SDK, not of the provider, and it is the same reason Java withholds it for flagd.

Worth knowing alongside it: Value.asInteger() silently truncates a Long to its low 32 bits — 4294967301 becomes 5, 2^53−1 becomes −1 — with no error. That is a defect in the Java SDK rather than anything this PR touches, but it is why the refusal matters: declaring the capability would not have errored, it would have returned a plausible wrong number.

Other capabilities

Declared: @object, @targeting, @disabled-flags, @string-typing.

@string-typing declared, @fully-typed-values withheld — and this adoption is why the suite
split them. Flagsmith's feature_state_value is natively boolean, integer or string, so this
provider correctly reports TYPE_MISMATCH for a boolean and an integer asked for as strings, while
floats and structures are stored as text and cannot be asked the question at all. One tag over all
four made this suite withhold everything and give up two passes it had earned; splitting them
recovered exactly those two, 33 → 35 passing and 21 → 19 skipped, with no change to the provider.
This file's own note ended "revisit if the capability is ever split by flag type" — this is that
revisit.

@disabled-flags holds because Flagsmith models a feature state as enabled plus a value, so the canonical set's four disabled-* flags map straight onto it, and this provider returns the caller default with reason DISABLED. Go agrees; Python and JS both raise GENERAL and withhold it.

@variants is withheld: Flagsmith has no variant concept for a plain feature, and the evaluation response carries no variant key. Permitted rather than defective — 2.2.4 makes it a SHOULD — so no deviation.

The lifecycle and event capabilities are withheld because this provider has no observable initialisation for the suite to assert against.

A finding about the TCK, not the provider

tools/tck exports jackson-databind 2.22.1, while flagsmith-java-client 7.4.3 pins jackson-annotations and jackson-core at 2.15.2. Maven's nearest-wins assembles databind 2.22 against annotations 2.15, and the Flagsmith client's ObjectMapper dies with ClassNotFoundException: com.fasterxml.jackson.annotation.JsonSerializeAs.

Verified this is the TCK's doing: with the TCK dependency removed the provider's own 35 tests pass; with it added, 22 error. Re-tested after the TCK moved testcontainers to provided/optional — still present, so it wants its own fix upstream: shade Jackson, or stop exporting a hard databind version.

The pin here is the workaround. It is compile scope and has to be, because the provider's own main source imports com.fasterxml.jackson.databind. It cannot be aligned upward — jackson-annotations has no 2.22.1 release, it tracks its own version line — so that route needs the Jackson BOM.

The backend

aepfli/flagsmith-tck-testbed 0.1.0, pinned rather than :latest because four adoptions pull it and a mutable tag lets a push change four PRs' results with no diff to explain it.

Why this is a draft

• The testbed image lives in a personal namespace.
• The Jackson conflict wants a decision upstream rather than a per-provider pin.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{Comment @coderabbitai help to get the list of available commands.}