Skip to content

docs: specify experiment holdouts in local flag evaluation - #76

Merged
marandaneto merged 3 commits into
mainfrom
docs/local-eval-holdouts
Sep 24, 2026
Merged

marandaneto merged 3 commits into
mainfrom
docs/local-eval-holdouts

Conversation

@marandaneto

@marandaneto marandaneto commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Problem

SDKs evaluating flag definitions locally can ignore filters.holdout and assign excluded experiment participants to regular variants. Python fixed this in PostHog/posthog-python#978, but the shared SDK specification did not cover holdouts.

Changes

  • Specify holdout evaluation before release conditions and the holdout-<id> result.
  • Define the exact SHA-1 input, inclusive percentage comparison, clamping, and flag-level bucketing identity.
  • Require holdout metadata to survive definition loading and cache round-trips.
  • Add acceptance scenarios for precedence, membership, boundaries, group/device identities, dependencies, and cache refreshes.
  • Explicitly skip holdouts with missing/null required fields, matching Python, with four acceptance examples.
  • Keep the exact-zero inclusive rule in the requirement text without requiring an injected-zero acceptance scenario.
  • Apply the requirements to the canonical specs and keep one archived OpenSpec change, including the review clarifications.

SDK implementation fixes are follow-up work. This PR does not change SDK code.

Validation

  • openspec validate define-local-eval-holdouts --strict passed before archiving.
  • openspec validate --specs --strict: 63 passed.
  • Independently checked the hash vectors and verified that the incorrect dot-separated prefix reverses both membership results.
  • Verified that both consolidated archived deltas match the canonical specs. Consolidating the archives leaves the canonical specs and acceptance scenarios unchanged.
  • git diff --check passed.

The Gherkin files document acceptance contracts. SDK conformance tests were not run. Autoreview was skipped because this is a documentation-only change.

@marandaneto

Copy link
Copy Markdown
Member Author

Missing holdout support

Follow-up SDK implementation work from the source audit. All 17 requested repositories were checked against freshly fetched default-branch snapshots. Links below pin those audited commits, rather than moving main branches.

These evaluators proceed to release conditions and regular variant assignment without checking filters.holdout. They also do not force server fallback specifically because a holdout is present.

Repository Affected implementation Evidence
posthog-dotnet .NET Evaluator
posthog-android Java/Kotlin server SDK, not Android mobile Evaluator
posthog-go Go Evaluator
posthog-elixir Elixir Evaluator
posthog-js Node.js Evaluator
posthog-js Convex Evaluator
posthog-php PHP Evaluator
posthog-ruby Ruby Evaluator
posthog-rs Rust (both evaluation paths) Evaluator

8 repositories, 9 SDK implementations need follow-up fixes. Go has hash tests containing holdout-, but no holdout evaluation logic.

Already fixed

Python PR #978 is merged. The audited default branch includes holdout handling, and its changelog lists the fix under 7.59.1.

Not affected by this local-evaluation bug

  • Android mobile, iOS, Unity, Godot, and the browser/React Native packages in posthog-js consume server-evaluated flag values.
  • Flutter and KMP delegate flag evaluation to their underlying client SDKs rather than owning a local definition evaluator.
  • posthog-pi, posthog-openclaw, and posthog-opencode use posthog-node but do not configure local evaluation.

Follow-up scope

Implement the contract in each affected evaluator and preserve holdout metadata through typed definitions and existing caches. Cover holdout precedence, exact hash parity, fractional/inclusive boundaries and clamping, group/device identity, and bulk/dependency evaluation. The posthog-js work must cover both Node.js and Convex.

This is a source audit, not a report of passing or failing runtime reproductions across these SDKs. No SDK implementation changes are included in this PR.

@marandaneto
marandaneto marked this pull request as ready for review September 23, 2026 13:49
@marandaneto
marandaneto requested a review from a team as a code owner September 23, 2026 13:49
@marandaneto
marandaneto requested a review from a team September 23, 2026 13:49
Comment thread openspec/specs/local-feature-flag-evaluator/spec.md Outdated
Comment thread acceptance/private/local-feature-flag-holdouts.feature Outdated
@marandaneto
marandaneto merged commit bec7797 into main Sep 24, 2026
10 checks passed
@marandaneto
marandaneto deleted the docs/local-eval-holdouts branch September 24, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants