Skip to content

feat(experiments): support configurable baseline variants - #37

Open
lordspline wants to merge 1 commit into
masterfrom
capy/support-configurable-baseline
Open

feat(experiments): support configurable baseline variants#37
lordspline wants to merge 1 commit into
masterfrom
capy/support-configurable-baseline

Conversation

@lordspline

@lordspline lordspline commented Aug 20, 2026

Copy link
Copy Markdown

Problem

Experiment creators cannot choose which variant acts as the statistical baseline, so every comparison assumes a variant keyed control.

This blocks exact-case variant keys and experiments whose natural baseline has another name.

Closes PostHog#30796

Changes

  • Users can select any included variant as the baseline during creation or from analysis settings.
  • The API stores the exact variant key in stats_config without normalizing its case.
  • Existing experiments use exact lowercase control when present, then fall back to their first configured variant.
  • Modern and legacy results put the baseline first and compare every included variant against it.
  • Baseline changes invalidate query caches and recalculation fingerprints, including inferred legacy baselines after flag reordering.
  • Validation protects excluded variants, deleted variants, duplicates, cleanup plans, and copied experiments.
  • Generated frontend and MCP schemas expose the setting while preserving extension fields in existing stats configuration.

Create an experiment with Challenger selected

Screenshot

Results with Challenger as the baseline

Screenshot

Results after switching the baseline to Control

Screenshot

How did you test this code?

  • Created and launched an experiment with exact-case Control, TreatmentA, and Challenger variants in the local app.
  • Ingested invented exposure and conversion events, then verified the Challenger comparison values in the results UI.
  • Switched the running experiment to Control and verified recalculated +40% and -40% deltas with baseline-first ordering.
  • Added backend coverage for validation, fallback rules, exclusions, cleanup, duplication, query runners, and recalculation fingerprints.
  • Added frontend coverage for creation, linked flags, legacy calculations, activity descriptions, and exact-case fallback behavior.
  • Added MCP coverage proving generated request schemas retain unknown top-level and nested stats configuration fields.
  • Ran focused Python and Jest suites, frontend and MCP type checks, repo-wide mypy, OpenAPI regeneration, migration checks, and CI preflight.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Automatic notifications

  • Publish to changelog?

Docs update

No matching product guide lives in this repository. The UI copy and generated API and MCP schemas document the setting and fallback.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Capy implemented and verified the change with repository tooling and browser-driven local UI testing.
  • Invoked /improving-drf-endpoints, /adopting-generated-api-types, /writing-ui-components, /writing-tests, /writing-user-facing-copy, /writing-code-comments, /implementing-mcp-tools, and /writing-pr-descriptions.
  • Chose stats_config persistence to preserve compatibility and materialized inferred baselines before variant order can change.
  • Kept legacy fallback exact and case-sensitive instead of normalizing variant keys.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

🦔 Hogbox preview · ❌ build failed

The preview didn't come up for commit 1014f9e. See the build log for the failing step. It'll retry on the next push.

Previews are optional and never block merging. A failure here is often a hogland or tailnet hiccup rather than anything in your PR, so the check stays green and this comment is the status.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Choose control group / baseline in experiments

1 participant