Skip to content

CLUE-578: select the chat tutor's AI backend per unit and per session - #2986

Merged
kswenson merged 10 commits into
masterfrom
CLUE-578-tutor-provider-selection
Aug 28, 2026
Merged

CLUE-578: select the chat tutor's AI backend per unit and per session#2986
kswenson merged 10 commits into
masterfrom
CLUE-578-tutor-provider-selection

Conversation

@kswenson

@kswenson kswenson commented Aug 28, 2026

Copy link
Copy Markdown
Member

A unit can now say which AI answers its tutor

#2984 gave the tutor a provider seam on the server, but nothing could reach it — every conversation went to OpenAI because there was no way to say otherwise. This adds the selection path: a chatTutorProvider unit config property, a chatProvider URL param that overrides it for a single session, and the Firestore rules that admit the choice.

Nothing selects a second provider yet, because none is implemented. That is the point of splitting this out: it is the half of the ForeverLearning spike (CLUE-578) that does not depend on ForeverLearning, and it is useful whatever comes of that spike — a second backend of any kind needs exactly this.

How the choice travels

The trigger cannot read unit config, so the client resolves the provider and stamps it on the message doc — the same channel the authored prompt overrides already ride. Precedence is query param > unit config > default, and an unrecognized value at either level is dropped rather than honored, so a typo cannot select a backend nothing implements.

Nothing reads the stamp on the server yet, and that is deliberate. The trigger builds an OpenAI provider unconditionally and never copies provider onto the parent doc, so selecting a non-default provider today changes the conversation id and the message field and nothing else — the turn is still answered by OpenAI. Trigger-side routing arrives with the second backend that makes it meaningful; adding it now would be machinery with nothing to route to. When it lands it should persist the provider from the first message and ignore the field thereafter, so a mid-conversation flip cannot split one conversation's state across two backends.

The default is a strict no-op

This is what the "existing conversations are untouched" guarantee rests on, so it is enforced in one place rather than remembered in several. nonDefaultTutorProvider turns the default provider into undefined, and both the conversation id and the message stamp go through it. An OpenAI conversation therefore resolves to exactly the id it resolved to before and writes exactly the fields it wrote before — not equivalent ones, the same ones.

For a non-default provider the two suffixes stack: _v<provider>_p<hash>. Each names something the conversation was built with and cannot be re-made with, so each has to fork on its own. The generic prompt installs once per conversation and its items are immutable, and the client attaches the authored overrides to install-eligible sends whatever the provider is — so a prompt edit has to land on a new conversation regardless of which backend ends up answering it.

One list, four places

The provider vocabulary lives in shared/chat-tutor-providers.ts because three places reference it and must not disagree: the client that resolves and stamps, the unit config schema, and eventually the trigger that routes. The fourth cannot import it — the enum pin in the rules — so both rules blocks carry a comment pointing back at the shared list, and tutor-provider-rules.test.ts reads firestore.rules from disk and fails if either pin drifts from kTutorProviders.

Both blocks, authed and demo, whitelist provider and pin it to the enum. Two things worth noting there:

  • The pin constrains the value, not just the shape. Once the trigger routes on this field, an arbitrary string would decide where a paid turn goes — the pin guards that, rather than anything the server reads today.
  • Whitelisting under authed alone would have broken the demo root, which is exactly where the chatProvider param gets exercised first — and it would have failed with a permission-denied that reads nothing like a config error. The branch's own history records this same authed-only mistake being made once before.

Deliberately not here: the authoring UI

The integration analysis calls for a provider select in chat-tutor-settings.tsx. I left it out. Offering curriculum authors a ForeverLearning option before any backend implements it would let them silently turn the tutor off for a unit — the config key is still authorable by hand, and the URL param covers the testing path. The doc entry says so explicitly, following the precedent chatTutorHighlights set on ai-highlights. Easy to add once a second provider exists.

The first commit is documentation rescued from ai-highlights

chatTutorEnabled, chatTutorIntro, and chatTutorPrompts have never been documented in docs/unit-configuration.md, even though the properties have been on master since the tutor landed. #2972 documented them — into the ai-highlights spike branch, not master, so the text lives only there.

Those three entries are copied verbatim, so they stay identical to what was reviewed in #2972 and do not conflict when that branch lands. chatTutorHighlights is documented in the same block there and is deliberately left behind: that property does not exist on master and belongs with its feature.

Testing

Full Jest suite: 4001 passed across 356 suites (one suite and five tests skipped). check:types clean. lint:build reports 0 errors (the two warnings in files touched here are pre-existing — a long groupSettings line and an unused DocumentSpecModel import).

Rules tests run against the Firestore emulator, 31 passing:

cd firebase-test && npx firebase emulators:exec --only firestore "npx jest src/chat-tutor-rules.test.ts"

Note that firebase-test needs its own npm install — the root npm ci postinstall covers only shared, so the rules tests will not compile until you run it there. firebase-tools 15 also refuses any JDK below 21, so if the emulator will not start, export JAVA_HOME=$(/usr/libexec/java_home -v 21) first.

Assertions here are easy to write so they cannot fail, so each new one was checked against deliberately broken code rather than trusted for being green:

  • making the message stamp unconditional fails the omit test, and nothing else
  • adding chatProvider to booleanParams fails the value test with Received: false
  • deleting the enum clause from the authed rules block fails the reject test, while the whitelist alone keeps the rest green — which is what proves the pin is doing work the hasOnly was not
  • re-gating the prompt send on the default provider fails the non-default prompt test and leaves the default one green. That pair is the other half of the stacking contract: the prompts key forks the conversation under a non-default provider precisely because the overrides are still sent there
  • dropping a pin, adding one, drifting either block's contents, and rewording both pins out of the regex's reach each fail the consistency test — the last case is why it asserts over the whole match set instead of looping, since a loop reports success over an empty one
  • dropping openai from both rules blocks and the shared list leaves the consistency test green, because it only compares the blocks to each other. The openai accept cases in the emulator suite are what fail there, on both the authed and demo blocks

chatTutorEnabled, chatTutorIntro, and chatTutorPrompts have never been
documented in docs/unit-configuration.md. They were documented in #2972,
but that PR merged to the ai-highlights spike branch rather than to
master, so the text exists only there while the properties themselves
have been on master since the tutor landed.

These three entries are copied verbatim from ai-highlights so they stay
identical to what was reviewed there and do not conflict when that branch
lands. chatTutorHighlights is deliberately left behind: it is documented
in the same block on ai-highlights, but the property does not exist on
master and belongs with its feature.
The tutor has had a provider seam on the server since #2984, but nothing
could reach it: every conversation went to OpenAI because no one could say
otherwise. This adds the selection path, from authored config and URL
param down to the rules that admit the choice.

The trigger cannot read unit config, so the client resolves the provider
and stamps it on the message doc, riding the same channel the authored
prompt overrides already use. Precedence is query param > unit config >
default, and an unrecognized value at either level is dropped rather than
honored: a typo should not select a backend nothing implements.

The default is a strict no-op, which is what keeps every existing
conversation intact. nonDefaultTutorProvider is the single choke point
that turns the default into undefined, and both the conversation id and
the message stamp go through it, so an OpenAI conversation writes exactly
the doc it wrote before and resolves to exactly the id it resolved to.

A non-default provider drops promptsKey from the conversation id rather
than stacking with it. Prompt overrides are an OpenAI-path feature, so a
prompt edit should not fork a conversation that cannot use them.

The provider vocabulary lives in shared/ because three places reference
the same list and must not disagree: the client, the unit config schema,
and eventually the trigger's routing. A fourth cannot import it - the
enum pin in the rules - so both rules blocks carry a comment pointing at
the shared list.

Both rules blocks, authed and demo, whitelist the field and pin it to the
enum. Pinning the value matters beyond shape: the trigger routes on it, so
an arbitrary string would decide where a paid turn goes. Whitelisting it
under authed alone would have broken the demo root, which is where the
chatProvider param gets exercised first.

Not included: the authoring UI select. Offering authors a provider before
a backend implements it would let them silently turn the tutor off for a
unit. The config key is still authorable by hand and the URL param covers
testing.

Tested with the emulator for the rules and mutation checks for the three
assertions that could have passed by construction: making the stamp
unconditional, adding chatProvider to booleanParams, and deleting the
rules enum clause each break exactly one test.
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 86.37%. Comparing base (4941cfd) to head (6484b63).

Files with missing lines Patch % Lines
src/components/chat-tutor/chat-sidebar.tsx 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2986      +/-   ##
==========================================
+ Coverage   86.31%   86.37%   +0.05%     
==========================================
  Files         994      996       +2     
  Lines       56836    56856      +20     
  Branches    15056    15060       +4     
==========================================
+ Hits        49059    49109      +50     
+ Misses       7757     7727      -30     
  Partials       20       20              
Flag Coverage Δ
cypress ?
cypress-regression 71.16% <10.00%> (-0.02%) ⬇️
cypress-smoke 41.16% <10.00%> (-0.01%) ⬇️
jest 57.75% <95.65%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cypress

cypress Bot commented Aug 28, 2026

Copy link
Copy Markdown

collaborative-learning    Run #20165

Run Properties:  status check passed Passed #20165  •  git commit 6484b6368d: refactor: make sessionTutorProvider the module's only export
Project collaborative-learning
Branch Review CLUE-578-tutor-provider-selection
Run status status check passed Passed #20165
Run duration 03m 42s
Commit git commit 6484b6368d: refactor: make sessionTutorProvider the module's only export
Committer Kirk Swenson
View all properties for this run ↗︎

Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 4
View all changes introduced in this branch ↗︎

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds client-side chat tutor provider selection through unit configuration and URL overrides.

Changes:

  • Defines provider resolution and conversation-key behavior.
  • Propagates provider metadata with Firestore rule coverage.
  • Documents and tests the new configuration path.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/utilities/url-params.ts Adds chatProvider.
src/utilities/url-params.test.ts Tests valued parameter parsing.
src/models/stores/unit-configuration.ts Adds provider configuration.
src/models/stores/configuration-manager.ts Exposes provider configuration.
src/models/stores/app-config-model.ts Adds the provider view.
src/models/stores/app-config-model.test.ts Tests provider configuration cascading.
src/components/chat-tutor/tutor-provider.ts Resolves and normalizes providers.
src/components/chat-tutor/tutor-provider.test.ts Tests provider precedence.
src/components/chat-tutor/firestore-transport.ts Stamps messages with providers.
src/components/chat-tutor/firestore-transport.test.ts Tests provider stamping.
src/components/chat-tutor/conversation-key.ts Separates non-default conversations.
src/components/chat-tutor/conversation-key.test.ts Tests provider-specific keys.
src/components/chat-tutor/chat-sidebar.tsx Connects configuration to transport.
shared/chat-tutor-providers.ts Defines the provider vocabulary.
firestore.rules Validates provider fields.
firebase-test/src/chat-tutor-rules.test.ts Tests provider security rules.
docs/unit-configuration.md Documents tutor configuration.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/components/chat-tutor/firestore-transport.ts Outdated
Copilot review caught that the trigger builds an OpenAI provider
unconditionally and pickOwnerFields never copies `provider` onto the
parent, so a stamped provider selects nothing today. That is intended for
this PR — the routing belongs with the second backend that makes it
meaningful — but three places described the server behavior in the present
tense as though it already existed.

The config doc entry was the worst of them: an author could set
chatTutorProvider to foreverlearning, get no error, and see no change.
It now leads with the fact that only openai is implemented.

The transport comment and the shared vocabulary header now say the same
thing, and the transport note asks whoever adds the routing to delete it,
since it goes stale the moment that lands.

No behavior change.

@dougmartin dougmartin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design is sound and the default-path no-op is genuinely well protected, but the two halves of the non-default provider path contradict each other today: the conversation id drops promptsKey on the grounds that prompts do not apply, while the transport still sends promptReplace/promptAppend to a conversation that OpenAI still answers. That needs resolving one way or the other before this lands.

Changes requested

  • src/components/chat-tutor/conversation-key.ts (lines 25-26) and src/components/chat-tutor/firestore-transport.ts (lines 192-195): a non-default provider drops promptsKey from the conversation id, but sendUserMessage still stamps promptReplace/promptAppend on install-eligible sends with no provider check, and the trigger answers every turn with OpenAI. So with chatTutorProvider: "foreverlearning" plus authored chatTutorPrompts, OpenAI installs the authored generic prompt on the first send, and a later prompt edit no longer changes the conversation id, so that conversation keeps the original prompt permanently with no way to reset it. That is exactly the failure the _p<hash> suffix was added to prevent (conversation-key.ts lines 6-11), silently reintroduced for the configuration this PR adds, and conversation-key.test.ts (lines 31-34) currently locks it in as intended. Fix either half: stack the suffixes (${base}_v${provider}_p${promptsKey} when both are present) and update that comment and test, or gate the prompt send on decision.attachLeft && !provider so the id's rationale is actually true, with a firestore-transport.test.ts case asserting promptReplace is absent under a non-default provider.

  • firestore.rules (lines 750 and 844) against shared/chat-tutor-providers.ts (line 10): ['openai', 'foreverlearning'] is hardcoded in both rules blocks and again as kTutorProviders, bound only by a prose comment saying to keep them in lockstep. Adding a third provider and missing either rules block gives a permission-denied on every message write under the new provider with the whole suite green, and it lands in demo/qa first, which is precisely the path the demo whitelist was added to protect. Assert the agreement in a test: src/models/stores/spikerbit-firmware-consistency.test.ts is the existing precedent for reading a committed artifact from disk in the main Jest suite. Read firestore.rules, match every data.provider in \[([^\]]*)\], assert exactly two matches were found (so an empty match set cannot pass), and assert each parsed list equals [...kTutorProviders].

  • firestore.rules (lines 731-733), firebase-test/src/chat-tutor-rules.test.ts (lines 81-82), and src/models/stores/unit-configuration.ts (line 115): commit 100d326 removed the present-tense "the trigger routes on it" claim from three files in response to the Copilot thread, but three sites still assert it. The rules one matters most because it is the stated justification for the enum pin and a rules reader cannot check it against the trigger source. Restate all three in the forward-looking form the other files now use, for example "once the trigger routes on this field an arbitrary string would decide where a paid turn goes; the trigger builds an OpenAI backend unconditionally today, so the field is inert on the server."

Non-blocking

  • src/components/chat-tutor/conversation-key.ts (lines 22 and 25): provider?: string is untyped where every other point on the path uses TutorProviderId, and it is interpolated straight into a document id without escapeKey, which problemPath on the adjacent line does need. A future caller passing a raw config or param value containing / would inject a path separator. Import TutorProviderId from ../../../shared/chat-tutor-providers (no cycle, the file already imports from shared/shared), or wrap as escapeKey(provider).

  • src/components/chat-tutor/chat-sidebar.tsx (lines 57-69): the pieces are each unit-tested but the line that joins them is not, and there is no chat-sidebar.test.tsx. Swapping the two arguments on line 58 inverts the documented query-param-over-config precedence, and deleting provider from line 62 or line 69 removes the wiring outright; all three leave the suite green. Consider exporting a sessionTutorProvider(paramValue, configValue) helper from tutor-provider.ts that composes the two functions, call it from the sidebar, and test that sessionTutorProvider("foreverlearning", "openai") is "foreverlearning" and sessionTutorProvider("openai", "foreverlearning") is undefined. That makes both argument order and the default carve-out fail-detectable in one place.

  • src/utilities/url-params.ts (lines 160-164) and src/components/chat-tutor/chat-sidebar.tsx (line 58): four comments call chatProvider a dev/qa tool (firestore.rules lines 830-831, firebase-test/src/chat-tutor-rules.test.ts lines 280-281, src/models/stores/unit-configuration.ts line 116, src/components/chat-tutor/tutor-provider.test.ts line 11) but nothing gates it, and the authed rules block admits the resulting write. A student in a real class can append ?chatProvider=foreverlearning and land on a fresh empty conversation, which reads as their transcript vanishing. Harmless today, but once routing lands the same URL picks which paid backend answers. Either resolve the param to undefined when appMode === "authed", or drop the dev/qa framing from those four comments. Note chatDebug is equally ungated, so ungated may simply be the house style here; if so, say that instead of implying a gate.

  • specs/CLUE-566-ai-chat-tutor.md (lines 121-124 and 145-147): line 147 documents the message doc shape without provider?, and lines 121-124 describe the _p<hash> suffix with no mention of the _v<provider> form that now replaces it. This spec is maintained with appended follow-on sections, so a short provider-selection section plus provider? in the field list would keep it current.

  • docs/unit-configuration.md (line 128): "an unrecognized value in either place falls back to the unit config, then to the default" describes something impossible for the config level, since an unrecognized config value cannot fall back to itself. Suggest "an unrecognized URL param value falls back to the unit config, and an unrecognized unit config value falls back to the default." The chatTutorPrompts sentence on the same line also needs to match whichever resolution you take on the first blocking item.

  • src/authoring/types.ts (line 56): IUnitConfig mirrors UnitConfiguration and carries the other three chatTutor* properties but not chatTutorProvider. No data loss, since the authoring workspace mutates an immer draft of the parsed unit JSON and unknown keys survive a save, but the two interfaces now describe different shapes and the next person on the Chat Tutor authoring page gets no type-level signal that the property exists. Either add it or leave a one-line note that the provider is deliberately not authored in the UI.

  • firebase-test/src/chat-tutor-rules.test.ts (lines 79-88 and 283-289): the authed test rejects a non-string (provider: 42), the demo test does not, though the demo block carries the identical pin and is described as the path exercised first. Neither block asserts provider: "openai" is accepted, so that vocabulary entry is untested. Adding a provider: 42 case to the demo test would catch the demo pin being weakened to an is string guard. Subsumed if you drive the accept cases off kTutorProviders per the enum-drift item above.

  • src/components/chat-tutor/firestore-transport.ts (lines 176-186): eleven comment lines above a two-line if, most of it narrating what the server does not do yet, instructing an unnamed future engineer how to implement routing, and asking them to delete the comment. It is an untracked TODO with no owner. Consider reducing to the live fact plus a ticket reference and moving the "persist the provider from the first message" guidance to that ticket, where it will actually be read when the work starts.

kswenson and others added 4 commits August 28, 2026 12:01
conversationDocId dropped promptsKey whenever a non-default provider was
given, on the stated grounds that prompt overrides are OpenAI-only. But
sendUserMessage attaches promptReplace/promptAppend to install-eligible
sends with no provider check, and the trigger answers every turn with
OpenAI. So with chatTutorProvider "foreverlearning" plus authored prompts,
OpenAI installed the authored prompt on the first send and a later prompt
edit no longer changed the conversation id — that conversation kept its
original prompt permanently, with no way to reset it. That is the exact
failure the _p<hash> suffix was added to prevent, and conversation-key.test
asserted it as intended behavior.

The two suffixes now stack (_v<provider>_p<hash>). Both name something the
conversation was built with and cannot be re-made with, so each has to fork
on its own, whichever backend ends up answering. The alternative fix was to
gate the prompt send on the default provider instead, but that rests on the
same claim that justified the drop: that other backends accept no prompt
from us. That is an assertion about a backend nobody has integrated yet,
and it is false today, since OpenAI answers these turns too. Stacking costs
a spurious fork per prompt edit if it turns out to be true; gating costs a
permanently wrong prompt if it turns out to be false.

The config doc entry and the transport comment no longer make the claim.
Also trims the transport's provider NOTE to the live fact plus the
constraint routing has to honor, dropping the instructions addressed to a
future engineer and the request that they delete the note.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy
The provider enum is spelled out twice in firestore.rules — once in the
authed chatMessageCreate block, once in the demo one — and neither can
import kTutorProviders. Nothing but a prose comment held the three in
agreement. Adding a provider to the shared list and missing either rules
block is a silent break: every message write under the new provider fails
with permission-denied while the whole Jest suite stays green, and demo/qa
is where a new provider gets exercised first.

tutor-provider-rules.test.ts reads firestore.rules from disk, following the
spikerbit-firmware-consistency precedent, and asserts that both pins equal
kTutorProviders. It asserts the match count first, so a rewording that puts
the pins out of the regex's reach fails loudly instead of passing over an
empty match set. Both failure modes were checked by mutation: adding a
provider to only the demo block fails the equality assertion, and replacing
both pins with `is string` fails the count assertion.

The rules blocks now point at that test, so a reader who has to keep the
lists in lockstep knows what catches them.

Also fills two gaps in the emulator suite: the demo block rejected no
non-string provider though it carries the identical pin, and neither block
asserted that 'openai' is accepted, leaving that vocabulary entry untested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy
The sidebar called resolveTutorProvider and nonDefaultTutorProvider itself.
Each is unit-tested, but the line joining them was not, and there is no
chat-sidebar test: swapping the two arguments there would silently invert
the documented query-param-over-config precedence, with nothing to catch
it. sessionTutorProvider composes the two steps and is tested on both the
argument order and the default carve-out.

Also types conversationDocId's provider parameter as TutorProviderId
instead of string. It is interpolated straight into a document id, and
every other point on the path is already typed; a raw config or param value
containing "/" would have injected a path separator. Typing it moots the
escapeKey question, since no vocabulary entry can contain one.

IUnitConfig gains chatTutorProvider so it stops describing a different
shape than UnitConfiguration. It is deliberately not offered by the Chat
Tutor authoring page, which the declaration now says.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy
…aram is gated

100d326 removed the present-tense "the trigger routes on it" claim from
three files, but three more sites still asserted it. The firestore.rules
one mattered most: it is the stated justification for pinning the provider
to an ENUM rather than a type, and a rules reader cannot check it against
the trigger source. All three now use the forward-looking form the other
files already use.

The unit-configuration entry carried a second unverifiable claim in the
same breath — that other backends accept no prompt from CLUE — which was
the premise for dropping the prompts key under a non-default provider.
That reasoning did not survive review, so the claim goes with it.

Four comments also described chatProvider as a dev/qa tool, but nothing
gates it and the authed rules admit the resulting write. A student can
append it today and land on a fresh empty conversation, which reads as
their transcript vanishing. It is harmless while the server ignores the
field, so this documents the situation rather than adding a gate:
chatDebug and chatTutor are equally ungated, so ungated is the house style
here, and whoever adds the routing decides whether picking a paid backend
stays this open.

The spec gains a provider-selection section in its follow-on style, plus
`provider?` in the message doc shape and a pointer from the _p<hash>
bullet to the suffix that now precedes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy
@kswenson

Copy link
Copy Markdown
Member Author

Addressed in 16d385f, d14ec44, c919d5b, cc58832 — all three blocking items and every non-blocking one. One deviation, flagged at the end.

1. The prompt/provider contradiction

Confirmed, and it fails exactly as you describe. Fixed by stacking rather than gating: the id is now _v<provider>_p<hash>, with each suffix independent.

I went that way because the case for gating rests on the same claim that justified dropping the key in the first place — that prompt overrides are OpenAI-only because other backends accept no prompt from us. That is an assertion about a backend nobody has integrated yet, and it is false today, since OpenAI answers foreverlearning turns too. Gating would strip authored prompts from precisely the conversations QA uses to exercise this, and leave a re-enable for the routing PR to remember. The asymmetry settled it: if the claim turns out true, stacking costs a spurious fork per prompt edit on a path only dev/qa reaches; if it turns out false, gating costs a permanently wrong prompt with no reset. So I removed the claim rather than building on it — it is gone from the config doc and the transport comment as well.

conversation-key.test.ts now asserts the stack instead of the drop, and the transport's prompt-send comment points back at the id, so the two halves reference each other rather than contradicting each other.

2. The rules enum

src/components/chat-tutor/tutor-provider-rules.test.ts, built on the spikerbit precedent as suggested. Your count assertion earned its keep immediately: I mutation-checked both failure modes, and replacing both pins with is string leaves the equality assertion passing over an empty match set — only the count catches it. Adding a provider to just the demo block fails the equality one. Both rules blocks now point at the test.

3. The stale present-tense claims

All three restated forward-looking. I also rewrote the sentence next to the unit-configuration.ts one — "the other backends configure tutor behavior on their own side and accept no prompt from us" — which is the same unverifiable class and, as it happens, was the stated premise for item 1's design.

Related: my reply on the Copilot thread above ends by saying that comment asks whoever adds the routing to delete it. That is no longer true — the note was trimmed per your last non-blocking item.

Non-blocking

All taken.

  • provider is now typed TutorProviderId, which moots escapeKey — no vocabulary entry can contain a /.
  • sessionTutorProvider composes the two functions, tested on argument order and the default carve-out. Worth noting it covers less than the full wiring: deleting provider from the FirestoreTransport construction still leaves the suite green. That would need a chat-sidebar test, which I did not add.
  • chatProvider: took the comment fix, not the gate. As you suspected, ungated is the house style — chatTutor and chatDebug are the same — so the comments now say that plainly, and say the gate question belongs to whoever adds the routing.
  • Spec has a provider-selection section in its follow-on style, provider? in the doc shape, and a pointer from the _p<hash> bullet. Also dropped the hardcoded "28 cases" count, which was already stale.
  • docs/unit-configuration.md sentence fixed; its chatTutorPrompts clause now matches the stacking resolution.
  • IUnitConfig gains the property with a one-line note on why the authoring page does not offer it.
  • Demo test gains provider: 42; the authed test gains the provider: "openai" accept case, so both vocabulary entries are covered.

The deviation

On the transport comment I trimmed to the live fact plus the constraint routing has to honor, and dropped the instructions to a future engineer — but I did not add the ticket reference you asked for. Jira keys don't go in this repo's source, since it's public and the tracker isn't. The routing guidance lives in the ticket; the comment just states what is true now.

Verification

check:types and lint:build clean. 239 Jest tests across the chat-tutor, authoring, app-config and url-params suites. The firebase-test rules suite runs 31 green against the firestore emulator — that needs JAVA_HOME on a JDK 21, since firebase-tools 15 now refuses anything older and JDK 11 is still the default on at least my machine.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Comment thread src/components/chat-tutor/conversation-key.ts
Comment thread src/components/chat-tutor/tutor-provider.ts Outdated
…swers

The module header said it turns a param and a config value into "the one
backend that answers this conversation's turns". Five other sites were
rewritten to stop asserting the trigger routes on the provider; this one
was the module doing the resolving, and it made the claim most directly.

It now says what resolution actually buys on this side: the provider a
conversation is stamped and partitioned by. Which backend answers is still
unconditionally OpenAI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy

@dougmartin dougmartin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍

All three blocking items are addressed. I verified each rather than taking the commits at their word: the id now stacks _v<provider>_p<hash> with the default path still byte-identical and no collision possible between the suffixes (tutorPromptsKey is base-36, so it can never contain _); tutor-provider-rules.test.ts is genuinely picked up by the main Jest suite and fails on a dropped pin, a typo'd entry, or drift from kTutorProviders; and the present-tense routing claim is gone from all three sites I named. The stacking rationale is the right call, and the docs now describe the contract the code actually implements.

A few non-blocking notes (take or leave):

  • src/components/chat-tutor/firestore-transport.ts (lines 30-33): the provider option doc still says "which backend answers a turn, stamped on every message so the trigger (which can't read unit config) can route", which the comment eight lines below at 176-182 correctly contradicts. This is a sixth site of the claim the sweep was for, and arguably the one that matters most, since the options interface is where a new caller lands first. Suggest matching the corrected wording: "which backend the conversation is stamped and partitioned by."

  • firebase-test/src/chat-tutor-rules.test.ts (lines 287-301): the comment now says the demo block "gets the identical cases", but the demo test asserts only foreverlearning accepted plus the two rejections, while the authed block (lines 79-84) also asserts provider: "openai" is accepted. That makes the comment false as written, and it leaves unasserted exactly the case the comment at lines 76-78 argues is load-bearing, on the block that gets exercised first. One more expectWriteToSucceed with provider: "openai" under kDemoParent makes the comment true. Note tutor-provider-rules.test.ts catches drift between the two rules blocks but not the case where both omit an entry and the shared list is edited to match.

  • src/components/chat-tutor/firestore-transport.ts (lines 190-192): the stacking decision is justified in four places by "prompt overrides ride install-eligible sends regardless of provider", and nothing tests that half. No test file under src/ mentions promptReplace or promptAppend at all, so deleting line 191 leaves the suite green, and re-gating the send on !provider would silently reintroduce the pinned-forever conversation this round fixed. firestore-transport.test.ts already has the fake Firestore and the mocked serverTimestamp, so it is one case: build a transport with provider: "foreverlearning" and tutorPrompts: { replace: "..." } and assert added[0].promptReplace.

  • src/components/chat-tutor/tutor-provider-rules.test.ts (lines 33-37): the content assertion loops over a possibly-empty array, so on its own it passes vacuously; the guard lives in the sibling it at lines 29-31 by convention only, and skipping or renaming that one silently turns this into decoration. Collapsing both into expect(parseProviderEnums()).toEqual([[...kTutorProviders], [...kTutorProviders]]) fails on a missing pin, an extra pin, and any content drift, and cannot pass vacuously.

  • src/components/chat-tutor/tutor-provider-rules.test.ts (line 22): .replace(/^'|'$/g, "") strips single quotes only, so rewriting the rules array with double quotes (which Firestore rules accept) fails the test on a purely cosmetic reformat. It fails loudly rather than silently, so this is only a nit: /^['"]|['"]$/ covers both.

  • src/components/chat-tutor/tutor-provider.ts (lines 20 and 30): resolveTutorProvider and nonDefaultTutorProvider now have no production caller anywhere in the repo; the only importers are their own test file and sessionTutorProvider in the same module. Dropping export from both would keep the default carve-out from being bypassable: calling resolveTutorProvider directly returns "openai" rather than undefined, which would stamp the default onto message docs and break the byte-identical guarantee the design rests on. The two behaviors only they cover move onto sessionTutorProvider without loss: sessionTutorProvider("forever-learning", "foreverlearning") is "foreverlearning", and sessionTutorProvider(undefined, "gpt") is undefined.

  • shared/chat-tutor-providers.ts (lines 7-9): both firestore.rules comments were updated to name tutor-provider-rules.test.ts, but this one was not, and it is the comment someone adding a provider reads first. Worth appending that the test fails if either block drifts from this list.

  • src/components/chat-tutor/firestore-transport.ts (line 6) and src/components/chat-tutor/conversation-key.ts (line 1): TutorProviderId is imported from ./tutor-provider in one file and from ../../../shared/chat-tutor-providers in the other, three files apart in the same directory. The export type re-export at tutor-provider.ts line 11 exists only to serve the former, so importing from shared/ in both and deleting the re-export would leave one answer for a reader tracing the type.

  • Comment length, four sites: src/components/chat-tutor/firestore-transport.ts (lines 176-182, seven lines above a three-line if), src/authoring/types.ts (lines 58-61, four lines above one optional property), src/components/chat-tutor/tutor-provider-rules.test.ts (lines 5-14, a ten-line header), and src/components/chat-tutor/tutor-provider.ts (lines 34-38, five lines on a one-line function). Each is accurate and each carries real rationale, so none is a delete candidate, but all four are longer than their content needs. In the first, the "nothing reads it yet" paragraph is already stated in three other places and goes stale the day routing lands, so the durable half is the two sentences about why it is stamped on every message and what routing must do.

kswenson and others added 2 commits August 28, 2026 15:19
Three assertions that read as coverage but were not.

The stacking decision is justified in four comments by "prompt overrides ride
install-eligible sends regardless of provider", and nothing tested that half —
no file under src/ referenced promptReplace or promptAppend at all. Re-gating
the send on the default provider would have silently reinstated the pinned
forever conversation this branch just fixed. Verified by doing exactly that:
the mutation fails the non-default case and leaves the default one green.

tutor-provider-rules.test.ts looped its content check over a possibly-empty
match set, so it passed vacuously on its own; the count assertion that guarded
it lived in a sibling it(), which is convention, not enforcement — skip or
rename that one and the loop becomes decoration. Both collapse into a single
toEqual over the whole match set, which cannot pass vacuously and fails on a
missing pin, an extra pin, or content drift. The quote strip now accepts double
quotes too, so reformatting the rules array does not fail the test cosmetically.

The demo rules test claimed it got "the identical cases" as the authed block
while omitting the openai accept case, which is the one that catches a provider
dropped from both rules blocks and the shared list together — agreement the
consistency test reads as correct because it only compares the blocks to each
other. Confirmed against that exact mutation: the consistency test passes and
both accept cases fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy
resolveTutorProvider and nonDefaultTutorProvider had no production caller left
once the sidebar moved onto the composed helper — only their own tests and the
composition one line below. Unexporting them makes the default carve-out
unbypassable: resolveTutorProvider returns "openai" where sessionTutorProvider
returns undefined, so a future caller reaching for the more obvious-sounding
name would stamp the default onto message docs and break the byte-identical
guarantee the whole design rests on. Their two behaviors move onto the public
function without loss.

TutorProviderId now comes from shared/ everywhere. It was imported from
./tutor-provider in one file and from shared/ in its neighbor, and the
`export type` re-export existed only to serve the former.

Also drops the transport's "nothing reads it yet" paragraph, which the shared
header, the config doc and the PR description all state, and which goes stale
the day routing lands; the durable half is why the field rides every message
and what routing has to do with it. The provider option's own doc comment was
the last site still saying the trigger routes on the field — it contradicted
the comment 140 lines below it, and the options interface is where a new
caller looks first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShKusLrXiJePGEhddxovsy
@kswenson

Copy link
Copy Markdown
Member Author

All nine taken, with the last one partially. 79c89f65b and 6484b6368.

The three that were real coverage gaps. The prompt test is the one that mattered most — you were right that nothing under src/ touched promptReplace/promptAppend at all. Added both halves (default and non-default provider) and confirmed the mutation: re-gating the send on !provider fails the non-default case and leaves the default one green, so the pair actually pins the contract rather than just asserting prompts get sent.

The vacuous loop is a fair hit, and a pointed one — I added the count assertion specifically to prevent vacuous passing, then put it in a sibling it where it guards by convention only. Collapsed to the single toEqual you suggested. Re-ran the mutations against it: a dropped pin, an added pin, content drift in either block, and rewording both pins out of the regex's reach now each fail one assertion.

On the demo openai case, I verified the scenario you described rather than just adding the line — dropping openai from both rules blocks and kTutorProviders leaves the consistency test green, and both accept cases fail. That is now written down next to the test, since the division of labor between the two guards is not obvious from either one alone.

The rest. Unexported both helpers; the carve-out argument is the right one and I put it in the comment, since the failure mode is a future caller reaching for the more obvious-sounding name. TutorProviderId now comes from shared/ everywhere and the re-export is gone. The transport options doc no longer contradicts the comment 140 lines below it. The shared header now names both guards. Quote class widened.

Item 9, partially. I trimmed the transport's "nothing reads it yet" paragraph — you are right that it is stated in three other places and goes stale the day routing lands, so the durable half is why the field rides every message and what routing must do with it. I left the other three. Each is longer than its content, but the rationale in them is not recoverable elsewhere: why the rules test reads from disk instead of importing, why the authoring page deliberately omits a property, and why the composition helper exists at all. Length felt like the wrong metric there — whether a reader can find the reasoning somewhere else seemed like the right one, and for those three they cannot.

Full suite 4001 across 356 suites, rules suite 31 green on the emulator, check:types and lint:build clean. PR description updated for the new counts and mutation checks.

@kswenson
kswenson merged commit 3bac359 into master Aug 28, 2026
27 of 28 checks passed
@kswenson
kswenson deleted the CLUE-578-tutor-provider-selection branch August 28, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants