Repair Firestore document metadata against the realtime database [CLUE-643]

[scytacki]

Repair scripts only — no product code changes, and nothing here has been run against a database yet.

The problem

Firestore document metadata has drifted from the realtime database in two ways:

1. context_id names the wrong class. A metadata row records a class the document does not
   actually live in. 35 rows in production, roughly 8 elsewhere. Ten of the 35 carry the literal
   string "ignored".
2. The metadata row is missing entirely. 6,240 realtime-database documents across all real
   spaces have no Firestore row at all, 4,996 of them in demo spaces. A document with no row is
   invisible to Sort Work, to the class dashboard, and to every other Firestore-driven view.

The client-side cause is already fixed, so this is a one-time repair rather than something that
needs to keep running. It should run before any other sweep that resolves a document through its
context_id, since both defects would make such a sweep mis-report — specifically before the
offeringId backfill in #2980.

What's here

scripts/lib/rtdb-document-index.ts            shared: index a space from both realtime-database halves
scripts/lib/repair-cli.ts                     space selection, database URLs, retrying REST reader
scripts/lib/curriculum-position.ts            decode an offering id, validate against content.json
scripts/lib/document-tools.ts                 derive `tools` from content, as the client does
scripts/lib/deletion-plan.ts                  what a deletion run may remove, and why it refuses the rest
scripts/repair-document-context-id.ts         repair 1
scripts/create-missing-document-metadata.ts   repair 2
scripts/delete-unrepairable-documents.ts      the residue repair 2 cannot fix
docs/document-metadata/firestore-migration.md what only the realtime database holds

Three scripts rather than one with modes: they have different risk profiles — one rewrites a field
on live rows, one creates rows, one deletes nodes — and an operator will want to run and judge them
separately.

The index

For one space, walk classes → users and read both child lists per user-class pair,
documents and documentMetadata. Reads are shallow (?shallow=true) over the REST API rather
than .once("value"), which would pull entire subtrees for data the index does not need.

Having both halves is what makes the skip rules below decidable: it distinguishes a document whose
content exists from a metadata node whose content is gone.

Both repairs drive from the index and never from the legacy contextId field, which is exactly the
field known to be wrong.

What repair 2 writes

The field set was checked against all 114,882 production rows rather than against the
IDocumentMetadata interface, since a field can be declared and never written. A row gets:

field	where it comes from
key type uid createdAt	the realtime-database metadata node
context_id	the class the document actually lives in, per the index
network	null, matching what the client stamps for students
properties	{}
title visibility	the node, when it has them
unit	the curriculum position, or an explicit null for a class-contained document
unit investigation problem offeringId	offering-contained documents only
originDoc	the class publication list, for the two types that use it
tools	derived from the document's content, as the client derives it

Three of those need justifying, because a naive reading of the interface would get them wrong:

unit: null, not omitted. Sort Work finds class-contained documents with
.where("unit", "==", null) (sorted-documents.ts), and Firestore cannot match a field that is
absent. A row without it is invisible under every filter but "All". All 19,649 class-contained rows
in production carry it, and the client's class container stamps it.

visibility and tools are copied rather than left to the client. Both are maintained by
useDocumentSyncToFirebase, whose updater finds rows by query and calls update() — so with no
row to match, every visibility toggle and every content save these documents ever had wrote nothing
at all. Waiting for the owner's next edit is not realistic for a document last touched years ago.
tools in particular decides how Sort Work groups a document; without it all 5,682 file under "No
Tools".

The same query filters on context_id == user.classHash, which gives repair 1 a consequence beyond
tidiness: a row whose context_id is wrong has been silently failing to record tools and
visibility for as long as the field has been wrong.

strategies is deliberately absent. It is recomputed by on-document-tagged from the comments
subcollection of the metadata row. A document with no row can have no comments, so there is nothing
to derive, and it fills in the first time a teacher tags one.

Two fields on many existing rows are deliberately not written. teachers is a denormalized
class-teacher list that firestore.rules uses only as a fallback for "legacy documents which contain
their own list" — the live path resolves teachers from context_id. And contextId (camelCase,
distinct from context_id) holds the literal string "ignored" in 130 of 130 sampled rows; it is
the dead field from the deleted v1 comment path, the same one repair 1 refuses to trust.

Repair 2's curriculum ladder

An offering-contained document needs unit/investigation/problem. The script tries, in order:

1. a sibling document in the same offering that already has a row;
2. for authed/ spaces only, the portal, which returns the offering's activity_url;
3. for demo spaces, decoding the offering id — `${unitCode}${investigation * 100 + problem}`
   — and a bare numeric id decodes against the default sas unit.

Every candidate is validated against the unit's content.json before it is used. A document that
survives none of these is reported and skipped, never guessed at.

Skip rules

Refusals, not filters — each prevents a write that would make things worse:

• Metadata with no content — never create a row. Its content is gone; a row would promote an
  invisible orphan into a Sort Work entry that throws when opened.
• Key absent from the realtime database — never touch. Firestore-native rows, meaningless to
  look up in documentMetadata.
• Content with no metadata node anywhere — out of scope. Six in the whole database, small
  enough to handle by hand.
• Keys that are not realtime-database-addressable (., #, $, [, ], /).

The deletion script

The skip rules leave a residue that no product surface can reach. delete-unrepairable-documents.ts
removes it from the realtime database, reading the report a repair-2 dry run writes. It never
touches Firestore, because by definition these documents have no Firestore row. Its guards:

• authed/learn_concord_org is never touched, under any flag.
• Nothing created inside the retention window — a year by default.
• Nothing whose path cannot be derived. A guessed path is worse than a skipped document.
• Nothing that names no node to delete, which would be a silent no-op counted as success.
• Content is deleted before metadata, so an interrupted run leaves documents the same report
  would classify the same way next time.
• Every document is re-checked against the live database immediately before removal. A stale
  report can only cause a skip, which the run reports — never a wrong deletion.

Dry-run results (2026-08-25, all spaces)

Repair 2:

created            5,682     rows it would write
alreadyPresent   119,819
skipped              665     unresolvedCurriculum 573, no content 86, no metadata node 6
unreadableContent      0     every created row got a tools value

The sweep takes about 19 minutes, up from 12 before tools was derived: the content read happens
last, after every skip check, so the documents the run declines never pull the largest node in the
database.

Production resolves completely: every offering-contained document there found its curriculum
position, and nothing in production is left unresolved.

Deletion: 662 documents / 1,235 realtime-database nodes planned, 3 refused as protected — the
three production entries, which are content with no metadata node and want looking at individually
rather than sweeping. None of the 665 was created in the past year; the newest dates from
2025-04-25 and 455 of them predate 2023.

Safety

Every script is dry run by default, APPLY=1 to act, and prints its mode at startup. Writes
batch at 400 and increment the written count only after a commit resolves, so a crash cannot
overstate what landed. Per-space and per-type counts are reported for every bucket including the
skipped ones — judge a run by the per-space lines rather than the totals. SPACES= limits a run to
named spaces for a staged rollout. qa, dev and test spaces are refused outright.

The realtime-database reader retries a 401 by refreshing its token, since a sweep over a large space
outlives the token it started with, and retries a thrown fetch with backoff, since a sweep making
tens of thousands of requests will hit a transport failure sooner or later. It never returns empty
on failure — that would read as "this space has no documents", and the run would report a clean
sweep having looked at nothing.

Also in this PR

docs/document-metadata/firestore-migration.md gains a section on metadata that still lives only in
the realtime database, including a field that would collide with an existing Firestore field if it
were mirrored under its current name.

scripts/output/ is added to .gitignore, and is where a run's reports go. Ignored as a directory
rather than by filename: a report names real classes and users, so the default has to be "not
committed" for anything a run writes rather than for the one filename that exists today.

Testing

120 unit tests across 8 suites, against a mock Firestore and a mock realtime database. Worth a look
in review: the batch-tail case (a commit that fails does not increment written), the duplicate-home
case (a key mapping to two class/user pairs is reported as a violation), the tools module's
distinction between a document that was never saved and content that will not parse, and the
deletion plan's refusal cases.

Not in this PR

Running any of it. The repairs and the deletion both need a separate go-ahead.

The intended order is repair 2, then re-run its dry run, then delete from a report that reflects
post-repair reality — rather than deleting first. Both orders cost the same three passes, because
the deletion's input is a repair dry run. What differs is the risk: 330 of the 573
unresolved-curriculum documents sit in authed/ spaces, where the curriculum position is resolved by
a call to the portal. A transient portal failure buckets them as unresolvable, and a deletion run
would then remove documents that a later run could have repaired. Deleting last means the residue has
been confirmed by the repair that actually ran, and the before/after reports can be compared for
stability first.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 69.33045% with 142 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.53%. Comparing base (274886b) to head (419ed88).
⚠️ Report is 37 commits behind head on master.

Files with missing lines	Patch %	Lines
scripts/create-missing-document-metadata.ts	51.30%	93 Missing ⚠️
scripts/repair-document-context-id.ts	52.27%	42 Missing ⚠️
scripts/lib/repair-cli.ts	89.23%	7 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (274886b) and HEAD (419ed88). Click for more details.

HEAD has 13 uploads less than BASE

Flag	BASE (274886b)	HEAD (419ed88)
cypress-regression	13	0

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2977       +/-   ##
===========================================
- Coverage   85.67%   70.53%   -15.15%     
===========================================
  Files         987      990        +3     
  Lines       56591    57038      +447     
  Branches    14946    15070      +124     
===========================================
- Hits        48487    40233     -8254     
- Misses       8084    16769     +8685     
- Partials       20       36       +16     

Flag	Coverage Δ
cypress-regression	?
cypress-smoke	41.30% <ø> (+<0.01%)	⬆️
jest	57.60% <69.33%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cypress]

collaborative-learning    Run #20091

Run Properties:   Passed #20091  •  419ed88163: docs: design for repairing Firestore metadata against the realtime database [CLU...

Project	collaborative-learning
Branch Review	CLUE-643-metadata-repair
Run status	Passed #20091
Run duration	03m 23s
Commit	419ed88163: docs: design for repairing Firestore metadata against the realtime database [CLU...
Committer	Scott Cytacki
View all properties for this run ↗︎

---

Test results
Failures	0
Flaky	0
Pending	0
Skipped	0
Passing	4
View all changes introduced in this branch ↗︎