Integration/promote to prod - #62
Merged
Merged
Conversation
- Move file-key building, .nlogo parsing, legacy row access, dedupe and the per-node migration routine out of archive.ts so patch.ts can reuse them. - Cover the extracted code with unit tests, including the exact write sequence createModelFromNode emits against a recording transaction. - Fix parseNlogox: it called DOMParser, which Node does not have, so every .nlogox model silently stored a null netlogoVersion and infoTab. Replaced with a dependency-free parser. Already-migrated rows still hold nulls. - Pin dob to UTC midnight. pg reads a `date` as local midnight while Prisma stores @db.Date from the UTC day, so a birthdate never compared equal to itself and shifted by a day in positive UTC offsets. - Replace the shell-based avatar copy with fs.copyFile; it interpolated a legacy filename into a command string. - Type-check prisma/lib and the migration scripts.
archive.ts only skips rows that already carry a legacyId, so it cannot see edits, deletions or new children of an already-migrated node. patch.ts applies those from a diffdb.sh diff. - Materialises the whole change set as a plan before writing, so the dry run and the real run compute the same thing. Applying replays it in one transaction and writes an expectations manifest for --verify-only. - Trusts only `side` and `id` from the diff and re-reads rows from the legacy snapshot, since diffdb.sh md5s the `contents` columns. Deleted rows are the exception; their row_to_json is the only surviving copy. - Addresses everything by legacyId. No existing row's id column is touched. - Recomputes previews from the whole attachment set rather than patching them: a preview is one column on the latest version and the highest legacy attachment id wins it. - Moves tags and the preview onto an appended version, matching archive.ts, which keeps them on the latest version only. - Leaves updatedAt alone; Prisma's @updatedat overrode it on create, so no migrated row ever held the legacy value. - Refuses to guess on mid-sequence version deletes, out-of-order appends and in-place attachment edits, none of which can be mapped back safely.
Builds a legacy database with a migrated baseline and an incoming schema carrying sixteen change scenarios, archives the baseline, diffs, patches, then archives the incoming snapshot into a second database and compares canonical uuid-free dumps of both. Asserts that archive(baseline) + patch(diff) equals archive(new snapshot), with the soft-deleted model checked separately since a fresh archive never sees it.
A node that gained a version and lost its last preview attachment in the same diff kept the old preview: the resync compared against the database row for the appended version, which does not exist yet, read null, and concluded a cleared preview was already applied. The append then carried the dead key forward. Compare against what the append will carry instead, and clear that carry when a resync supersedes it. Covered by a rehearsal scenario; the script's own verify passed this because the manifest was built from the same wrong plan.
From a review pass over the patch script. - Derive a version's object key from its legacy id instead of a fresh uuid, and diff the row before writing. A modified version previously uploaded a duplicate object and repointed netlogoFileKey on every run, so re-applying the same diff was not the no-op it was documented to be. - Refuse an ambiguous ModelAdditionalFile match rather than deleting whichever row came back first. Two attachments on one model sharing a filename, or a null created_at, could otherwise delete the wrong file irrecoverably. - Drop a tag from the resolution map once it is scheduled for deletion, so a tagging in the same diff is not created and then cascade-deleted. - Assert the appended version's preview in the manifest; only carried tags were covered, so a wrong carried preview could pass verification. - Strip every uuid segment when comparing preview keys, not just the first, which was the model uuid and identical on both sides. Rehearsal now covers an in-place version edit.
…gacy-migration - rename archive.ts, patch.ts and archive-upload.ts to initial-import.ts, apply-diff.ts and upload-files.ts, naming the job rather than the source - move the staged output and avatar snapshot under the same directory - run db:legacy:patch through tsx directly; wrapping it in `sh -c` swallowed the --apply / --verify-only / --skip-upload flags it reads from argv
- ModelAuthor.collaboratorType carries legacy collaborator_types.name - NonMemberContributor archives credited people who never held an account Both are provenance only; no business logic reads either.
initial-import.ts derived ModelAuthor from version history alone, so a collaborator who never uploaded a version had no row at all. - members fold into ModelAuthor as role=contributor, which grants write access since canWrite treats any contributor as a writer - non-members are archived in NonMemberContributor, unread by the app - idempotent: keyed on the existing (modelId, userId) rows and on NonMemberContributor.legacyId, so a second run reports nothing to do
migrateInteractions tracked its per-table count locally and logged it, but never wrote it back to the report, so report.json always claimed zero. The docs took that at face value: the phase writes ~6.2M rows on the production snapshot. Also records the full-scale rehearsal and the recompute step, which nothing referenced even though the import leaves the denormalized Model counters at 0.
sessions (1.3M rows) and ip_locations (120k) map to nothing in the new schema, so apply-diff.ts always discarded their diffs. Scanning them was most of the script's runtime. Output is byte-identical without them.
- Show a one-time modal on login and sign-up pointing pre-migration modelingcommons.org users at password reset - Suppress it for logged-in users, after dismissal, and past the sunset date constant - Extract getResetPasswordLink so the modal and ReclaimAccount share one link builder
- Primary action reads Next and advances until the last step, where it publishes - Move Revert changes to an mdi icon button in the editor header - Keep edit mode single-shot: its steps are freely navigable sections, not a gated wizard
- Drop the assumption that the visitor has an old account - Name the emailed link instead of telling them to reset a password - Label the actions Reclaim account and I didn't have an old account - Add a note for visitors who do not remember having an account
findByIdOrName used a UUID-shaped regex to decide between an id lookup and a name lookup. A NanoID is an ordinary-looking string that a tag name could also be, so the heuristic misroutes any NanoID-shaped tag name to a 404. Look up by id first and fall back to name on a miss; ids are unique, so a hit is authoritative. This costs one extra query on the name path, which is acceptable.
Switch x-correlation-id and Fastify genReqId generation from randomUUID to newId. Rename validateUUIDv4 to validateRequestId and widen it to accept either a canonical UUID or a NanoID, since these headers commonly arrive as UUIDs from upstream proxies and tracing systems. This is the only place in the codebase that still accepts UUID-shaped ids, and it is deliberate upstream interop, not backward compatibility.
Hash the namespaced key with SHA-256 and map 21 digest bytes into the nanoid alphabet via byte & 63, instead of bit-forcing a UUIDv5 shape from SHA-1. Determinism and idempotency are unchanged; only the output encoding moves from UUID to nanoid.
Replace randomUUID with newId in initial-import.ts and apply-diff.ts for every generated row id. Idempotency comes from legacyId columns, not id determinism, so this is a pure generator swap. Rename NodeMigrationDeps.newUuid to newId and update all callers and specs. Rename derivedUuid to storagePathHash and add a pinned-output test: its uuid-shaped S3 key format is frozen on purpose and must stay byte-for-byte identical to avoid orphaning existing storage objects.
Swap every remaining randomUUID() entity-id generator for the shared
newId() helper (model, permission, tag, additional-file, interaction,
draft domains, plus the draft-file entries embedded in draft JSON) and
point Better Auth's generateId at the same helper.
Widen storage-key path segments from randomUUID().substring(0, 8) (32
bits of entropy) to nanoid(10), applied consistently across every
storage-key call site since none of them parse the segment back.
Replace every Type.String({ format: 'uuid' }) validator with idSchema()
across the 18 remaining DTO/schema files, drop the now-unused 'uuid'
ajv format, and fix ID_EXAMPLE to a real NanoID literal so Swagger
stops advertising a UUID-shaped example that fails its own pattern.
Tighten existing domain and storage-key specs to assert the NanoID
shape instead of the old UUID/hex regex.
filenameFromKey still sliced a fixed 37 chars off the last path segment, a leftover assumption from the 36-char UUID prefix the writer no longer emits. Against the new nanoid(10) staging-key prefix this either leaked the prefix into the filename (short keys) or truncated a genuine long filename (keys over 37 chars). Three shapes actually reach this reader: createStorageKey keys put the random segment in its own path directory, so the last segment is already the bare filename; legacy pre-migration keys and the current stagingKey both join the random segment and filename with a dash in one segment. Strip a canonical-UUID-shaped prefix for the legacy case, and a fixed STAGING_KEY_RANDOM_SEGMENT_LENGTH-char prefix only when the key path is staging-shaped, since the nanoid alphabet includes '-' and an ordinary filename could otherwise be mistaken for a random segment. The length is now a named constant shared with stagingKey instead of a magic number. Also drop the stale 'Tag UUID or case-insensitive name' description in tag.schemas.ts; the param is a NanoID now.
Replace the UUID/hex id regexes in the e2e timing collector with a matcher anchored to the 21-char nanoid alphabet, plus the legacy dashed 36-char format for older captured reports, so requests to routes like /v1/models/:id collapse correctly again instead of fragmenting the perf report. Also swap the nil UUID literal in the user profile auth feature for a nanoid-shaped placeholder so that scenario still exercises 401 instead of tripping the id-format 400.
- Drop and recreate foreign keys around the swap rather than relying on ON UPDATE CASCADE, whose firing order is undefined for Model.id - Resolve id-bearing columns from the catalog to a fixpoint, so columns that reach a mapped table indirectly are not missed - Rewrite soft references by map membership, never by shape, leaving the frozen storagePathHash segments intact - Purge unpublished drafts and their uploads instead of migrating them
- ":memory:" was never in-memory: Nuxt Content rewrites it to
`{ name: "memory" }` and db0 only takes its in-memory branch when the
name is still ":memory:", so it fell back to a file under the process
CWD and every content query threw EACCES in the image
- /tmp is writable both in the container and on a machine running the
e2e suite in server mode, and matches Nuxt Content's own presets
The schema added ModelVersion.changeSummary but nothing in src ever referenced it, so every hand-built ModelVersionEntity was missing a required field and check-types failed across five files.
The tag search sent an `offset` query param, but the API only accepts `limit` and `page`. Fastify dropped the unknown param and defaulted page to 0, so every "load more" refetched the first 20 tags and the tag select menu could never reach the rest.
Lists tags by popularity with their model-version counts, and switches to prefix search while the search box has a value. Both listings page in on scroll. Prefix search carries no counts, so TagCard now takes an optional description.
The home page issued six per-request API calls, all of them public and identical for every visitor. They now come from a single `/_data/home` handler cached for ten minutes with stale-while-revalidate, so the page costs one internal call and the backend sees one set of queries per TTL rather than one per visit. Nitro's cached handler strips every header but the declared `varies` list, so the shared entry is built without a session and cannot leak one.
Recents went stale under the shared feed's 45 minute TTL. They now come from their own `/_data/home-recent` handler cached for a minute, loaded lazily so they never block the sections around them, and rendered with skeletons until they land. The Trending Tags sidebar was anchored to the section's array index, which no longer holds once recents resolve on their own, so it keys off the section instead. Recents are also kept out of the marquee, which would otherwise reshuffle every column when they arrive.
Which section loads separately was re-derived in three places by comparing against the recent section's key, and the feed's section list was hand maintained alongside the render order. A section now declares `deferred`, the feed list is derived from it, and a test asserts every rendered section has exactly one fetcher.
On a checkout with no .nuxt, `nuxt build` exits 0 but emits a client.manifest chunk containing `default: default`, which is a syntax error. Nitro loads that chunk lazily, so the server starts, /health and every server route answer 200, and only page renders fail. `nuxt:generate` already guarded against this; `nuxt:build` did not.
…ion/promote-to-prod
Runbook, decisions, refusal cases and known divergences from a from-scratch archive.
The macOS periodic cleaner deletes files under /tmp by access time, which silently emptied template0/template1 and removed PG_VERSION and every .conf from the postgres cluster while leaving directories behind. Point the bind mounts at XDG_DATA_HOME instead.
Enumerates every row through Prisma and asserts the API endpoints, stored objects and legacy URLs each row implies. Route coverage is derived from the OpenAPI document, so a GET route that is neither expanded nor listed in SKIP aborts the run. Expected status is carried per row rather than assumed to be 200, so a private model answering anything but 403 and a soft-deleted one answering anything but 404 are both failures.
…n CI The Dockerfile ARG defaults are what ships when a build forgets to pass the GitHub environment variables, so leaving them on the beta host meant a silent wrong-host image. The CI step fails the build if the host reappears anywhere outside doc/ and .ongoing/.
The paginated response field is 'count' (shared/api/paginated.response.base.ts). Reading 'total' yielded undefined, so every walk collapsed to page 1 and the sweep never requested a second page.
Scopes the truncating Before hook to 'not @data-integrity' so cleanDatabase is unreachable from the tagged path, and adds a dataIntegrity profile that runs read-only invariant checks against any populated environment. A row-count snapshot around the run proves at runtime that the cohort writes nothing, and an empty database fails loudly rather than passing vacuously.
Wires verify-inputs, restore, migrate, diff, patch, collaborators, recompute, id-migration, verify and dump into one checkpointed entry point. A failed step names itself and leaves no checkpoint, so a bare re-run resumes there. Loading the beta snapshot over the target is opt-in behind --restore-target and requires --yes. On cutover the target is the production beta database, where restoring a snapshot would discard every write since it was taken.
Sourcing .env overwrote already-exported values, so an operator who exported a production DATABASE_URL would have it silently replaced by the development one. In testing this retargeted a --restore-target run onto the local dev database. Only unset names are filled in now. Also adds --page-size to the sweep so the multi-page walk is exercisable against collections smaller than one page.
GET /v1/users/:id/models returned every authorship row for a user with no visibility or deletion filter, so an anonymous caller learned the ids of that user's private and soft-deleted models. Adds readableModelFilter, canRead expressed as a Prisma where clause, and applies it to both the page and the count. A listing cannot call the policy per row because the paginated count has to reflect the same predicate as the page. The filter shares viewerIsActive and viewerIsGlobalAdmin with the policy rather than restating them, and a spec pins the two together across every viewer, visibility, role, grant and deletion state.
buildMap emitted one INSERT for the whole map and spread its parameters into the call. The production map holds 6190632 rows, so that is 18.5 million spread arguments: it dies with RangeError: Maximum call stack size exceeded, and would have exceeded the 65535 bind-parameter limit had it got that far. Only the persist path was affected, so --dry-run reported success immediately before --yes crashed, and no development-sized database reaches either limit. Statements are now capped at 5000 rows. Placeholder numbering restarts per statement, which the spec pins: carrying a running index across chunks fails only on datasets large enough to need a second statement.
The production map serialises to 415MB against a V8 maximum string length near 512MB, so JSON.stringify was running at ~80% of a hard ceiling and the cutover uses a larger, fresher dump. Exceeding it throws RangeError: Invalid string length from writeMapFile, which runs after the id swap has committed and after superseded storage objects have been deleted, leaving a migrated database with no checkpoint and no map. Output is byte-identical to the previous format; the spec pins it to JSON.stringify so the artifact a human opens is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #53 #54 #55 #59 #61.