feat(migrations)!: ship SQL migrations applied like runMigrations

CHANGELOG.md

@@ -124,6 +124,13 @@ always called out under their own heading.
 
 ### Breaking
 
+- `runArtifactMigrations(config, { schema })` takes the same arguments as
+  Interchange's `runMigrations`: a `DBConfig` and the host schema holding
+  `tenant` and `principal`. It applies the SQL files shipped under
+  `migrations/`, all idempotent, with no ledger. The `adopt` option,
+  `RunArtifactMigrationsOptions`, `MigrationChecksumError` and
+  `MigrationAdoptError` are removed, and the `artifacts.migrations` ledger
+  table is dropped on the next boot.
 - The drizzle tables (`artifact`, `artifactVersion`, `upload`,
   `mailAttachmentRef`) are no longer exported from the package entry. Hosts
   reach artifacts through the routes and functions; `ARTIFACTS_SCHEMA` and the

CONTRIBUTING.md

@@ -70,12 +70,12 @@ make this package uninstallable outside the project that defines it.
 
 ## Migrations
 
-Shipped migrations are immutable. Each ledger row records a checksum of the migration's
-rendered SQL, so editing one that has already been applied fails with
-`MigrationChecksumError` on the next boot rather than letting fresh and existing
-databases diverge. Add a new migration instead.
+Migrations are SQL files under `migrations/`, applied in filename order on every
+boot, so every statement must be idempotent (`IF NOT EXISTS`, `IF EXISTS`). There
+is no ledger: a schema change is a new file whose statements are safe to re-run,
+never an edit that assumes it runs once.
 
-`schema.ts` and `migrations.ts` must agree — every query goes through the drizzle
+`schema.ts` and `migrations/` must agree — every query goes through the drizzle
 table objects, and a test asserts the migrations create exactly the tables
 `schema.ts` declares, no more and no less. Change one, change the other, in the
 same commit.
@@ -235,20 +235,20 @@ which store is installed.
 ### Data model
 
 Four physical tables — `artifact`, `artifact_version`, `upload`,
-`mail_attachment_ref` — plus this package's own migration ledger.
+`mail_attachment_ref`.
 
 **Hard control-plane foreign keys, by design.** `tenant_id` is `NOT NULL` and
-references `public.tenant(id)` (`ON DELETE CASCADE` — a deleted tenant takes its
+references the host's `tenant(id)` (`ON DELETE CASCADE` — a deleted tenant takes its
 artifacts with it) and `principal_id` / `owner_principal_id` reference
-`public.principal(id)` (`ON DELETE SET NULL` — a removed principal detaches its
+the host's `principal(id)` (`ON DELETE SET NULL` — a removed principal detaches its
 artifacts rather than destroying them). This package is coupled to Interchange:
 it mounts on Interchange-shaped hosts only, and the host's own migrations must
 have run before `runArtifactMigrations`. The internal key —
 `artifact_version.artifact_id` — cascades with its artifact.
 
 **Cheap row-local CHECKs.** `artifact.version` and `artifact_version.version`
 must be ≥ 1; `upload.size` and `mail_attachment_ref.size` must be ≥ 0. These are
-single-column constraints applied by a ledgered migration — free at write time.
+single-column constraints — free at write time.
 
 **Principal↔tenant alignment is host-owned.** The package FKs each column into
 the control plane independently; it does **not** enforce that `principal_id` (or
@@ -302,9 +302,7 @@ path that promises it.
 Separately, this package also has no way to confirm existing tenants are
 already free of duplicate `(title, kind)` rows, which would make even a
 scoped constraint risky to backfill. That is not the main reason for
-rejecting the constraint, and it is not by itself decisive. See
-`0003_schema_invariants` for this repo's own pattern for guarding a
-migration against exactly that kind of bad existing data.
+rejecting the constraint, and it is not by itself decisive.
 
 Instead, `findOrVersionArtifact(db, args)` (in `artifacts.ts`) closes the
 race with a transaction-scoped advisory lock keyed by
@@ -336,55 +334,35 @@ behind it.
 
 ### Migration runner
 
-`runArtifactMigrations(db)` is idempotent and safe to call unconditionally on
-every boot of every replica.
+`runArtifactMigrations(config, { schema })` takes the same arguments as
+Interchange's `runMigrations`, and a host calls it right after that, with the
+same values. `schema` is where the host's `tenant` and `principal` tables live;
+the runner rewrites the `"public".` foreign-key references in the SQL files to
+it. It is idempotent and safe to call on every boot of every replica.
 
-- The whole run is one transaction whose first statements are
-  `SET LOCAL client_min_messages = warning` and a **transaction-scoped**
-  advisory lock. A transaction pins one pooled connection, so the lock, the
-  ledger read and the DDL are the same session; the lock releases on commit or
-  rollback, so there is no unlock call to lose on an error path.
+- The whole run is one transaction whose first statement takes a
+  **transaction-scoped** advisory lock, so the lock releases on commit or
+  rollback and there is no unlock call to lose on an error path.
   `CREATE TABLE IF NOT EXISTS` is not itself race-safe, so the lock — not the
   `IF NOT EXISTS` — is what makes concurrent cold starts safe.
-- Lowering `client_min_messages` is why a re-run prints **nothing**: every
-  statement is `IF NOT EXISTS`, and on the second boot Postgres answers each with
-  a NOTICE that postgres.js would otherwise dump to the console, making a clean
-  re-boot look like a wall of errors. `SET LOCAL` scopes it to the transaction
-  and stops at NOTICE — WARNING and above still reach the host.
-- Each migration applies inside a nested transaction (a savepoint) together with
-  its ledger row, so a migration can never be recorded as applied with only some
-  of its statements run.
-- The ledger is this package's own table, `artifacts.migrations`,
-  never shared with a host's. Each row records a **checksum of the migration's
-  rendered SQL**, so editing a shipped migration fails with
-  `MigrationChecksumError` on the next boot instead of letting existing and
-  fresh databases diverge silently. Ship a new migration instead. The column is
-  `NOT NULL`, so the guarantee is unconditional: there is no unrecorded row for
-  the runner to adopt and wave through.
+- The runner opens its own single-connection client and discards NOTICEs, so a
+  re-run, where Postgres answers every `IF NOT EXISTS` with a NOTICE, prints
+  nothing.
 - Event timestamps (`created_at`, `updated_at`, `archived_at`) are
-  **`timestamptz`**. The initial create migration still lays them down as
-  zoneless `timestamp`; a follow-on migration retypes them with
-  `USING col AT TIME ZONE 'UTC'`, treating existing walls as the UTC clocks the
-  package always assumed. List keyset cursors project through
-  `AT TIME ZONE 'UTC'` and compare with `::timestamptz`, so paging and date
-  filters stay on the absolute instant under any session `TimeZone`. Rollback is
-  the reverse cast (`TYPE timestamp USING col AT TIME ZONE 'UTC'`) plus a new
-  ledgered migration — never edit a shipped one.
-- A later ledgered migration sets `artifact.tenant_id NOT NULL` and adds the
-  version/size CHECKs. If null-tenant rows still exist, that migration raises
-  before altering the column so the operator can clean them up first.
-- Empty ledger + pre-existing package objects fails closed
-  (`MigrationAdoptError`). `{ adopt: true }` records checksums without re-DDL
-  only after shape validation: tables, column types, required nullability, and
-  the named CHECK constraints. Column presence alone is not enough.
-
-**The package owns its own Postgres schema.** Every table, index and the ledger
-live in `artifacts`, created by the runner and qualified in every
-DDL statement and every query — nothing resolves through `search_path`, so the
-package shares a database with the host's control plane without ever being able
-to collide with (or silently adopt) a host table of the same name. The coupling
-to the host is explicit instead: `tenant_id` and the principal columns are hard
-FKs into `public.tenant` / `public.principal` (see the data model).
+  **`timestamptz`**. List keyset cursors project through `AT TIME ZONE 'UTC'`
+  and compare with `::timestamptz`, so paging and date filters stay on the
+  absolute instant under any session `TimeZone`.
+- Releases up to 0.1.0 kept a checksum ledger in `artifacts.migrations`.
+  `0002_drop_migration_ledger.sql` removes it; a database migrated by 0.1.0
+  already has the shape `0001_artifacts.sql` creates, so its statements no-op.
+
+**The package owns its own Postgres schema.** Every table and index lives in
+`artifacts`, created by the runner and qualified in every DDL statement and
+every query — nothing resolves through `search_path`, so the package shares a
+database with the host's control plane without ever being able to collide with
+(or silently adopt) a host table of the same name. The coupling to the host is
+explicit instead: `tenant_id` and the principal columns are hard FKs into the
+host schema's `tenant` / `principal` (see the data model).
 
 ### Boundaries
 

README.md

@@ -10,13 +10,17 @@ npm add @corbits/artifacts
 
 Requires Node 24 or newer and `@intx/*` 0.4.0 or newer.
 
-Open a database handle and apply this package's migrations at boot, before mounting any routes. A host that already has a drizzle handle passes that instead of calling `createArtifactDb`.
+At boot, right after Interchange's `runMigrations`, apply this package's migrations with the same `config` and `schema`. The tables go in their own `artifacts` Postgres schema, with tenant and principal foreign keys pointing into `schema`. Then open a database handle for the routes; a host that already has a drizzle handle passes that instead of calling `createArtifactDb`.
 
 ```ts
+import { runMigrations } from "@intx/db";
 import { createArtifactDb, runArtifactMigrations } from "@corbits/artifacts";
 
+// `config` is the host's `DBConfig` from `@intx/db`.
+await runMigrations(config, { schema: "public" });
+await runArtifactMigrations(config, { schema: "public" });
+
 const { db, close } = createArtifactDb(process.env.DATABASE_URL!);
-await runArtifactMigrations(db);
 
 // on shutdown
 await close();

examples/reference-host/src/index.ts

@@ -22,7 +22,13 @@ import {
   type RequireGrant,
   type TenantEnv,
 } from "@intx/hub-api";
-import { createDB, createGrantStore, runMigrations, schema as intxSchema } from "@intx/db";
+import {
+  createDB,
+  createGrantStore,
+  runMigrations,
+  schema as intxSchema,
+  type DBConfig,
+} from "@intx/db";
 // Interchange owns its id scheme; the host mints its OWN control-plane rows
 // with it rather than inventing a second one.
 import { generateId } from "@intx/hub-common";
@@ -50,7 +56,7 @@ export const DATABASE_URL =
 
 const EPOCH = new Date(0);
 
-function parsePostgresUrl(raw: string) {
+function parsePostgresUrl(raw: string): DBConfig {
   const url = new URL(raw);
   return {
     host: url.hostname,
@@ -103,6 +109,8 @@ export type Session = { userId: string } | null;
 
 export type ReferenceHost = {
   db: ArtifactDb;
+  /** The `DBConfig` the host migrates with. */
+  config: DBConfig;
   /** Interchange tenant id every artifact in this host is scoped to. */
   tenantId: string;
   /** Principal id of the agent Alice owns. */
@@ -130,13 +138,14 @@ export async function createReferenceHost(): Promise<ReferenceHost> {
   // ONE pool. The artifact module mounts on the handle the host already has
   // from `createDB` — the seam takes any drizzle postgres-js instance, so there
   // is no second connection to the same database.
-  const hub = createDB(parsePostgresUrl(DATABASE_URL));
+  const config = parsePostgresUrl(DATABASE_URL);
+  const hub = createDB(config);
   const db: ArtifactDb = hub.db;
 
   // This host resets and truncates its database on boot — refuse to run
   // against anything that doesn't look like a throwaway database unless
   // explicitly opted in.
-  const { database } = parsePostgresUrl(DATABASE_URL);
+  const { database } = config;
   if (
     !database.startsWith("artifact_") &&
     process.env.ARTIFACT_REFERENCE_ALLOW_RESET !== "1"
@@ -156,8 +165,8 @@ export async function createReferenceHost(): Promise<ReferenceHost> {
   // principal stand-ins as FK targets; on a shared dev database those look
   // present by name but lack Interchange's columns, so detect by shape
   // (`tenant.slug`), drop the stand-ins, and migrate for real.
-  // `runArtifactMigrations` needs no such guard — carrying its own ledger is
-  // precisely why it can be called unconditionally on every boot.
+  // `runArtifactMigrations` needs no such guard: every statement is idempotent,
+  // so it runs unconditionally on every boot.
   const [hostSchema] = await db.execute<{ present: boolean }>(sql`
     SELECT EXISTS (
       SELECT 1 FROM information_schema.columns
@@ -168,9 +177,9 @@ export async function createReferenceHost(): Promise<ReferenceHost> {
     await db.execute(sql`DROP TABLE IF EXISTS "public"."principal" CASCADE`);
     await db.execute(sql`DROP TABLE IF EXISTS "public"."tenant" CASCADE`);
     await db.execute(sql`DROP SCHEMA IF EXISTS "artifacts" CASCADE`);
-    await runMigrations(parsePostgresUrl(DATABASE_URL), { schema: "public" });
+    await runMigrations(config, { schema: "public" });
   }
-  await runArtifactMigrations(db);
+  await runArtifactMigrations(config, { schema: "public" });
   await db.execute(
     sql`TRUNCATE TABLE "artifacts"."artifact", "artifacts"."artifact_version", "artifacts"."upload", "artifacts"."mail_attachment_ref" CASCADE`,
   );
@@ -362,6 +371,7 @@ export async function createReferenceHost(): Promise<ReferenceHost> {
 
   return {
     db,
+    config,
     tenantId: tenant.id,
     agentPrincipal,
     scope: () => ({

examples/reference-host/test/acceptance.test.ts

@@ -679,15 +679,8 @@ describe("a skill-draft is invisible over the mounted host", () => {
 });
 
 describe("the migration runner is re-runnable", () => {
-  test("re-running applies nothing new and destroys no data", async () => {
-    const before = await host.db.execute<{ id: string }>(
-      sql`SELECT "id" FROM "artifacts"."migrations"`,
-    );
-    await runArtifactMigrations(host.db);
-    const after = await host.db.execute<{ id: string }>(
-      sql`SELECT "id" FROM "artifacts"."migrations"`,
-    );
-    expect(after.length).toBe(before.length);
+  test("re-running destroys no data", async () => {
+    await runArtifactMigrations(host.config, { schema: "public" });
 
     const survived = await json<{ artifacts: unknown[] }>(
       await host.request("/api/artifacts?limit=100"),

migrations/0001_artifacts.sql

@@ -0,0 +1,111 @@
+CREATE SCHEMA IF NOT EXISTS "artifacts";
+--> statement-breakpoint
+CREATE TABLE IF NOT EXISTS "artifacts"."artifact" (
+  "id" text PRIMARY KEY DEFAULT gen_random_uuid()::text,
+  "tenant_id" text NOT NULL REFERENCES "public"."tenant"("id") ON DELETE CASCADE,
+  "principal_id" text REFERENCES "public"."principal"("id") ON DELETE SET NULL,
+  "owner_principal_id" text REFERENCES "public"."principal"("id") ON DELETE SET NULL,
+  "kind" text NOT NULL,
+  "title" text NOT NULL,
+  "content" text NOT NULL,
+  "source" jsonb,
+  "version" integer NOT NULL DEFAULT 1,
+  "metadata" jsonb,
+  "content_sha256" text,
+  "archived_at" timestamptz,
+  "created_at" timestamptz NOT NULL DEFAULT now(),
+  "updated_at" timestamptz NOT NULL DEFAULT now(),
+  CONSTRAINT "artifact_version_gte_1" CHECK ("version" >= 1)
+);
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "artifact_tenant_updated_id_idx"
+  ON "artifacts"."artifact" ("tenant_id", "updated_at", "id");
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "artifact_principal_idx"
+  ON "artifacts"."artifact" ("principal_id");
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "artifact_owner_principal_idx"
+  ON "artifacts"."artifact" ("owner_principal_id");
+--> statement-breakpoint
+CREATE TABLE IF NOT EXISTS "artifacts"."artifact_version" (
+  "id" text PRIMARY KEY DEFAULT gen_random_uuid()::text,
+  "artifact_id" text NOT NULL REFERENCES "artifacts"."artifact"("id") ON DELETE CASCADE,
+  "version" integer NOT NULL,
+  "title" text NOT NULL,
+  "content" text NOT NULL,
+  "author_id" text NOT NULL,
+  "metadata" jsonb,
+  "parent_version_ids" text[],
+  "content_sha256" text,
+  "created_at" timestamptz NOT NULL DEFAULT now(),
+  CONSTRAINT "artifact_version_artifact_id_version" UNIQUE ("artifact_id", "version"),
+  CONSTRAINT "artifact_version_version_gte_1" CHECK ("version" >= 1)
+);
+--> statement-breakpoint
+CREATE TABLE IF NOT EXISTS "artifacts"."upload" (
+  "id" text PRIMARY KEY DEFAULT gen_random_uuid()::text,
+  "tenant_id" text NOT NULL REFERENCES "public"."tenant"("id") ON DELETE CASCADE,
+  "principal_id" text REFERENCES "public"."principal"("id") ON DELETE SET NULL,
+  "filename" text NOT NULL,
+  "mime_type" text NOT NULL,
+  "content" bytea NOT NULL,
+  "size" integer NOT NULL,
+  "created_at" timestamptz NOT NULL DEFAULT now(),
+  CONSTRAINT "upload_size_gte_0" CHECK ("size" >= 0)
+);
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "upload_tenant_idx" ON "artifacts"."upload" ("tenant_id");
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "upload_principal_idx" ON "artifacts"."upload" ("principal_id");
+--> statement-breakpoint
+CREATE TABLE IF NOT EXISTS "artifacts"."mail_attachment_ref" (
+  "id" text PRIMARY KEY DEFAULT gen_random_uuid()::text,
+  "tenant_id" text NOT NULL REFERENCES "public"."tenant"("id") ON DELETE CASCADE,
+  "principal_id" text REFERENCES "public"."principal"("id") ON DELETE SET NULL,
+  "instance_id" text NOT NULL,
+  "mail_id" text NOT NULL,
+  "artifact_id" text NOT NULL,
+  "name" text NOT NULL,
+  "mime_type" text NOT NULL,
+  "size" integer NOT NULL,
+  "created_at" timestamptz NOT NULL DEFAULT now(),
+  CONSTRAINT "mail_attachment_ref_mail_id_artifact_id" UNIQUE ("mail_id", "artifact_id"),
+  CONSTRAINT "mail_attachment_ref_size_gte_0" CHECK ("size" >= 0)
+);
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "mail_attachment_ref_instance_idx"
+  ON "artifacts"."mail_attachment_ref" ("instance_id");
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "mail_attachment_ref_tenant_idx"
+  ON "artifacts"."mail_attachment_ref" ("tenant_id");
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "mail_attachment_ref_principal_idx"
+  ON "artifacts"."mail_attachment_ref" ("principal_id");
+--> statement-breakpoint
+-- A database from before 0.1.0 that never booted on 0.1.0 still lacks the
+-- columns 0.1.0 added and keeps zoneless timestamps; bring it to 0.1.0's shape.
+ALTER TABLE "artifacts"."artifact"
+  ADD COLUMN IF NOT EXISTS "metadata" jsonb,
+  ADD COLUMN IF NOT EXISTS "content_sha256" text;
+--> statement-breakpoint
+ALTER TABLE "artifacts"."artifact_version"
+  ADD COLUMN IF NOT EXISTS "metadata" jsonb,
+  ADD COLUMN IF NOT EXISTS "parent_version_ids" text[],
+  ADD COLUMN IF NOT EXISTS "content_sha256" text;
+--> statement-breakpoint
+-- 0.1.0 treated existing zoneless values as UTC; only columns still zoneless
+-- are rewritten, so a current database is untouched.
+DO $$
+DECLARE
+  "col" record;
+BEGIN
+  FOR "col" IN
+    SELECT "table_name", "column_name" FROM information_schema.columns
+    WHERE "table_schema" = 'artifacts' AND "data_type" = 'timestamp without time zone'
+  LOOP
+    EXECUTE format(
+      'ALTER TABLE "artifacts".%I ALTER COLUMN %I TYPE timestamptz USING %I AT TIME ZONE ''UTC''',
+      "col"."table_name", "col"."column_name", "col"."column_name"
+    );
+  END LOOP;
+END $$;

migrations/0002_drop_migration_ledger.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS "artifacts"."migrations";