diff --git a/.changeset/change-multi-statement.md b/.changeset/change-multi-statement.md new file mode 100644 index 00000000..341abd80 --- /dev/null +++ b/.changeset/change-multi-statement.md @@ -0,0 +1,6 @@ +--- +'@noormdev/cli': patch +'@noormdev/sdk': patch +--- + +A change file with more than one statement now runs every statement on SQLite, and `GO` batch separators in change files are honored on SQL Server, as they already were for `sql/` files. Before, SQLite ran only the first statement of a change file and reported success. diff --git a/.changeset/change-parse-failures.md b/.changeset/change-parse-failures.md new file mode 100644 index 00000000..0d070d9d --- /dev/null +++ b/.changeset/change-parse-failures.md @@ -0,0 +1,6 @@ +--- +'@noormdev/cli': patch +'@noormdev/sdk': patch +--- + +`noorm change ff` and `change next` now fail at the position of a change folder that cannot be parsed when they would run it (never run, `reverted`, or `stale`), for example a `.txt` manifest that names a file no longer in `sql/`. Before, the folder was skipped, later changes ran without it, and the command exited 0. `change list` shows such a change as `failed` with the parse error, the TUI change screens and home count include it, `changes.pending()` returns it, and `change run ` reports the parse error instead of "Change not found". An applied change keeps its status, and the batch adds a warning instead of failing. A folder with neither `change/` nor `revert/` is not a change and is ignored. diff --git a/.changeset/init-db-layout-agent-files.md b/.changeset/init-db-layout-agent-files.md new file mode 100644 index 00000000..2154b20b --- /dev/null +++ b/.changeset/init-db-layout-agent-files.md @@ -0,0 +1,8 @@ +--- +'@noormdev/cli': minor +'@noormdev/sdk': minor +--- + +`noorm init` now creates `db/sql/` and `db/changes/` instead of `sql/` and `changes/`, and writes `db/AGENTS.md` and `db/CLAUDE.md`. `AGENTS.md` points coding agents at the `/noorm` skill and points data modeling at ignatius and its `/ignatius-modeling` skill. It also leaves a marker for your own steering. `CLAUDE.md` imports `AGENTS.md`. Neither file is overwritten on re-init. Overwriting a settings file (`noorm init --force`, `noorm settings init --force`) keeps the `paths` already in it. The `noorm` skill now states what `sql/` and `changes/` are for, which edits belong in each, how to reference `sql/` files from a change with a `.txt` manifest, and when to use a `.sql.tmpl`. + +The built-in default for `paths.sql` and `paths.changes` is now `./db/sql` and `./db/changes`, matching what `noorm init` creates. `noorm init` writes `paths` into `settings.yml`, so an initialized project keeps its layout. A project laid out at `./sql` and `./changes` whose `settings.yml` is missing, empty, or has no `paths` key now resolves to `./db/sql` and `./db/changes`. Add `paths: { sql: ./sql, changes: ./changes }` to `settings.yml`, or set `NOORM_PATHS_SQL` and `NOORM_PATHS_CHANGES`. diff --git a/.changeset/prefetch-run-gate.md b/.changeset/prefetch-run-gate.md deleted file mode 100644 index 95c2c523..00000000 --- a/.changeset/prefetch-run-gate.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -'@noormdev/cli': patch -'@noormdev/sdk': patch ---- - -Decide which files to skip with one or two tracking queries per batch instead of one or two per file. Skipped files no longer write execution rows, and a build where nothing changed writes no operation record. The SDK's `run.file(path, { preview: true })` now renders without executing. The TUI's re-run confirmation now finds earlier runs. diff --git a/.changeset/update-skill-prompt.md b/.changeset/update-skill-prompt.md new file mode 100644 index 00000000..a54276de --- /dev/null +++ b/.changeset/update-skill-prompt.md @@ -0,0 +1,8 @@ +--- +'@noormdev/cli': minor +'@noormdev/sdk': minor +--- + +After a successful update, `noorm update` offers the noorm agent skill. It runs `npx skills add noormdev/noorm/skills` when `npx` is on `PATH`, else `gh skill install noormdev/noorm skills/noorm` (GitHub CLI 2.90 or newer). With neither, or when the command fails, it prints both commands and where to get Node.js and the GitHub CLI. On a terminal it asks first; piped output prints the command instead, and `--json` skips the offer. A declined or failed skill install does not change the exit code. + +On a terminal, `noorm update` now rewrites its status line in place: `Checking for updates...` is replaced by the result, and the download counter ends on `Downloaded X MB (100%)`. `noorm update --json` now writes only the JSON document to stdout; before, status lines preceded it. diff --git a/.claude/rules/documentation.md b/.claude/rules/documentation.md index 8e57c69c..a0de70db 100644 --- a/.claude/rules/documentation.md +++ b/.claude/rules/documentation.md @@ -19,8 +19,6 @@ Every claim has to be one you can point at. Do not invent a position to argue against. No "some people say", no "you may have been told", no opponent who does not hold the view. State the finding and let it stand without a foil. -`docs/guide/changes/overview.md:15` currently breaks this ("Some argue that you *are* moving data..."). Fix it when you next touch that file, and do not copy the shape. - Measurements carry their conditions: the tool, the scale, the run. Never combine numbers from different runs or builds into one comparison. If the conditions differ, re-measure or say which figure came from where. Mark inference as inference. An extrapolation from a mechanism reads as a measured result unless the text says otherwise. @@ -28,6 +26,6 @@ Mark inference as inference. An extrapolation from a mechanism reads as a measur ## Structure -Lead with the problem the reader wants to solve, then the mechanism. Code follows the explanation that motivates it. +Lead with the problem the reader wants to solve, in one sentence. The example comes next, the file or command with its real output, and the explanation follows it. A reader should see the thing working before the first `##` heading. Prefer a table, tree, or diagram wherever the content has a shape. Reserve prose for reasoning and motivation, which is the one thing a table cannot carry. diff --git a/CLAUDE.md b/CLAUDE.md index 2b67db0e..a113cb86 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -125,18 +125,20 @@ For setup wizards where the target database may not exist yet, use `testConnecti | Path | Covers | Voice | |------|--------|-------| | `README.md` | project overview, install, quick start | atomic-writing | -| `docs/index.md` | docs landing, why noorm, quick start | atomic-writing | +| `docs/index.md` | landing: problems noorm and ignatius remove, how it works, FAQ | atomic-writing | +| `docs/download.md` | install noorm and ignatius, agent skills, SDK, install options | atomic-writing | | `docs/why-noorm.md` | rationale, prior tools, history | atomic-writing | | `docs/tui.md` | TUI screens, navigation, keyboard shortcuts | terse-technical | | `docs/headless.md` | CLI reference, global flags, command discovery | terse-technical | -| `docs/getting-started/installation.md` | requirements, CLI install, SDK install | atomic-writing | -| `docs/getting-started/concepts.md` | SQL files as source of truth, execution order, changes | atomic-writing | +| `docs/getting-started/installation.md` | CLI install, SDK install, agent skill, update, download verification | atomic-writing | +| `docs/getting-started/concepts.md` | db/sql vs db/changes, execution order, templates, configs and roles, secrets, what lives where | atomic-writing | | `docs/getting-started/first-build.md` | init, first build walkthrough | atomic-writing | -| `docs/getting-started/building-your-sdk.md` | monorepo setup, database package, SDK wiring | atomic-writing | +| `docs/getting-started/building-your-sdk.md` | database package layout, schema types, domain classes, client, tests | atomic-writing | | `docs/guide/automation/ci.md` | test CI, prod CI shapes | atomic-writing | | `docs/guide/automation/mcp.md` | MCP server, AI agent integration, tools | atomic-writing | | `docs/guide/automation/non-interactive.md` | --yes semantics, CI bootstrap | atomic-writing | -| `docs/guide/changes/overview.md` | changes vs migrations, directory structure | atomic-writing | +| `docs/guide/changes/overview.md` | change folder anatomy, file types, .txt manifests, ordering | atomic-writing | +| `docs/guide/changes/add-a-column.md` | adding a column: db/sql/ declaration, change with backfill, revert | atomic-writing | | `docs/guide/changes/forward-revert.md` | apply and revert lifecycle | atomic-writing | | `docs/guide/changes/history.md` | execution history, TUI history views | atomic-writing | | `docs/guide/database/create.md` | db create, configs-first workflow | atomic-writing | @@ -189,10 +191,13 @@ For setup wizards where the target database may not exist yet, use `testConnecti | `docs/dev/version.md` | version layers, migration | atomic-writing | | `docs/dev/ink-cheatsheet.md` | Ink API cheatsheet | terse-technical | | `docs/dev/ink-testing-library-cheatsheet.md` | ink-testing-library cheatsheet | terse-technical | -| `docs/modeling/index.md` | ignatius overview, IDEF1X modeling | atomic-writing | -| `docs/modeling/installation.md` | ignatius install | atomic-writing | +| `docs/modeling/index.md` | ignatius landing: data flows, live model, two-minute try-it, links into the section | atomic-writing | +| `docs/modeling/why.md` | case for modeling first, why a separate tool, the two notations, the name | atomic-writing | +| `docs/parts/` | fragments shared by the two landings through ``; not pages | atomic-writing | +| `docs/modeling/installation.md` | ignatius commands: install, serve, validate, export, index, approve, update | atomic-writing | | `docs/modeling/entities.md` | entity format, key inheritance | atomic-writing | | `docs/modeling/data-flows.md` | SSADM data flow diagrams | atomic-writing | +| `docs/modeling/model-to-sql.md` | entity file fields mapped to DDL, constraint naming | atomic-writing | | `docs/modeling/best-practices.md` | modeling best practices | atomic-writing | | `docs/modeling/branding.md` | model branding | atomic-writing | | `docs/modeling/modeling-skill.md` | /ignatius-modeling skill | atomic-writing | diff --git a/README.md b/README.md index bcdddafe..72c0640d 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ The install script is [fully transparent and open source](https://github.com/noo ## Quick start ```bash -# Bootstrap a project — creates your identity and the sql/ and changes/ layout +# Bootstrap a project — creates your identity and the db/sql/ and db/changes/ layout noorm init # Add a database config diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index f5ec6b7f..279a0268 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -1,5 +1,6 @@ import { defineConfig } from 'vitepress'; import { withMermaid } from 'vitepress-plugin-mermaid'; +import { mermaid } from './mermaid'; // https://vitepress.dev/reference/site-config const SITE_URL = 'https://noorm.dev'; @@ -11,6 +12,7 @@ export default withMermaid( title: 'noorm', description: DESCRIPTION, base: process.env.VITEPRESS_BASE || '/', + mermaid, // `docs/wiki/` is generated repo-analysis output for tooling and // contributors, not published documentation. VitePress compiles every @@ -24,8 +26,9 @@ export default withMermaid( // `models/**` joins the list for the same reason as `tapes/**`: it is // source for a generated artifact, not pages. Left in, VitePress // compiles every entity file into a route like - // /models/todo-list/data/core/users.html. - srcExclude: ['wiki/**', 'spec/**', 'design/**', 'superpowers/**', 'tmp/**', 'tapes/**', 'models/**'], + // /models/todo-list/data/core/users.html. `parts/**` holds fragments + // that pages pull in with ``; they are not pages either. + srcExclude: ['wiki/**', 'spec/**', 'design/**', 'superpowers/**', 'tmp/**', 'tapes/**', 'models/**', 'parts/**'], markdown: { // The terminal recordings are the heaviest assets on the site @@ -41,7 +44,7 @@ export default withMermaid( // The `title`/`description` config fields cover and <meta name>; // everything social has to be spelled out here. head: [ - ['link', { rel: 'icon', href: '/image/logo.svg', type: 'image/svg+xml' }], + ['link', { rel: 'icon', href: '/image/favicon.svg', type: 'image/svg+xml' }], ['link', { rel: 'apple-touch-icon', href: '/image/logo.png' }], ['link', { rel: 'canonical', href: `${SITE_URL}/` }], ['meta', { name: 'theme-color', content: '#E05742' }], @@ -83,17 +86,15 @@ gtag('config', 'G-Y69K95866J');`], }, themeConfig: { // https://vitepress.dev/reference/default-theme-config - logo: '/image/logo.svg', + logo: { light: '/image/logo.svg', dark: '/image/logo-dark.svg' }, siteTitle: 'noorm', - // No Home entry: the logo already links to `/`. Dev docs are not a - // top-bar concern either — they live in the Reference group of the - // main sidebar as "Contributor Documentation". + // No Home entry: the logo already links to `/`. Contributor docs are + // not a reader's concern, so they sit in the footer, not the nav. nav: [ - { text: 'Getting Started', link: '/getting-started/installation' }, - { text: 'Guide', link: '/guide/sql-files/organization' }, - { text: 'CLI', link: '/headless' }, - { text: 'TUI', link: '/tui' }, + { text: 'Docs', link: '/getting-started/installation' }, { text: 'Modeling', link: '/modeling/' }, + { text: 'Reference', link: '/headless' }, + { text: 'Download', link: '/download' }, ], sidebar: { @@ -149,7 +150,6 @@ gtag('config', 'G-Y69K95866J');`], { text: 'Getting Started', items: [ - { text: 'Why noorm', link: '/why-noorm' }, { text: 'Installation', link: '/getting-started/installation' }, { text: 'First Build', link: '/getting-started/first-build' }, { text: 'Building Your SDK', link: '/getting-started/building-your-sdk' }, @@ -160,15 +160,6 @@ gtag('config', 'G-Y69K95866J');`], // is what pins them open with no toggle. Every other group // is collapsible and starts closed, so the sidebar opens as // a short list of topics rather than a wall of links. - { - text: 'Features', - collapsed: true, - items: [ - { text: 'CLI Reference', link: '/headless' }, - { text: 'Terminal UI', link: '/tui' }, - { text: 'Relational Design', link: '/guide/relational-design' }, - ], - }, { // ignatius is a separate binary with its own release // cadence, so these pages carry the narrative and link @@ -178,13 +169,15 @@ gtag('config', 'G-Y69K95866J');`], collapsed: true, items: [ { text: 'Overview', link: '/modeling/' }, - { text: 'Installation', link: '/modeling/installation' }, + { text: 'Commands', link: '/modeling/installation' }, { text: 'Entities & Key Inheritance', link: '/modeling/entities' }, { text: 'Data Flows', link: '/modeling/data-flows' }, + { text: 'From Model to SQL', link: '/modeling/model-to-sql' }, { text: 'Modeling Skill', link: '/modeling/modeling-skill' }, { text: 'Reverse-Engineering', link: '/modeling/reverse-engineering' }, { text: 'Branding', link: '/modeling/branding' }, { text: 'Best Practices', link: '/modeling/best-practices' }, + { text: 'Why ignatius', link: '/modeling/why' }, ], }, { @@ -215,6 +208,7 @@ gtag('config', 'G-Y69K95866J');`], collapsed: true, items: [ { text: 'Overview', link: '/guide/changes/overview' }, + { text: 'Add a Column', link: '/guide/changes/add-a-column' }, { text: 'Forward & Revert', link: '/guide/changes/forward-revert' }, { text: 'History', link: '/guide/changes/history' }, ], @@ -241,8 +235,17 @@ gtag('config', 'G-Y69K95866J');`], { text: 'Reference', items: [ + { text: 'CLI', link: '/headless' }, { text: 'SDK', link: '/reference/sdk' }, - { text: 'Contributor Documentation', link: '/dev/' }, + { text: 'Terminal UI', link: '/tui' }, + ], + }, + { + text: 'Background', + collapsed: true, + items: [ + { text: 'Why noorm', link: '/why-noorm' }, + { text: 'Relational Design', link: '/guide/relational-design' }, ], }, ], @@ -268,15 +271,16 @@ gtag('config', 'G-Y69K95866J');`], }, }, + // Depth 4 because the CLI reference puts each command at ####. outline: { - level: [2, 3], + level: [2, 4], }, footer: { message: 'Database Schema & Change Manager', // Apache-2.0 to match LICENSE and every package.json in the repo. This // read "ISC" until 2026-08 and matched nothing that ships. - copyright: `© ${new Date().getFullYear()} <a href="https://github.com/noormdev">noorm</a> · <a href="https://github.com/noormdev/noorm">GitHub</a> · <a href="https://www.npmjs.com/org/noormdev">npm</a> · <a href="https://github.com/noormdev/noorm/blob/master/LICENSE">Apache 2.0</a>`, + copyright: `© ${new Date().getFullYear()} <a href="https://github.com/noormdev">noorm</a> · <a href="https://github.com/noormdev/noorm">GitHub</a> · <a href="https://www.npmjs.com/org/noormdev">npm</a> · <a href="https://github.com/noormdev/noorm/blob/master/LICENSE">Apache 2.0</a> · <a href="/dev/">Contributor docs</a>`, }, }, }), diff --git a/docs/.vitepress/mermaid.ts b/docs/.vitepress/mermaid.ts new file mode 100644 index 00000000..c1af2956 --- /dev/null +++ b/docs/.vitepress/mermaid.ts @@ -0,0 +1,96 @@ +/** + * Mermaid look for the docs: solid brand-colored nodes with ink borders. + * + * Node fills and their ink are fixed in both themes. Connectors, edge labels, and the + * canvas read --mm-* custom properties set on .vp-doc .mermaid in brand.css, so a theme + * toggle needs no re-render. Anything that changes label size (font, size) lives here, + * because Mermaid measures labels with this config. HTML label weight is not set: Mermaid + * 11 measures it at the default weight, so a heavier one clips. + */ + +const INK = '#161A20'; +const CREAM = '#F2ECE0'; +const FILLS = ['#E05742', '#D2A47A', '#8DB26A', '#6FA0CC', CREAM]; + +const SHAPE = ':is(rect, polygon, circle, ellipse, path):not(:is(.row-rect-odd, .row-rect-even, .divider) *)'; +const NODE = `.node ${SHAPE}`; + +const cycle = FILLS.map( + (fill, i) => `.node:nth-child(${FILLS.length}n+${i + 1}) ${SHAPE} { fill: ${fill}; }`, +).join('\n'); + +const themeCSS = ` + ${NODE}, rect.actor, rect.note, .cluster rect, .labelBox { + stroke: ${INK}; + stroke-width: 2.5px; + rx: 8px; + ry: 8px; + filter: drop-shadow(3px 5px 7px rgba(0, 0, 0, 0.35)); + } + ${cycle} + .nodeLabel, .nodeLabel p, .label, .label p, .node .label, .node .label p { color: ${INK}; } + + .cluster rect { + fill: transparent; + filter: none; + stroke: var(--mm-dim); + stroke-width: 1.5px; + stroke-dasharray: 5 4; + } + .cluster-label .nodeLabel, .cluster-label p { color: var(--mm-line); } + + .flowchart-link, .edgePath path, .transition, .messageLine0, .messageLine1 { + stroke: var(--mm-line); + stroke-width: 2.5px; + fill: none; + } + /* ER markers scale with the line width (markerUnits=strokeWidth), so keep it thin. */ + .relationshipLine { stroke: var(--mm-line); stroke-width: 1.5px; fill: none; } + .arrowMarkerPath { fill: var(--mm-line); stroke: var(--mm-line); } + marker:not([id*="Arrow"]):not([id$="-pointEnd"]):not([id$="-pointStart"]) path { + fill: none; + stroke: var(--mm-line); + stroke-width: 1px; + } + marker circle { fill: var(--mm-canvas); stroke: var(--mm-line); stroke-width: 1px; } + marker:is([id$="-pointEnd"], [id$="-pointStart"]) { overflow: visible; } + marker[id$="-pointEnd"] .arrowMarkerPath { transform-origin: 10px 5px; transform: translateX(-1px) scale(1.2); } + marker[id$="-pointStart"] .arrowMarkerPath { transform-origin: 0 5px; transform: translateX(1px) scale(1.2); } + + .edgeLabel, .edgeLabel p, .labelBkg, .edgeLabel span { + background-color: var(--mm-canvas); + color: var(--mm-text); + } + .edgeLabel rect, .relationshipLabelBox { fill: var(--mm-canvas); opacity: 1; } + + rect.actor { fill: ${FILLS[1]}; } + text.actor > tspan, .actor tspan { fill: ${INK}; stroke: none; font-weight: 600; } + .actor-line { stroke: var(--mm-dim); stroke-width: 1.5px; stroke-dasharray: 5 4; } + .messageText { fill: var(--mm-text); stroke: none; } + rect.note { fill: ${CREAM}; } + .noteText, .noteText > tspan { fill: ${INK}; stroke: none; } + .loopText, .loopText > tspan, .labelText, .labelText > tspan { fill: var(--mm-text); stroke: none; } + .loopLine { stroke: var(--mm-dim); stroke-width: 1.5px; stroke-dasharray: 5 4; } + .activation0, .activation1, .activation2 { fill: ${CREAM}; stroke: ${INK}; } + + .state-start, .state-end circle:last-child { fill: var(--mm-text); stroke: var(--mm-text); filter: none; } + .end-state-inner { fill: var(--mm-canvas); } + + .row-rect-odd path { fill: ${CREAM}; } + .row-rect-even path { fill: #E7DFCF; } + .row-rect-odd path, .row-rect-even path, .divider path { stroke: ${INK}; stroke-width: 1.5px; } +`; + +export const mermaid = { + theme: 'base', + fontFamily: 'Geist, ui-sans-serif, system-ui, sans-serif', + flowchart: { padding: 14, nodeSpacing: 44, rankSpacing: 52, diagramPadding: 8 }, + sequence: { actorMargin: 56, messageMargin: 36, diagramMarginX: 8, diagramMarginY: 8 }, + themeVariables: { + fontFamily: 'Geist, ui-sans-serif, system-ui, sans-serif', + fontSize: '14px', + textColor: '#808080', + lineColor: '#808080', + }, + themeCSS, +}; diff --git a/docs/.vitepress/og-source.html b/docs/.vitepress/og-source.html index 43b370e7..10f43c54 100644 --- a/docs/.vitepress/og-source.html +++ b/docs/.vitepress/og-source.html @@ -116,7 +116,7 @@ <div> <div class="mark"> - <img src="../public/image/logo.svg" alt=""> + <img src="../public/image/logo-dark.svg" alt=""> <span class="wordmark">noorm</span> </div> <h1>Write SQL. Skip the ORM.</h1> diff --git a/docs/.vitepress/theme/HeroEyebrow.vue b/docs/.vitepress/theme/HeroEyebrow.vue deleted file mode 100644 index ac637718..00000000 --- a/docs/.vitepress/theme/HeroEyebrow.vue +++ /dev/null @@ -1,23 +0,0 @@ -<template> - <!-- Apache-2.0 to match LICENSE and every package.json. This said ISC until - 2026-08 and was the most visible license claim on the site. --> - <div class="hero-eyebrow">V1.0 · APACHE 2.0 · POSTGRES · MYSQL · SQLITE · MSSQL</div> -</template> - -<style scoped> -.hero-eyebrow { - font-family: var(--vp-font-family-mono); - font-size: 0.75rem; - font-weight: 500; - letter-spacing: 0.12em; - text-transform: uppercase; - /* Bronze, not ember: the brand keeps ember for signal (primary CTA, cursor, - errors) and gives editorial lines like this one the bronze register. */ - color: #6E4A22; - margin-bottom: 20px; -} - -.dark .hero-eyebrow { - color: #D2A47A; -} -</style> diff --git a/docs/.vitepress/theme/HeroStats.vue b/docs/.vitepress/theme/HeroStats.vue deleted file mode 100644 index ee0072c6..00000000 --- a/docs/.vitepress/theme/HeroStats.vue +++ /dev/null @@ -1,26 +0,0 @@ -<template> - <div class="hero-stats"> - <span>single binary</span> - <span class="sep">·</span> - <span>zero dependencies at runtime</span> - <span class="sep">·</span> - <span>type-safe SDK</span> - </div> -</template> - -<style scoped> -.hero-stats { - font-family: var(--vp-font-family-mono); - font-size: 0.8125rem; - color: var(--vp-c-text-3); - margin-top: 32px; - display: flex; - flex-wrap: wrap; - gap: 8px; -} - -.sep { - color: var(--vp-c-text-3); - opacity: 0.5; -} -</style> diff --git a/docs/.vitepress/theme/HeroTerminal.vue b/docs/.vitepress/theme/HeroTerminal.vue deleted file mode 100644 index 9df0ee53..00000000 --- a/docs/.vitepress/theme/HeroTerminal.vue +++ /dev/null @@ -1,156 +0,0 @@ -<template> - <div class="hero-terminal"> - <div class="terminal-header"> - <span class="terminal-label">bash · install.sh</span> - <button class="terminal-copy" @click="copy"> - <svg viewBox="0 0 24 24" width="14" height="14" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="13" height="13" rx="2" ry="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg> - {{ copied ? 'copied' : 'copy' }} - </button> - </div> - <div class="terminal-body"> - <div class="line comment"># Install (no sudo)</div> - <div class="line"><span class="prompt">$</span> curl -fsSL <span class="url">https://noorm.dev/install.sh</span> | sh</div> - <div class="line spacer"></div> - <div class="line"><span class="prompt">$</span> cd /my/project && noorm init</div> - <div class="line"><span class="prompt">$</span> npm i <span class="arg">@noormdev/sdk</span></div> - <div class="line spacer"></div> - <div class="line"><span class="prompt">$</span> noorm run build<span class="cursor"></span></div> - </div> - </div> -</template> - -<script setup> -import { ref } from 'vue' - -const copied = ref(false) - -function copy() { - const text = `curl -fsSL https://noorm.dev/install.sh | sh\ncd /my/project && noorm init\nnpm i @noormdev/sdk\nnoorm run build` - navigator.clipboard.writeText(text) - copied.value = true - setTimeout(() => { copied.value = false }, 2000) -} -</script> - -<style scoped> -.hero-terminal { - width: 100%; - max-width: 520px; - margin: 0 auto; - /* The hero container centers its text below 960px; a terminal has to stay flush left. */ - text-align: left; - border-radius: 10px; - border: 1px solid var(--vp-c-border); - background: var(--vp-code-block-bg, var(--vp-c-bg-alt)); - font-family: var(--vp-font-family-mono); - font-size: 0.8125rem; - line-height: 1.7; - overflow: hidden; - box-shadow: - 0 0 1rem rgba(0, 0, 0, 0.15), - 0 0 3rem rgba(0, 0, 0, 0.1); -} - -.terminal-header { - display: flex; - align-items: center; - justify-content: space-between; - padding: 8px 16px; - border-bottom: 1px solid var(--vp-c-border); -} - -.terminal-label { - font-size: 0.75rem; - color: var(--vp-c-text-3); - letter-spacing: 0.01em; -} - -.terminal-copy { - display: flex; - align-items: center; - gap: 4px; - background: none; - border: none; - font-family: var(--vp-font-family-mono); - font-size: 0.75rem; - color: var(--vp-c-text-3); - cursor: pointer; - padding: 2px 6px; - border-radius: 4px; - transition: color 120ms ease; -} - -.terminal-copy:hover { - color: var(--vp-c-text-1); -} - -.terminal-body { - padding: 16px 20px; -} - -/* Phone widths: shrink the type so the longest command still lands on one line, - and let the body scroll rather than spill past the card border. */ -@media (max-width: 639px) { - .hero-terminal { - font-size: clamp(0.625rem, 2.7vw, 0.8125rem); - } - - .terminal-body { - padding: 14px 16px; - overflow-x: auto; - } -} - -.line { - white-space: pre; - color: var(--vp-c-text-1); -} - -.line.spacer { - height: 0.85em; -} - -.line.comment { - color: var(--vp-c-text-3); -} - -.line.output { - padding-left: 1em; - color: var(--vp-c-text-2); -} - -.prompt { - color: var(--vp-c-text-3); - margin-right: 0.5em; -} - -.url { - color: #6E4A22; -} - -.dark .url { - color: #D2A47A; -} - -.check { - color: #6E8F4E; - margin-right: 0.25em; -} - -.arg { - color: #6E8F4E; -} - -.cursor { - display: inline-block; - width: 0.55em; - height: 1.1em; - background: #E05742; - vertical-align: text-bottom; - animation: blink 1s step-end infinite; -} - -@keyframes blink { - 50% { opacity: 0; } -} -</style> diff --git a/docs/.vitepress/theme/ModelViewer.vue b/docs/.vitepress/theme/ModelViewer.vue new file mode 100644 index 00000000..966b94b2 --- /dev/null +++ b/docs/.vitepress/theme/ModelViewer.vue @@ -0,0 +1,65 @@ +<script setup lang="ts"> +import { computed, ref, watch } from 'vue' +import { useData, withBase } from 'vitepress' + +const props = withDefaults(defineProps<{ + model: string + title: string + hash?: string + tabs?: boolean + label?: string + caption?: string +}>(), { + hash: 'view=graph', + tabs: true, + label: '', + caption: 'rendered by ignatius', +}) + +const views = [ + { hash: 'view=graph', label: 'Graph' }, + { hash: 'view=dict', label: 'Dictionary' }, + { hash: 'view=flow', label: 'Flows' }, +] + +const { isDark } = useData() +const frame = ref<HTMLIFrameElement | null>(null) +const active = ref(props.hash) +const file = computed(() => withBase(`/models/${props.model}${isDark.value ? '-dark' : ''}.html`)) +const src = ref(`${file.value}#${active.value}`) + +watch(file, (next) => { + + src.value = `${next}#${active.value}` + +}) + +// Assigning src would add a browser history entry per tab click; replace() swaps the hash in place. +function show(hash: string) { + + active.value = hash + frame.value?.contentWindow?.location.replace(`${file.value}#${hash}`) + +} +</script> + +<template> + <div class="model-embed model-viewer"> + <div class="model-viewer-bar"> + <span v-if="tabs" class="model-viewer-tabs" role="group" aria-label="Model view"> + <button + v-for="view in views" + :key="view.hash" + type="button" + :aria-pressed="view.hash === active" + :class="{ on: view.hash === active }" + @click="show(view.hash)" + >{{ view.label }}</button> + </span> + <span v-else>{{ label }}</span> + <span>{{ caption }}</span> + </div> + <iframe ref="frame" :src="src" :title="title" loading="lazy"></iframe> + <slot /> + </div> +</template> diff --git a/docs/.vitepress/theme/brand.css b/docs/.vitepress/theme/brand.css index 7ca20190..66337edf 100644 --- a/docs/.vitepress/theme/brand.css +++ b/docs/.vitepress/theme/brand.css @@ -80,6 +80,9 @@ --vp-code-block-bg: #F0EBE0; --vp-code-color: #212931; --vp-code-bg: rgba(69, 74, 89, 0.06); + + --noorm-bronze: #6E4A22; + --noorm-bronze-strong: #916336; } /* ───────────────────────────────────────────── @@ -113,6 +116,9 @@ --vp-code-color: #F2ECE0; --vp-code-bg: rgba(255, 255, 255, 0.06); + --noorm-bronze: #D2A47A; + --noorm-bronze-strong: #D2A47A; + --vp-button-brand-bg: #E05742; --vp-button-brand-hover-bg: #ED7561; --vp-button-brand-active-bg: #F49A8A; @@ -151,7 +157,7 @@ code, kbd, samp, pre { Body links — bronze for docs, ember on hover ───────────────────────────────────────────── */ .vp-doc a { - color: #6E4A22; + color: var(--noorm-bronze); text-decoration-color: rgba(110, 74, 34, 0.3); } @@ -161,7 +167,7 @@ code, kbd, samp, pre { } .dark .vp-doc a { - color: #D2A47A; + color: var(--noorm-bronze); text-decoration-color: rgba(210, 164, 122, 0.3); } @@ -310,116 +316,7 @@ code, kbd, samp, pre { } /* ───────────────────────────────────────────── - Buttons — override scoped VPButton radius - ───────────────────────────────────────────── */ -.VPHero .actions .action .VPButton.medium { - border-radius: 8px; - font-weight: 500; -} - -.VPHero .actions .action .VPButton.brand { - font-weight: 600; -} - -.VPHero .actions .VPButton.alt[href*="github"]::before { - content: ''; - display: inline-block; - width: 16px; - height: 16px; - margin-right: 6px; - vertical-align: -3px; - background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='currentColor' stroke-width='1.5' stroke-linecap='round' stroke-linejoin='round'%3E%3Cpath d='M15 22v-4a4.8 4.8 0 0 0-1-3.5c3 0 6-2 6-5.5.08-1.25-.27-2.48-1-3.5.28-1.15.28-2.35 0-3.5 0 0-1 0-3 1.5-2.64-.5-5.36-.5-8 0C6 2 5 2 5 2c-.3 1.15-.3 2.35 0 3.5A5.403 5.403 0 0 0 4 9c0 3.5 3 5.5 6 5.5-.39.49-.68 1.05-.85 1.65-.17.6-.22 1.23-.15 1.85v4'/%3E%3Cpath d='M9 18c-4.51 2-5-2-7-2'/%3E%3C/svg%3E"); - background-size: contain; - background-repeat: no-repeat; -} - -.dark .VPHero .actions .VPButton.alt[href*="github"]::before { - filter: invert(1) brightness(0.85); -} - -/* ───────────────────────────────────────────── - Hero - ───────────────────────────────────────────── */ -.VPHero .name { - --vp-home-hero-name-color: transparent; - --vp-home-hero-name-background: linear-gradient(135deg, #E05742 0%, #B47A45 100%); -} - -.VPHero .name .clip { - font-family: var(--vp-font-family-mono); - font-weight: 800; - letter-spacing: -0.04em; -} - -.VPHero .text { - font-weight: 700; - letter-spacing: -0.03em; - line-height: 1.15; -} - -.VPHero .tagline { - max-width: 480px; - font-weight: 400; - color: var(--vp-c-text-2); -} - -.VPHero .VPImage { - border-radius: 10px; - border: 1px solid var(--vp-c-border); -} - -/* Below 960px the theme sizes the image slot as a fixed 320/392px square and - bleeds it upward under the nav — sized for a floating logo, not the terminal - card we render there. Size it to content and move it below the pitch. */ -/* Doubled selectors for the same reason as the nav block above — the theme's - scoped `.image[data-v-…]` would otherwise tie or win. */ -@media (max-width: 959px) { - .VPHero .image.image { - order: 3; - margin: 40px 0 0; - /* Flex items default to min-width:auto, so the terminal's `white-space: pre` - lines would set the slot's floor. Zeroing it lets the card shrink and - its own overflow-x take over. */ - min-width: 0; - } - - .VPHero .image-container.image-container { - width: 100%; - max-width: 100%; - height: auto; - min-width: 0; - } -} - -.dark .VPHero .VPImage { - box-shadow: 0 18px 40px -12px rgba(0, 0, 0, 0.35), 0 6px 12px rgba(0, 0, 0, 0.15); -} - -/* ───────────────────────────────────────────── - Feature cards - ───────────────────────────────────────────── */ -.VPFeature { - border-radius: 10px; - transition: border-color 180ms ease-out, box-shadow 180ms ease-out; -} - -.VPFeature:hover { - border-color: var(--vp-c-brand-1); - box-shadow: 0 4px 10px -2px rgba(33, 41, 49, 0.08), 0 2px 4px rgba(33, 41, 49, 0.05); -} - -.VPFeature .title { - font-weight: 600; - letter-spacing: -0.01em; -} - -.VPFeature .details { - line-height: 1.65; - color: var(--vp-c-text-2); -} - -/* ───────────────────────────────────────────── - Next steps — 3-up card row on the home page + Next steps — 3-up card row (why-noorm, modeling overview) ───────────────────────────────────────────── */ .next-steps { display: grid; @@ -613,3 +510,55 @@ html.dark { height: 460px; } } + +/* ───────────────────────────────────────────── + Paired code — a source and what it becomes + ───────────────────────────────────────────── */ + +/* Stacked: side by side, the doc column clips both blocks' longest lines. */ +.pair { + margin: 16px 0; +} + +.vp-doc .pair div[class*='language-'] { + margin: 0; +} + +.vp-doc .pair .pair-joint { + margin: 0 0 0 24px; + padding: 6px 0 6px 20px; + border-left: 2px solid var(--vp-c-divider); + font-family: var(--vp-font-family-mono); + font-size: 0.75rem; + letter-spacing: 0.08em; + text-transform: uppercase; + color: var(--vp-c-text-3); +} + +/* Mermaid diagrams: palette read by docs/.vitepress/mermaid.ts, card on the container. */ +.vp-doc .mermaid { + --mm-canvas: var(--vp-c-bg); + --mm-line: var(--vp-c-text-2); + --mm-dim: var(--vp-c-text-3); + --mm-text: var(--vp-c-text-1); + + display: flex; + justify-content: center; + margin: 16px 0; + padding: 20px 16px; + border: 1px solid var(--vp-c-border); + border-radius: 10px; + background-color: var(--vp-c-bg); + background-image: radial-gradient(var(--vp-c-divider) 1px, transparent 1px); + background-size: 18px 18px; +} + +.vp-doc .mermaid svg { + overflow: visible; +} + +/* Mermaid sizes each label for its own 1.5 line height; .vp-doc p's 28px clips the last line. */ +.vp-doc .mermaid p { + margin: 0; + line-height: 1.5; +} diff --git a/docs/.vitepress/theme/index.ts b/docs/.vitepress/theme/index.ts index 0565c116..ddec2e26 100644 --- a/docs/.vitepress/theme/index.ts +++ b/docs/.vitepress/theme/index.ts @@ -1,5 +1,4 @@ import DefaultTheme from 'vitepress/theme' -import { h } from 'vue' import { library } from '@fortawesome/fontawesome-svg-core' import { FontAwesomeIcon } from '@fortawesome/vue-fontawesome' import { @@ -17,10 +16,10 @@ import { faBook } from '@fortawesome/free-solid-svg-icons' import { faGithub } from '@fortawesome/free-brands-svg-icons' -import HeroTerminal from './HeroTerminal.vue' -import HeroEyebrow from './HeroEyebrow.vue' -import HeroStats from './HeroStats.vue' +import mermaid from 'mermaid' +import ModelViewer from './ModelViewer.vue' import './brand.css' +import './landing.css' library.add( faCodeBranch, @@ -40,14 +39,26 @@ library.add( export default { extends: DefaultTheme, - Layout() { - return h(DefaultTheme.Layout, null, { - 'home-hero-info-before': () => h(HeroEyebrow), - 'home-hero-image': () => h(HeroTerminal), - 'home-hero-actions-after': () => h(HeroStats), - }) - }, enhanceApp({ app }) { app.component('FontAwesomeIcon', FontAwesomeIcon) + app.component('ModelViewer', ModelViewer) + + // Mermaid measures each label once and memoizes the width, so a render before + // Geist loads keeps fallback-font widths for good. Hold every render until it is in. + if (typeof document !== 'undefined') { + // load() resolves at once while the font stylesheet has not registered its + // @font-face rules, so wait for the rules (up to 3s), then for the files. + const fontsReady = (async () => { + const hasGeist = () => [...document.fonts].some((f) => f.family.replace(/"/g, '') === 'Geist') + const deadline = Date.now() + 3000 + while (!hasGeist() && Date.now() < deadline) await new Promise((r) => setTimeout(r, 25)) + await Promise.allSettled(['400', '600'].map((w) => document.fonts.load(`${w} 14px Geist`))) + })() + const render = mermaid.render.bind(mermaid) + mermaid.render = async (...args: Parameters<typeof render>) => { + await fontsReady + return render(...args) + } + } } } diff --git a/docs/.vitepress/theme/landing.css b/docs/.vitepress/theme/landing.css new file mode 100644 index 00000000..2f9598ea --- /dev/null +++ b/docs/.vitepress/theme/landing.css @@ -0,0 +1,645 @@ +/* ───────────────────────────────────────────── + Landing pages — docs/index.md, docs/modeling/index.md, docs/download.md + (.nh-file panels also frame the files on docs/modeling/model-to-sql.md) + ───────────────────────────────────────────── */ +.nh-hero { + padding: 56px 0 24px; +} + +.vp-doc .nh-eyebrow { + margin: 0 0 12px; + font-family: var(--vp-font-family-mono); + font-size: 0.75rem; + font-weight: 500; + letter-spacing: 0.12em; + text-transform: uppercase; + color: var(--noorm-bronze); +} + +.vp-doc .nh-title { + margin: 0 0 24px; + font-size: clamp(2.2rem, 2.4vw + 1rem, 3.5rem); + line-height: 1.08; + font-weight: 700; + letter-spacing: -0.03em; +} + +.vp-doc .nh-title span, +.vp-doc .nh-big span { + color: var(--noorm-bronze-strong); +} + +/* The wordmark gradient from the previous hero. */ +.vp-doc .nh-title span { + background: linear-gradient(135deg, #E05742 0%, #B47A45 100%); + -webkit-background-clip: text; + background-clip: text; + color: transparent; +} + +.nh-row { + display: grid; + grid-template-columns: 7fr 5fr; + gap: 48px; + align-items: center; + margin-bottom: 32px; +} + +.vp-doc .nh-lead { + margin: 0; + max-width: 60ch; + font-size: 1.125rem; + line-height: 1.6; + color: var(--vp-c-text-2); +} + +.nh-actions { + display: flex; + flex-wrap: wrap; + gap: 12px; + align-items: center; +} + +.vp-doc .nh-btn { + display: inline-flex; + align-items: center; + height: 42px; + padding: 0 18px; + border-radius: 8px; + border: 1px solid transparent; + font-size: 15px; + font-weight: 500; + text-decoration: none; + transition: background-color 120ms ease-out, border-color 120ms ease-out; +} + +.vp-doc .nh-btn.brand { + background: var(--vp-button-brand-bg); + color: var(--vp-button-brand-text); + font-weight: 600; +} + +.vp-doc .nh-btn.brand:hover { + background: var(--vp-button-brand-hover-bg); + color: var(--vp-button-brand-hover-text); +} + +.vp-doc .nh-btn.alt { + background: var(--vp-c-bg-elv); + border-color: var(--vp-c-border); + color: var(--vp-c-text-1); +} + +.vp-doc .nh-btn.alt:hover { + border-color: var(--vp-c-text-3); +} + +.vp-doc .nh-caption { + flex-basis: 100%; + margin: 4px 0 0; + font-family: var(--vp-font-family-mono); + font-size: 0.75rem; + color: var(--vp-c-text-3); +} + +/* Two-panel comparisons: the dim panel is the usual way, the other is noorm's or ignatius's. */ +.nh-compare { + display: grid; + grid-template-columns: 5fr 7fr; + gap: 14px; + align-items: stretch; +} + +.nh-panel, +.nh-file { + display: flex; + flex-direction: column; + min-width: 0; + border: 1px solid var(--vp-c-border); + border-radius: 10px; + overflow: hidden; + background: var(--vp-c-bg-elv); +} + +.nh-file { + background: var(--vp-code-block-bg); +} + +/* Not this, but that: red marks the usual way, green marks noorm's. */ +.nh-bad { + border-color: color-mix(in srgb, var(--vp-c-danger-1) 45%, var(--vp-c-border)); +} + +.nh-good { + border-color: color-mix(in srgb, var(--vp-c-success-1) 55%, var(--vp-c-border)); +} + +.nh-bad .nh-panel-head { + color: var(--vp-c-danger-1); + background: color-mix(in srgb, var(--vp-c-danger-1) 10%, transparent); +} + +.nh-good .nh-panel-head { + color: var(--vp-c-success-1); + background: color-mix(in srgb, var(--vp-c-success-1) 10%, transparent); +} + +.vp-doc .nh-bad .nh-panel-foot { + color: var(--vp-c-danger-1); +} + +.nh-mark { + margin-right: 4px; + font-size: 0.875rem; +} + +.nh-panel-head { + display: flex; + justify-content: space-between; + gap: 12px; + padding: 8px 14px; + border-bottom: 1px solid var(--vp-c-divider); + font-family: var(--vp-font-family-mono); + font-size: 0.6875rem; + letter-spacing: 0.04em; + color: var(--vp-c-text-3); +} + +.vp-doc .nh-panel-body { + flex: 1; + margin: 0; + padding: 14px 16px; + font-family: var(--vp-font-family-mono); + font-size: 0.8125rem; + line-height: 1.6; + color: var(--vp-c-text-3); + background: transparent; + overflow-x: auto; +} + +.vp-doc .nh-panel-foot { + margin: 0; + padding: 8px 14px; + border-top: 1px dashed var(--vp-c-divider); + font-size: 0.8125rem; + line-height: 1.5; + color: var(--vp-c-text-2); +} + +.vp-doc .nh-file div[class*='language-'], +.vp-doc .nh-cmds div[class*='language-'], +.vp-doc .nh-split div[class*='language-'] { + margin: 0; + border-radius: 0; +} + +.vp-doc .nh-file div[class*='language-'] { + flex: 1; + border: 0; +} + +.vp-doc .nh-split div[class*='language-'] + div[class*='language-'] { + margin-top: 12px; +} + +.vp-doc .nh-split div[class*='language-'], +.vp-doc .nh-cmds div[class*='language-'] { + border-radius: 10px; +} + +.nh-section { + padding: 64px 0 8px; + border-top: 1px solid var(--vp-c-divider); + margin-top: 56px; +} + +.vp-doc .nh-h2 { + margin: 0 0 24px; + padding: 0; + border-top: 0; + font-size: 1.875rem; + line-height: 1.2; + letter-spacing: -0.02em; +} + +.vp-doc .nh-big { + font-size: clamp(2rem, 2vw + 1rem, 3rem); + line-height: 1.1; +} + +.vp-doc .nh-h3 { + margin: 0 0 10px; + font-size: 1.5rem; + letter-spacing: -0.01em; +} + +.nh-r { + display: grid; + grid-template-columns: 4fr 5fr 3fr; + gap: 28px; + align-items: start; + padding: 18px 0; +} + +.nh-r + .nh-r { + border-top: 1px solid var(--vp-c-divider); +} + +.vp-doc .nh-pain { + margin: 0; + font-weight: 500; + color: var(--vp-c-text-1); +} + +.vp-doc .nh-fix { + margin: 0; + font-size: 0.9375rem; + color: var(--vp-c-text-2); +} + +.vp-doc .nh-proof { + justify-self: start; + max-width: 100%; + overflow-wrap: anywhere; + font-size: 0.75rem; +} + +.nh-split { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 48px; + align-items: start; +} + +.vp-doc .nh-flow { + margin: 20px 0; + padding: 14px 18px; + border: 1px solid var(--vp-c-border); + border-radius: 10px; + background: var(--vp-c-bg-elv); + font-family: var(--vp-font-family-mono); + font-size: 0.8125rem; + line-height: 1.9; + color: var(--vp-c-text-1); + overflow-x: auto; +} + +.nh-dim { + color: var(--vp-c-text-3); +} + +.nh-arrow { + color: var(--noorm-bronze); +} + +.vp-doc .nh-links { + font-size: 0.875rem; +} + +/* Full-width band for the second tool. --vp-offset is VPHomeContent's distance to the + viewport edge; padding by its negation keeps the content on the page column. */ +.nh-section.nh-band { + padding-top: 0; + border-top: 0; +} + +.nh-tool { + margin-left: var(--vp-offset, calc(50% - 50vw)); + margin-right: var(--vp-offset, calc(50% - 50vw)); + padding: 64px calc(-1 * var(--vp-offset, calc(50% - 50vw))) 56px; + border-top: 1px solid var(--vp-c-border); + border-bottom: 1px solid var(--vp-c-border); + background: var(--vp-c-bg-soft); +} + +.nh-dfd { + margin: 8px 0 40px; + padding-top: 28px; + border-top: 1px solid var(--vp-c-divider); +} + +.vp-doc .nh-dfd-lead { + max-width: 76ch; + margin: 0 0 20px; +} + +.nh-dfd-compare { + display: grid; + grid-template-columns: 4fr 8fr; + gap: 14px; + align-items: stretch; +} + +.vp-doc .nh-ascii { + display: flex; + align-items: center; + justify-content: center; + font-size: 0.875rem; + line-height: 1.18; + color: var(--vp-c-text-2); +} + +.vp-doc .model-embed.nh-dfd-frame, +.vp-doc .nh-viewer .model-embed { + display: flex; + flex-direction: column; + margin: 0; +} + +.vp-doc .nh-dfd-frame iframe, +.vp-doc .nh-viewer .model-embed iframe { + flex: 1; + min-height: 470px; +} + +.model-viewer-bar { + display: flex; + justify-content: space-between; + align-items: center; + gap: 12px; + padding: 7px 12px; + border-bottom: 1px solid var(--vp-c-divider); + font-family: var(--vp-font-family-mono); + font-size: 0.6875rem; + letter-spacing: 0.04em; + color: var(--vp-c-text-3); + background: var(--vp-c-bg-elv); +} + +.model-viewer-tabs { + display: inline-flex; + gap: 2px; +} + +.model-viewer-tabs button { + padding: 2px 8px; + border: 1px solid transparent; + border-radius: 4px; + font-family: var(--vp-font-family-mono); + font-size: 0.6875rem; + color: var(--vp-c-text-2); + cursor: pointer; +} + +.model-viewer-tabs button.on { + border-color: var(--vp-c-border); + background: var(--vp-c-bg); + color: var(--vp-c-text-1); +} + +.nh-viewer { + display: grid; + grid-template-columns: 5fr 7fr; + gap: 14px; + align-items: stretch; +} + +.vp-doc .nh-note { + margin: 12px 0 28px; + font-size: 0.875rem; + color: var(--vp-c-text-2); +} + +.nh-tool .nh-rows { + margin-top: 8px; +} + +.nh-cmds { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 14px; + margin-top: 28px; +} + +.nh-cmds > div { + display: flex; + flex-direction: column; +} + +.vp-doc .nh-cmds div[class*='language-'] { + flex: 1; +} + +.nh-band + .nh-section { + border-top: 0; +} + +.nh-start { + text-align: center; +} + +.nh-start .nh-h2 { + max-width: 32ch; + margin-left: auto; + margin-right: auto; +} + +.vp-doc .nh-start div[class*='language-'] { + max-width: 640px; + margin: 0 auto; + text-align: left; +} + +.nh-inline { + justify-content: center; + margin-top: 20px; +} + +.nh-faq { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: 18px; +} + +.nh-faq > div { + padding: 20px; + border: 1px solid var(--vp-c-border); + border-radius: 10px; + background: var(--vp-c-bg-elv); +} + +.vp-doc .nh-faq h3 { + margin: 0 0 8px; + font-size: 1rem; +} + +.vp-doc .nh-faq p { + margin: 0; + font-size: 0.9375rem; + color: var(--vp-c-text-2); +} + +@media (max-width: 959px) { + + .nh-row, + .nh-compare, + .nh-split, + .nh-dfd-compare, + .nh-viewer, + .nh-cmds { + grid-template-columns: 1fr; + } + + .nh-r { + grid-template-columns: 1fr; + gap: 6px; + } + + .nh-faq { + grid-template-columns: 1fr; + } + + .nh-tool { + padding: 28px 20px; + } +} + +/* ───────────────────────────────────────────── + Download page + ───────────────────────────────────────────── */ + +/* Wider than VPDoc's 784px; doubled classes outrank its scoped width rule. */ +.page-download .VPDoc.VPDoc .content.content { + max-width: 1104px; +} + +.dl-grid { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 22px; + margin: 28px 0 8px; +} + +.dl-tool { + min-width: 0; + padding: 4px 26px 22px; + border: 1px solid var(--vp-c-border); + border-radius: 14px; + background: var(--vp-c-bg-elv); +} + +.vp-doc .dl-tool h2 { + margin-top: 20px; + padding-top: 0; + border-top: 0; +} + +.vp-doc .dl-what { + margin: 8px 0 16px; + color: var(--vp-c-text-2); +} + +.vp-doc .dl-label { + margin: 20px 0 6px; + font-family: var(--vp-font-family-mono); + font-size: 0.6875rem; + letter-spacing: 0.1em; + text-transform: uppercase; + color: var(--vp-c-text-3); +} + +.vp-doc .dl-label + div[class*='language-'] { + margin-top: 0; +} + +.vp-doc .dl-small { + font-size: 0.8125rem; + color: var(--vp-c-text-2); +} + +@media (max-width: 959px) { + + .dl-grid { + grid-template-columns: 1fr; + } +} + +/* ───────────────────────────────────────────── + Modeling landing — docs/modeling/index.md + ───────────────────────────────────────────── */ +.nh-steps { + display: grid; + gap: 28px; +} + +.nh-steps > div { + display: grid; + grid-template-columns: 4fr 8fr; + gap: 28px; + align-items: start; +} + +.vp-doc .nh-step { + margin: 0 0 8px; + font-family: var(--vp-font-family-mono); + font-size: 0.75rem; + letter-spacing: 0.08em; + text-transform: uppercase; + color: var(--noorm-bronze); +} + +.vp-doc .nh-steps div[class*='language-'] { + margin-top: 0; +} + +.vp-doc .nh-step-note { + margin: 10px 0 0; + font-size: 0.875rem; + color: var(--vp-c-text-2); +} + +.nh-cards { + display: grid; + grid-template-columns: repeat(4, minmax(0, 1fr)); + gap: 14px; +} + +.vp-doc .nh-cards a { + display: flex; + flex-direction: column; + gap: 6px; + padding: 18px; + border: 1px solid var(--vp-c-border); + border-radius: 10px; + background: var(--vp-c-bg-elv); + text-decoration: none; + transition: border-color 120ms ease-out; +} + +.vp-doc .nh-cards a:hover { + border-color: var(--vp-c-text-3); +} + +.vp-doc .nh-cards strong { + color: var(--vp-c-text-1); +} + +.vp-doc .nh-cards span { + font-size: 0.875rem; + line-height: 1.5; + color: var(--vp-c-text-2); +} + +@media (max-width: 959px) { + + .nh-cards { + grid-template-columns: 1fr; + } + + .nh-steps > div { + grid-template-columns: 1fr; + gap: 12px; + } +} + +.vp-doc .nh-steps .nh-file + .nh-file { + margin-top: 10px; +} + +.vp-doc .nh-chapter { + max-width: 76ch; + margin: 0 0 24px; + font-size: 1.0625rem; + line-height: 1.65; + color: var(--vp-c-text-2); +} diff --git a/docs/cli/flags.md b/docs/cli/flags.md index d5745ea3..dd2bfd28 100644 --- a/docs/cli/flags.md +++ b/docs/cli/flags.md @@ -1,100 +1,69 @@ # CLI flag conventions -`noorm`'s CLI is built on [citty](https://github.com/unjs/citty), which parses flags -relative to the subcommand they belong to — a flag typed before the subcommand name never -reaches the leaf command's own parser. `noorm` works around that for the one flag that -genuinely needs to run before dispatch, by stripping it out of `argv` before citty ever sees -it. This page covers that flag and the rule for everything else. +Put every flag after the subcommand. The one exception is `-c <path>`, which goes before it. +```bash +noorm -c packages/db run build --dry-run +``` -## Global flags vs per-subcommand flags +Here `-c packages/db` sets the working directory, and `--dry-run` belongs to `run build`. A flag in the wrong position is an error, not a silent no-op: -A flag is **global** when `noorm` recognizes and hoists it regardless of where it appears — -before or after the subcommand. A flag is **per-subcommand** when it's owned by an -individual command like `noorm sql query` and only works *after* the subcommand name. +``` +$ noorm --dry-run run build +Error: Unrecognized flag '--dry-run' before the subcommand — noorm can't forward it there. The only root-level flag is -c/--cwd <path>; every other flag goes on the command that uses it. Move '--dry-run' after the subcommand instead, e.g. noorm <command> ... --dry-run. +``` -`noorm` hoists exactly one flag on purpose — most flags vary in meaning per command (`-c` is -`--cwd` at the root, but `--config` on many subcommands), so per-subcommand parsing is the -default for everything else, `--dry-run`/`--json`/`--yes` included. +The first non-flag token is the subcommand. It decides who reads each flag: +```mermaid +flowchart TD + F["a flag in argv"] --> Q{"before the<br/>subcommand?"} + Q -->|no| CMD["the command reads it:<br/>--config, --json, -c name"] + Q -->|yes| W{"-c or --cwd?"} + W -->|yes| CWD["working directory<br/>for this run"] + W -->|no| ERR["Error, exit 2"] +``` -### The one global: `-c` / `--cwd <path>` +`--help`, `-h`, `--version`, and `-v` skip this check. `--version` and `-v` work only as the sole argument. -| Flag | Purpose | -|------|---------| -| `--cwd <path>` / `-c <path>` | Change working directory before resolving `.noorm/`. Same idea as `git -C`. Recognized only **before** the subcommand — see the note below. | -`--cwd`/`-c` is global for two reasons: +## Which flags go where? -1. It's consumed **before dispatch** — it sets the working directory everything else - (project discovery, config resolution) resolves against, so it's genuinely the CLI's own - flag rather than any subcommand's. -2. `-c` already means `--config` *after* the subcommand (see the overload note below). - Hoisting it lets `noorm -c <path> run build` mean cwd while `noorm run build -c <name>` - means config, without either shadowing the other. -Examples: +| Flag | Before the subcommand | After the subcommand | +|------|-----------------------|----------------------| +| `-c <path>`, `--cwd <path>`, `--cwd=<path>` | Working directory, like `git -C` | `-c` means `--config` | +| `-c <name>`, `--config <name>` | `-c` is read as `--cwd`; `--config` is an error | Config to use instead of the active one | +| `--json` | Error | Print JSON on stdout | +| `--dry-run` | Error | Preview without executing | +| `--force`, `-f` | Error | Force the operation; the effect varies per command | +| `--yes`, `-y` | Error | Skip confirmations, same as `NOORM_YES=1` | - noorm -c packages/db run build - noorm --cwd /repos/myapp change ff +Each command declares the subset it accepts, and `noorm <command> --help` lists them. It prints `--dryRun` for `--dry-run`; the two spellings are one flag. After the subcommand, a misspelled flag is dropped without an error, so check the spelling there. The full table is in the [CLI reference](/headless#where-do-flags-go). -`--help`/`-h` and `--version`/`-v` are also recognized outside this mechanism — that's citty's -own framework behaviour, not a `noorm`-specific hoist. -An unrecognized flag placed before the subcommand is an error, not a silent no-op: +## What does `-c` / `--cwd` do? - $ noorm --bogus-flag config list - Error: Unrecognized flag '--bogus-flag' before the subcommand — noorm can't forward it there. - The only root-level flag is -c/--cwd <path>; every other flag goes on the command that uses it. - Move '--bogus-flag' after the subcommand instead, e.g. noorm <command> ... --bogus-flag. +noorm normally walks up from the current directory to the nearest `.noorm/` and runs there. `--cwd` skips the walk and runs in the path you name. -### Per-subcommand flags (after the subcommand) +```bash +noorm -c packages/db run build +noorm --cwd /repos/shop change ff +``` -Everything else lives on a specific subcommand — `--config <name>`, `--force`, `--dry-run`, -`--json`, `--yes`/`-y`, and any command-specific flag. These are **not** hoisted, so they only -work after the subcommand name: +| Situation | Result | +|-----------|--------| +| Path is not a directory | `Error: --cwd path is not a directory: <absolute path>`, exit 2 | +| `-c` followed by the subcommand, `noorm -c run build` | `run` is taken as the path, so this fails as above | +| `-c` with no value | Ignored; noorm walks up as usual | +| `-c prod` meant as a config, `noorm -c prod run build` | `prod` is a path, not a config. Use `noorm run build --config prod` | - noorm run build --dry-run - noorm config list --json - noorm secret rm OLD_KEY --yes -Placed before the subcommand instead, none of them are honoured — `noorm --dry-run run build` -is rejected exactly like `noorm --bogus-flag config list` above, naming `--dry-run` as the -offending flag. To see exactly which flags a command accepts, append `--help` to it: +## Next - noorm change ff --help - noorm db create --help - noorm sql query --help - -## The `--config` / `-c` overload - -`-c` is overloaded by position, which is the one sharp edge left in this system: - -- `noorm -c <path> run build` — global `--cwd`, because `-c` appears **before** the subcommand. -- `noorm run build -c <name>` — per-command `--config`, because `-c` appears **after** the subcommand. - -The first non-flag token (the subcommand name) is the boundary that decides which meaning -applies. `--config` in long form has no global meaning at all — it only works after the -subcommand, so `noorm --config prod run build` is not the same as `noorm run build --config prod` -(the former is rejected as an unrecognized global flag). - - -## Mental model - -`noorm` has exactly one true global, `-c`/`--cwd`, and everything else is command-scoped. Put -every flag right next to the subcommand it modifies: - - noorm <subcommand> <flags…> <positional args…> - -If a flag isn't being honored, it's placed before the subcommand boundary — or it's a typo, -in which case `noorm` now says so instead of silently dropping it. - - -## Related - -- [`noorm` help conventions](./help.md) — how to find a command's flags. -- [Headless reference](../headless.md) — full table of common flags. -- [Troubleshooting](../guide/troubleshooting.md) — quick fixes for common surprises. +- [Discovering command help](/cli/help) +- [CLI reference](/headless): every command and the exit-code table +- [Troubleshooting](/guide/troubleshooting) diff --git a/docs/cli/help.md b/docs/cli/help.md index ef68c9ef..6c4eeb3e 100644 --- a/docs/cli/help.md +++ b/docs/cli/help.md @@ -1,47 +1,51 @@ # Discovering command help -Every `noorm` command exposes its own help via the standard `--help` -(or `-h`) flag. There is no `noorm help <subcommand>` form — `noorm`'s -CLI framework (citty) renders help natively per command, and that's the -canonical surface. +Append `--help` or `-h` to any command to see its usage, flags, and examples. +``` +$ noorm run inspect --help +Inspect template context for a .sql.tmpl file (noorm run inspect) -## Canonical form +USAGE noorm run inspect [OPTIONS] <PATH> -Append `--help` (or the short `-h`) to any command — including -multi-segment commands like `db create` or `change ff`: +ARGUMENTS - noorm --help - noorm db --help - noorm db create --help - noorm change ff --help - noorm sql query --help + PATH Path to the .sql.tmpl file (Required) -The help output includes the usage line, every flag with its -description, and a curated `EXAMPLES` block where the command author -added one. +OPTIONS + -c, --config=<config> Use specific configuration + --json Output JSON -## Why not `noorm help <sub>`? -citty (the CLI framework `noorm` is built on) doesn't ship a -re-dispatching `help` subcommand. Adding one would mean a parallel -copy of every command's help text — easy to let drift. The `--help` -flag is uniform, always current, and matches the convention used by -`git`, `kubectl`, `docker`, and most other modern CLIs. +EXAMPLES -If you arrive from a tool like `npm`, `yarn`, or older `commander`-based -CLIs that DO support `npm help <sub>`, just swap to the flag form: + noorm run inspect db/sql/users/001_create.sql.tmpl + noorm run inspect db/sql/core/05_Cron/Crons.sql.tmpl --json +``` - # this errors with "Unknown command help" - noorm help db create +The same flag works at every depth: `noorm --help`, `noorm db --help`, `noorm db create --help`, `noorm change ff --help`. A parent command lists its subcommands. A leaf command lists its flags and an `EXAMPLES` block when its author wrote one. - # this is the equivalent that works - noorm db create --help +## What if I type `noorm help <command>`? -## Related -- [Flag conventions](./flags.md) — where `--help` and other flags must - be placed relative to the subcommand. +noorm has no `help` subcommand. citty renders help per command, so one source serves every command and no second copy can drift. + +| Input | Result | +|-------|--------| +| `noorm <command> --help`, `-h` | Usage, flags, examples; exit 0 | +| `noorm --help` | Command list; exit 0 | +| `noorm` | Command list, then `No command specified.`; exit 1 | +| `noorm help db create` | Command list, then `Unknown command help`; exit 1 | +| `noorm --version`, `noorm -v` | Version only, when it is the sole argument | + +For shell completion, run `noorm complete <shell>` and source the script it prints. [Flag placement](/cli/flags) explains where `--help` and the other flags may go. + + +## Next + + +- [CLI flag conventions](/cli/flags) +- [CLI reference](/headless) diff --git a/docs/cli/identity.md b/docs/cli/identity.md index 06e52575..c6480103 100644 --- a/docs/cli/identity.md +++ b/docs/cli/identity.md @@ -1,79 +1,180 @@ # Identity Management -This guide covers noorm's cryptographic identity system — the keys that authenticate you to shared state, sign your changes, and let teammates encrypt vault secrets to you. +Create the keypair that unlocks your local state, tags your changes in the audit trail, and lets teammates share vault secrets with you. +```bash +noorm identity init --name "Alice Cooper" --email alice@example.com +``` + +``` +Identity created. + Name: Alice Cooper + Email: alice@example.com + Fingerprint: 5e217a60eb7c27d5390cf2f3c8639fb7158dc3006df5057be03d83e3be51d721 +``` -## Overview +`init` writes three files in `~/.noorm/`. The private key never leaves your machine. -noorm uses X25519 cryptographic identities for secure config sharing and audit tracking. Each identity is stored locally in `~/.noorm/identity.key` (private key) and `~/.noorm/identity.json` (metadata). The public key is derived deterministically from the private key, so it never has to be stored separately to be recovered. +| File | Mode | Holds | +|------|------|-------| +| `identity.key` | `0600` | Private key | +| `identity.pub` | `0644` | Public key, derived from the private key | +| `identity.json` | umask default | Name, email, public key, machine, OS, fingerprint | -A disk identity is hashed as `SHA-256(email + name + machine + os)`, where `machine` is the hostname. The same person on two laptops is therefore two identities in the audit trail, and each needs vault access propagated to it. Env-var identities hash differently on purpose, so a CI fleet reads as one user: see [Env-Var Identity (CI)](#env-var-identity-ci) below. -For CI runners, identities load from environment variables instead of files — see the [CI automation guide](../guide/automation/ci.md) for end-to-end setup, including `noorm ci identity new` (generate keypair) and `noorm ci identity enroll` (grant vault access on a target database). +## What is an identity used for? -## Creating an Identity +One identity serves every project on the machine. On CI, environment variables replace the files. -Run `noorm identity init` on your machine to create a keypair. `--name` and `--email` are both required; the command does not prompt. +```mermaid +flowchart TD + ENV["NOORM_IDENTITY_*<br/>environment"] -->|wins when valid| ID["identity in use"] + DISK["~/.noorm/identity.key<br/>identity.json"] -->|otherwise| ID + ID --> ST[".noorm/state/state.enc<br/>decrypted"] + ID --> VT["vault secrets<br/>encrypted to you"] + ID --> AU["fingerprint in<br/>change history"] +``` -```bash -noorm identity init --name "Alice Cooper" --email alice@example.com +The fingerprint identifies you in the database. A disk identity hashes the machine too, so the same person on two laptops appears twice, and each laptop needs vault access. + +| | Disk identity | Environment identity | +|---|---|---| +| Fingerprint input | `SHA-256(email, name, hostname, OS)` | `SHA-256(email, name, public key, "env")` | +| Same person on two machines | Two identities | One identity | +| Files written | `~/.noorm/*` | None | + + +## identity init + + +``` +noorm identity init --name <name> --email <email> [--force --yes] [--json] ``` -This generates a keypair and writes: +| Flag | Effect | +|------|--------| +| `--name <name>` | Display name. Required | +| `--email <email>` | Email address. Required | +| `--force` | Replace an existing identity. Needs `--yes` | +| `--yes`, `-y` | Confirm the replacement. `NOORM_YES` does not count | +| `--json` | JSON result with `name`, `email`, `fingerprint`, `publicKey` | -- Private key: `~/.noorm/identity.key` (mode `0600`) -- Public key: `~/.noorm/identity.pub` (mode `0644`) -- Metadata: `~/.noorm/identity.json` +| Condition | Behavior | +|-----------|----------| +| `--name` or `--email` missing | Usage text and `Missing required argument`, exit 1. No prompt | +| Identity exists, no `--force` | `Identity already exists. Use --force to overwrite.`, exit 1 | +| `--force` without `--yes` | Prints the warning below and `Pass --yes to confirm.`, exit 1 | +| `--force --yes` | Copies the key files to `<file>.bak-<timestamp>`, then replaces them. Prints the backup paths | +| Backup fails | Refuses to overwrite, exit 1 | -The private key never leaves your machine. Your public key is what teammates use to share encrypted state with you. +::: danger +Replacing an identity makes every `.noorm/state/state.enc` on the machine unreadable. State encryption is keyed off the private key, and nothing re-encrypts it under the new one. Restore `identity.key` from the `.bak-` copy to get the configs, secrets, and database passwords back. +::: -If an identity already exists, `init` refuses. Replacing it takes `--force --yes`, and it is destructive: state encryption is keyed off the private key, so every noorm project on the machine loses access to the configs, secrets, and database passwords in its `.noorm/state/state.enc`. Nothing re-encrypts existing state under the new key. noorm copies the old key files to `.bak-<timestamp>` siblings before overwriting, and aborts if it cannot. An ambient `NOORM_YES` does not satisfy this confirmation; only the literal `--yes` flag does. +## identity edit -## Editing Your Identity -Update the display name or email on the existing identity. At least one of `--name` or `--email` must be provided. +``` +noorm identity edit [--name <name>] [--email <email>] [--json] +``` ```bash -noorm identity edit --name "Alice Cooper" -noorm identity edit --email "alice@newjob.com" +noorm identity edit --name "Alice C. Cooper" +``` + +``` +Identity updated. + Name: Alice C. Cooper + Email: alice@example.com + Fingerprint: bf151889b92fdc4f65e0b717dc3e93465488beba27f4df00be06095a70547936 + +NOTE: the fingerprint changed, because it is derived from your name and email. +Vault access is granted against the old fingerprint, so you will need a team +member with vault access to propagate it to the new one. + Previous: 5e217a60eb7c27d5390cf2f3c8639fb7158dc3006df5057be03d83e3be51d721 ``` +| Condition | Behavior | +|-----------|----------| +| Neither `--name` nor `--email` | `At least one of --name or --email must be provided.`, exit 2 | +| No identity yet | ``No identity found. Run `noorm identity init` first.``, exit 2 | +| Name or email changes | The fingerprint changes. `--json` reports `previousFingerprint` and `fingerprintChanged` | +| Machine | Carried over unchanged | -## Exporting Your Public Key -Print your public key so teammates can add you to encrypted vaults: +## identity export + + +``` +noorm identity export [--json] +``` + +Prints your name, email, fingerprint, and public key. Send the public key to a teammate who is adding you to the vault. ```bash -noorm identity export noorm identity export --json ``` +```json +{"success":true,"name":"Alice Cooper","email":"alice@example.com","fingerprint":"5e217a60…","publicKey":"302a3005…"} +``` -## Listing Known Identities +| Condition | Behavior | +|-----------|----------| +| No identity | ``No identity found. Run `noorm identity init` to create one.``, exit 2 | +| Public key unreadable | `Failed to load public key. Identity files may be corrupted.`, exit 1 | -Show every identity discovered from database syncs (the audit trail of who has touched shared state): -```bash -noorm identity list -noorm identity list --json +## identity list + + +``` +noorm identity list [--json] ``` +Lists the people this project has seen, collected when a config syncs with its database. + +``` +Known users (1): + Alice Cooper <alice@example.com> + Fingerprint: 5e217a60eb7c27d5390cf2f3c8639fb7158dc3006df5057be03d83e3be51d721 + Machine: alice-mbp + Source: prod + Last seen: 2026-10-21 05:41:59 +``` + +| Condition | Behavior | +|-----------|----------| +| Nobody synced yet | `No known users found.`, exit 0. `--json` gives `{"users":[]}` | +| State cannot be decrypted | `Failed to load state: …`, exit 1 | + + +## How does CI get an identity? + + +Set three variables before the first `noorm` command. Every command in that process then uses the environment identity and writes nothing to `~/.noorm/`. + +| Variable | Value | +|----------|-------| +| `NOORM_IDENTITY_PRIVATE_KEY` | Hex-encoded X25519 private key, 96 hex characters | +| `NOORM_IDENTITY_NAME` | Display name | +| `NOORM_IDENTITY_EMAIL` | Email address | -## Env-Var Identity (CI) +| Condition | Behavior | +|-----------|----------| +| All three valid | Environment identity is used, even if `~/.noorm/` also holds one | +| One missing, blank, or the key is not valid hex | Ignored without an error, and the disk identity is used | +| Leading or trailing whitespace | Trimmed | -When `NOORM_IDENTITY_PRIVATE_KEY`, `NOORM_IDENTITY_NAME`, and `NOORM_IDENTITY_EMAIL` are set at process startup, every `noorm` command in that process uses the env-derived identity without touching `~/.noorm/`. +`noorm ci identity new` generates a CI keypair, and `noorm ci identity enroll` grants it vault access on a database. The [CI guide](/guide/automation/ci) walks through both. -The variables are read once when the process starts, so they apply to every command that process runs. -**Guarantees:** +## Next -- The public key is **derived** from the private key, not separately specified — same key, same public half. -- The identity hash is **independent of hostname or OS** (CI uses `os='env'`, `machine=publicKey`) — same key, same identity across every runner. -- Env vars are **trimmed automatically** — leading/trailing whitespace is removed. -- **Env vars win over disk** — if both are present, env takes precedence. -- **No files written** — env-based bootstrap is read-only. -To generate a CI-shaped keypair without touching the database, use `noorm ci identity new` (see the [CI automation guide](../guide/automation/ci.md)). +- [CI automation](/guide/automation/ci): provision and enroll a CI identity +- [Vault](/guide/environments/vault): share secrets between identities +- [Secrets](/cli/secret): config-scoped secrets, which the identity key protects diff --git a/docs/cli/init.md b/docs/cli/init.md index d2099464..361a1b56 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -1,70 +1,105 @@ # noorm init -Bootstrap a new noorm project interactively. Creates identity (if not present), -project structure (`sql/`, `changes/`, `.noorm/`), `settings.yml`, and `state.enc`. +Create a noorm project in the current repository: the `.noorm/` state folder and the `db/` folder that holds your schema. + +```bash +noorm init --yes +``` + +``` +┌ noorm init +│ +● Using existing identity: Alice Cooper <alice@example.com> +│ +◇ Done +│ +└ noorm initialized. + Created: + db/sql/.gitkeep + db/changes/.gitkeep + db/AGENTS.md + db/CLAUDE.md + .noorm/.gitignore + .noorm/settings.yml + .noorm/state/state.enc +``` + +``` +. +├── .gitignore # gains "# noorm" and ".noorm/state/" +├── .noorm/ +│ ├── .gitignore # state/ +│ ├── settings.yml # paths: ./db/sql, ./db/changes +│ └── state/state.enc # configs and secrets, encrypted with your identity key +└── db/ + ├── AGENTS.md # points coding agents at the /noorm skill + ├── CLAUDE.md # imports AGENTS.md + ├── changes/ + └── sql/ +``` + +`db/AGENTS.md` also points data modeling at [ignatius](/modeling/) and its [`/ignatius-modeling` skill](/modeling/modeling-skill), and holds a marker comment for your own steering. `db/CLAUDE.md` is one line, `@AGENTS.md`. Without `--yes`, `init` prompts for a name and email when no identity exists yet. ## Flags -- `--force` / `-f` — overwrite an existing `.noorm/` directory -- `--here` — initialize in the original cwd, ignoring any parent `.noorm/` discovered while walking up -- `--yes` / `-y` — proceed non-interactively when a full identity already exists at `~/.noorm/identity.{key,pub,json}`. Equivalent to setting `NOORM_YES=1`. See [Non-interactive operation](../guide/automation/non-interactive.md) for details. +| Flag | Effect | +|------|--------| +| `--force`, `-f` | Initialize a folder that already has `.noorm/`. Resets `settings.yml` to defaults and keeps its `paths` | +| `--here` | Initialize the directory you launched from, ignoring a parent `.noorm/` | +| `--yes`, `-y` | Skip prompts. Needs an identity in `~/.noorm/`. `NOORM_YES=1` is the same | -## Picking the project root +`noorm init` has no `--json` flag. -Every `noorm` invocation walks up from the current directory looking for a -`.noorm/` folder, then `chdir`s into it. That is what lets you run commands from -deep inside a repo without thinking about paths. It also means `noorm init` from -a subfolder of an already-initialized project tries to re-init the *parent*. -Two flags opt out of that behavior, depending on where you want to invoke from: +## Which folder becomes the project root? -- `noorm init --here` — stay in the cwd you launched from. Use this when you want to nest a fresh noorm project inside one that already exists (e.g. a `packages/db/` workspace under a repo that has its own `.noorm/`). -- `noorm -c <path> init` — global flag (must precede the subcommand, like `git -C`). Resolves `<path>` relative to the current shell, `chdir`s into it, and skips the walk-up entirely. +Every `noorm` command walks up from the current directory to the nearest `.noorm/` and runs there. `init` follows the same rule, so it targets the parent project unless you say otherwise. -## Environment +| Invocation | Project root | +|------------|--------------| +| `noorm init` | Nearest parent holding `.noorm/`; the current directory when none does | +| `noorm init --here` | The directory you launched from. Nests a project inside another, such as `packages/db/` | +| `noorm -c packages/db init` | `packages/db`, with no walk up. `-c` must come before `init` | -Interactive TTY required by default. Fails with exit code 2 (usage error) in CI or -piped stdin unless `--yes` / `NOORM_YES=1` is set AND a full identity already exists. +## What happens to an existing project? -## Non-interactive (CI) flow -`noorm init --yes` (or `NOORM_YES=1 noorm init`) skips all prompts when a full -crypto identity already lives at `~/.noorm/identity.{key,pub,json}`. The identity -must be in place first — `--yes` does NOT bootstrap one. If any of the three -files is missing, `noorm init --yes` exits 2 with: +| Condition | Behavior | +|-----------|----------| +| `.noorm/` exists, no `--force` | `Error: noorm project already initialized at <path>. Use --force to reinitialize.`, exit 1 | +| `--force` | `settings.yml` resets to defaults except `paths`. `db/AGENTS.md` and `db/CLAUDE.md` are written only when absent, so your edits stay | +| `paths` are not `./db/sql` and `./db/changes` | Layout kept; no `db/` folder is created. A project at `./sql` and `./changes` stays there | +| `.gitignore` already lists `.noorm/state/` | Left alone; otherwise the block is appended | - Error: noorm init --yes requires an existing identity at ~/.noorm/identity.{key,pub,json}. - Run: noorm identity init --name "Your Name" --email "you@example.com" - Then re-run: noorm init --yes +`db/sql/.gitkeep` and `db/changes/.gitkeep` are listed under `Created:` on every run for a default layout. They are empty files and are not overwritten. -The end-to-end CI bootstrap is two commands: - # CI flow - noorm identity init --name "$CI_USER" --email "$CI_EMAIL" - noorm init --yes +## How do I run it in CI? -## Example +`--yes` skips every prompt but cannot create an identity. Create one first. A checkout that already has `.noorm/` needs no `init`, and running it there exits 1, so guard it: - noorm init - noorm init --force - noorm init --here - noorm init --yes # non-interactive, requires existing identity +```bash +[ -f ~/.noorm/identity.key ] || noorm identity init --name "$CI_USER" --email "$CI_EMAIL" +[ -d .noorm ] || noorm init --yes +``` - # From the repo root, init a sub-project at packages/db - noorm -c packages/db init +| Condition | Behavior | +|-----------|----------| +| No terminal and no `--yes` | `Error: noorm init requires an interactive terminal.`, exit 2 | +| `--yes`, but one of `identity.key`, `identity.pub`, `identity.json` is missing | `Error: noorm init --yes requires an existing identity at ~/.noorm/identity.{key,pub,json}.` with the two commands to fix it, exit 2 | +| Ctrl+C at a prompt | `Cancelled.`, exit 0 | +| Bootstrap fails after the prompts | The error text, exit 1 | -## Next steps +## Next -After `init` lays down `settings.yml` and `.noorm/`, the database -itself still has to be provisioned. See -[Creating a database](../guide/database/create.md) for the configs-first -workflow (`config import` → `db create` → `config use`), and -[CLI flag conventions](./flags.md) for `--json` placement and the -global-vs-per-subcommand rules. + +- [Creating a database](/guide/database/create): add a config, create the database, build +- [First build](/getting-started/first-build) +- [CLI flag conventions](/cli/flags): where `-c` and `--yes` go diff --git a/docs/cli/run.md b/docs/cli/run.md index 3e32b952..66e49e33 100644 --- a/docs/cli/run.md +++ b/docs/cli/run.md @@ -1,134 +1,190 @@ # noorm run -Execute SQL files against the active database. Five subcommands execute SQL — -`build`, `file`, `dir`, `files`, and `exec` — and they all share the same -result shape and exit-code semantics. Two more, `inspect` and `preview`, read a -`.sql.tmpl` file without touching the database. +Execute SQL files against the active database, or render a template without touching it. +```bash +noorm run build +noorm run build +``` -## Subcommands +``` +• log Build completed status=success filesRun=3 filesSkipped=0 filesFailed=0 durationMs=13.153750000000002 -| Command | Purpose | -|---------|---------| -| `noorm run build` | Execute every file under `paths.sql/` | -| `noorm run file <path>` | Execute a single file | -| `noorm run dir <path>` | Execute every file in a directory | -| `noorm run files --paths a.sql,b.sql` | Execute an explicit, ordered list | -| `noorm run exec <dir-or-glob>` | Discover files by glob, then execute | -| `noorm run inspect <path>` | Show the template context for a `.sql.tmpl` file | -| `noorm run preview <path>` | Render a `.sql.tmpl` file and print the SQL | +• log /work/shop/db/sql/01_tables/01_party.sql (skipped: unchanged) +• log /work/shop/db/sql/01_tables/02_sales_order.sql (skipped: unchanged) +• log /work/shop/db/sql/02_views/01_open_orders.sql (skipped: unchanged) +• log Build completed status=success filesRun=0 filesSkipped=3 filesFailed=0 durationMs=3.4714580000000126 +``` + +The first run executes all three files. The second skips them, because their checksums match a prior success. Log lines go to stderr. With `--json`, the result is one JSON object on stdout. + + +## Which command runs what? + + +| Command | Runs | Needs the database | +|---------|------|--------------------| +| `noorm run build` | Every file under `paths.sql`, narrowed by `build.include` and `build.exclude` | Yes | +| `noorm run file <path>` | One file | Yes | +| `noorm run dir <path>` | Every `.sql` and `.sql.tmpl` file under a directory, recursively, sorted | Yes | +| `noorm run files --paths a.sql,b.sql` | The listed files, in the order given | Yes | +| `noorm run exec <dir-or-glob>` | A directory (as `dir`) or the `.sql` and `.sql.tmpl` files a glob matches, sorted | Yes | +| `noorm run inspect <path>` | Nothing. Lists what a `.sql.tmpl` can read | Only to read vault secrets | +| `noorm run preview <path>` | Nothing. Prints the rendered SQL | Only to read vault secrets | + +Each file takes one of these paths, and the first failure stops the batch: + +```mermaid +flowchart TD + F["next file"] --> C{"last run succeeded<br/>on the same SQL?"} + C -->|"yes, no --force"| S["skipped: unchanged"] + C -->|"no, or --force"| X["execute"] + X -->|ok| N["next file"] + X -->|error| STOP["failed: batch stops"] + S --> N +``` + +Files after the failed one do not run and do not appear in the result. ## Flags -Common to the five executing subcommands: -- `--config <name>` / `-c <name>` — use a named config from `state.enc` -- `--force` / `-f` — re-run even if the file's checksum matches a prior success -- `--dry-run` — render templates and write to `tmp/` without executing -- `--json` — emit machine-readable JSON on stdout +| Flag | Commands | Effect | +|------|----------|--------| +| `--config <name>`, `-c <name>` | all seven | Use a stored config instead of the active one | +| `--force`, `-f` | the five executing commands | Run even when the checksum matches a prior success | +| `--dry-run` | the five executing commands | Render templates into `tmp/` and write nothing to the database | +| `--json` | all seven | Print the result as JSON on stdout | -`inspect` and `preview` take `--config` and `--json` only. Neither executes, so -`--force` and `--dry-run` have nothing to act on. +`inspect` and `preview` never execute, so they have no `--force` or `--dry-run`. +::: warning +A dry run writes every rendered file with its secrets resolved, in plaintext, to `tmp/`. noorm creates the files owner-only (`0600`) but does not add `tmp/` to `.gitignore`. A template at `db/sql/09_seeds/01_products.sql.tmpl` renders to `tmp/db/sql/09_seeds/01_products.sql`. +::: -## Exit codes -| Code | Meaning | -|------|---------| -| `0` | Every file succeeded or was skipped | -| `1` | Total failure — setup failed (could not connect, project not initialized), or the runner ran and nothing succeeded | -| `2` | Usage error — a missing flag, a directory or glob that matched no SQL files (`dir`, `exec`), or a template that isn't there (`inspect`, `preview`). Nothing was attempted | -| `3` | Partial failure — some files succeeded and some failed. Re-running is not automatically safe | +## What does each command do at the edges? + +| Command | Condition | Behavior | +|---------|-----------|----------| +| `build` | A `build.include` entry matches no file | Warns `Ignored N build.include entries that matched no files`. Paths are relative to `paths.sql`: `01_tables`, not `sql/01_tables` | +| `build` | A `build.exclude` entry matches no file | Warns that nothing was excluded, so those files ran | +| `build` | `--dry-run` | Header reads `Build (dry-run) completed`; each file line ends `-> <path in tmp/>`; `--json` adds `"dryRun": true` | +| `file` | File does not exist | `error: Failed to read file: <path>`, exit 1 | +| `dir` | Directory does not exist | `Error: Directory not found: <path>`, exit 2 | +| `dir` | No SQL files inside | `Error: No SQL files found in: <path>`, exit 2 | +| `files` | `--paths` omitted | Usage text and `Missing required argument: --paths`, exit 1 | +| `files` | `--paths ""` or only commas | `Error: --paths must contain at least one file path`, exit 1 | +| `exec` | Glob or directory matches nothing | `Error: No SQL files found matching: <input>`, exit 2 | +| `inspect`, `preview` | Template does not exist | `Error: Template not found: <path>`, exit 2 | +| `inspect`, `preview` | The database is unreachable | Vault secrets are skipped, local secrets still render, and the output says `Vault-backed secrets were not resolved (no reachable database) — showing local-only secrets.`; `--json` adds `vaultProbeFailed: true` and `notice` | +| `inspect`, `preview` | The role lacks `run:file` on the config | Policy message, exit 1 | +| any executing command | Role lacks the permission, such as the default `viewer` agent role | `Cannot run.build on config "dev": "run:build" is not allowed on config "dev" (role: viewer).`, exit 1 | +| MSSQL files | `GO` alone on a line | Split into batches. A failure is prefixed `[batch N of M]`. See [execution](/guide/sql-files/execution) | -## Output: failure -When a file errors, the human-readable output includes the SQL error -beneath the file's status line so you don't have to scroll the log -looking for the cause: +## What do inspect and preview show? - sql/02_tables/Memory.sql (failed) - error: relation "memory" does not exist - Build completed status=partial filesRun=12 filesSkipped=0 filesFailed=1 durationMs=84 -A batch where some files ran and some failed reports `partial` and exits `3`. -Only a batch where nothing succeeded reports `failed` and exits `1`. +`inspect` lists the variables a template can read. `preview` prints the SQL the template renders to, on stdout, so it pipes to a file. -The same information lives on the JSON output's per-file entries. The -`--json` payload is the `BatchResult` (or `FileResult` for `noorm run file`) -returned by the SDK plus the envelope's `success` flag, so `files[].error` is -populated for every failed file: +```bash +noorm run inspect db/sql/09_seeds/01_products.sql.tmpl +``` - { - "success": false, - "status": "partial", - "files": [ - { - "filepath": "sql/02_tables/Memory.sql", - "status": "failed", - "error": "relation \"memory\" does not exist", - "durationMs": 4.2, - "checksum": "..." - } - ], - "filesRun": 12, - "filesSkipped": 0, - "filesFailed": 1, - "durationMs": 84 - } +``` +Template: db/sql/09_seeds/01_products.sql.tmpl +Config: dev +Data Files: + $.products Array [1] -## Output: skip +Built-ins: include, escape, quote, json, now, uuid +Secrets: 0 resolved, 0 global +``` -A file is skipped when its checksum matches a previous successful -execution. The reason is shown inline: +```bash +noorm run preview db/sql/09_seeds/01_products.sql.tmpl > rendered.sql +``` - sql/seeds/Sentinels.sql.tmpl (skipped: unchanged) +```sql +INSERT INTO product (sku, name, list_price) +VALUES ('HW-DONGLE', 'USB security key', 49); +``` -In JSON form, `skipReason` is set to `'unchanged'` (or `'already-run'` -for the change-level checks): +`inspect` adds `Helpers ($helpers):` and `Helper Errors:` sections when the folder has a `$helpers` file. `preview` prints resolved secrets, and both commands execute the template's helper scripts, so both check the `run:file` permission first. `--json` gives `{ filepath, context, … }` for `inspect` and `{ filepath, sql, durationMs }` for `preview`. - { - "filepath": "sql/seeds/Sentinels.sql.tmpl", - "status": "skipped", - "skipReason": "unchanged", - "checksum": "..." - } +## What does the output look like? + + +A failed file prints its SQL error under its status line: + +``` +✗ log /work/shop/db/sql/03_mixed/02_bad.sql (failed) +✗ log error: [Err 1] no such table: invoice_line_missing +✗ log Run directory partial filesRun=1 filesSkipped=0 filesFailed=1 +``` + +The summary line reads `Build completed` for `build`, `Run directory <status>` for `dir`, `Run files <status>` for `files`, and `Run exec <status>` for `exec`. `--json` prints the batch result, with `success` added: + +```json +{ + "success": false, + "status": "partial", + "files": [ + { "filepath": "/work/shop/db/sql/03_mixed/01_ok.sql", "checksum": "8ece28fd…", "status": "success", "durationMs": 1.36 }, + { "filepath": "/work/shop/db/sql/03_mixed/02_bad.sql", "checksum": "3dd15369…", "status": "failed", "error": "[Err 1] no such table: invoice_line_missing", "durationMs": 0.24 } + ], + "filesRun": 1, + "filesSkipped": 0, + "filesFailed": 1, + "durationMs": 7.4, + "changeId": 1 +} +``` + +| Field | Appears when | +|-------|--------------| +| `files[].skipReason` | A file was skipped: `unchanged` | +| `files[].outputPath` | A dry run wrote the file | +| `unmatchedInclude`, `unmatchedExclude` | `build` had patterns that matched nothing | +| `error` | The batch failed before any file ran, or `dir` found no files | +| `dryRun: true` | `build --dry-run` | + +`run file --json` returns one file result, with `success` added. + + +## Exit codes -If a file was skipped because you forgot `--force` after wiping the -database, this is the signal to look for. Pass `--force` to re-run -regardless of the checksum cache. +| Code | Meaning | Remedy | +|------|---------|--------| +| 0 | Every file succeeded or was skipped | None | +| 1 | Nothing succeeded: no connection, project not initialized, role denied, a missing required flag, `file` or `files` target missing, or every file that ran failed | Read the `error:` line, fix the SQL or the connection, run again | +| 2 | Nothing was attempted: `dir` or `exec` found no SQL files, or `inspect` or `preview` found no template | Fix the path or glob | +| 3 | Some files succeeded and some failed | Part of the database changed. Check the failed file's effect, fix it, run again; files that succeeded skip | -## MSSQL: `GO` batches +A re-run that fails again at the same file reports `failed` and exits 1, not 3: the earlier files skip, so nothing executed successfully in that run. -On MSSQL connections, files containing multiple statements separated by `GO` -(the T-SQL batch separator) work as you'd expect — `CREATE PROCEDURE`, -`CREATE FUNCTION`, `CREATE TRIGGER`, `CREATE VIEW`, and `CREATE TYPE` for -table-valued parameters can be grouped in a single file. The runner splits on -`GO` (anchored to its own line) and executes batches sequentially. If a batch -fails, the error is prefixed with `[batch N of M]` so you can identify the -offending statement without re-reading the file. See -[Writing MSSQL Files with GO](/guide/sql-files/execution#mssql-multiple-statements-per-file) -for examples and the -[runner internals](/dev/runner#mssql-batch-handling) for the splitter rules -and known limitations. +## What bites -## Examples +| Symptom | Cause | Fix | +|---------|-------|-----| +| `--force` fails with `table party already exists; 1 prior successful execution` | `--force` re-executes files that already ran, and `CREATE TABLE` is not guarded | Use `--force` only on files that are safe to run twice; new schema reaches an existing database through a [change](/guide/changes/add-a-column) | +| `run build` fails after you edit a table file | The new checksum makes the file run again against a table that exists | Build only fresh databases. Existing ones catch up with `noorm change ff` | +| Seed files report `skipped: unchanged` after `db truncate` | Truncate keeps noorm's tracking tables, so each file still has a successful record | Re-run the seed files with `--force` | - noorm run build - noorm run build --force - noorm run build --json - noorm run file sql/init.sql - noorm run file seeds/test-data.sql.tmpl --dry-run +## Next - noorm run dir sql/02_views/ - noorm run files --paths sql/01_tables/users.sql,sql/01_tables/orders.sql - noorm run exec "sql/**/*.sql" +- [Execution](/guide/sql-files/execution): skipping, tracking, and slow files +- [Templates](/guide/sql-files/templates): what `inspect` lists +- [Forward and revert](/guide/changes/forward-revert): the other way SQL reaches a database +- [CLI reference](/headless#exit-codes): the shared exit-code contract diff --git a/docs/cli/secret.md b/docs/cli/secret.md index d23560c0..3a3f7c57 100644 --- a/docs/cli/secret.md +++ b/docs/cli/secret.md @@ -1,77 +1,114 @@ # noorm secret -Manage **config-scoped local secrets** — values stored on disk in `.noorm/state/state.enc`, -tied to a single config, and never shared with the rest of the team. This is one of three -places a secret value can live; see [Which tier?](#which-tier) below before reaching for it. +Store a secret on one config, encrypted on your machine and shared with no one. +```bash +printf %s "$GATEWAY_KEY" | noorm secret set PAYMENT_GATEWAY_KEY --stdin --config prod +noorm secret list --config prod +``` -## Subcommands +``` +Secret "PAYMENT_GATEWAY_KEY" set for config "prod". +Secrets for "prod" (1): + PAYMENT_GATEWAY_KEY +``` -| Command | Purpose | -|---------|---------| -| `noorm secret set <key> [value]` | Store a secret for the active or named config | -| `noorm secret list` | List secret keys for a config (values never shown) | -| `noorm secret rm <key>` | Remove a secret from a config | +The value lands in `.noorm/state/state.enc`. `list` shows keys only. A template reads it as `$.secrets.PAYMENT_GATEWAY_KEY`. -## Flags +## Where should a secret live? -Common to every subcommand: -- `--config <name>` / `-c <name>` — target a config other than the active one -- `--json` — emit machine-readable JSON on stdout +A template takes the first tier that holds the key: config, then global, then vault. A miss throws `Secret "KEY" not found (searched: config-local, global-local, vault)`. -`secret set` additionally accepts: +| Tier | Stored in | Set with | Shared with the team | +|------|-----------|----------|----------------------| +| Config | `.noorm/state/state.enc`, per config | `noorm secret set`, or the TUI (`c`, highlight a config, `k`) | No. Per user, per machine | +| Global | `state.enc`, all your configs | No CLI or TUI command writes one | No | +| Vault | The database, encrypted | [`noorm vault set`](/guide/environments/vault) | Yes | -- `--stdin` — read the value from stdin instead of argv, so it never reaches the process table, shell history, or a `set -x` trace. Omit the positional value when you pass it. -- `--yes` / `-y` — confirm the write on a config whose `secret:write` access is a confirm cell (an `operator` user role). Without it the command exits non-zero. +Use `noorm secret` for a value that differs for you. Use `noorm vault set` for a value the whole team and CI need. A team value set with `noorm secret set` lives in your `state.enc` only, so a teammate or a CI runner finds nothing, or finds a stale vault value. [`noorm settings secret`](/cli/settings-secret) declares which secrets a stage requires; it never stores a value. -`secret rm` additionally requires: -- `--yes` / `-y` — confirm the deletion; omitted, the command exits non-zero without deleting +## secret set -## Which tier? +``` +noorm secret set <key> [value] [--stdin] [--config <name>] [--json] [--yes] +``` -noorm resolves `$.secrets.KEY` in a template from three tiers, config-local first: +| Flag | Effect | +|------|--------| +| `<key>` | Secret name. Required | +| `[value]` | The value. Omit it with `--stdin` | +| `--stdin` | Read the value from stdin, keeping it out of the process table, shell history, and `set -x` traces. One trailing newline is stripped | +| `--config <name>`, `-c <name>` | Target a config other than the active one | +| `--json` | `{"success":true,"configName":"prod","key":"PAYMENT_GATEWAY_KEY"}` | +| `--yes`, `-y` | Confirm the write when your role on the config asks first | -| Tier | Command | Scope | Shared with team? | -|------|---------|-------|--------------------| -| Config-local | `noorm secret set` | One config | No — per-user, per-machine | -| Global-local | TUI only (`noorm ui` → Settings → Secrets) | Every config on this machine | No — per-user, per-machine | -| Vault | [`noorm vault set`](../guide/environments/vault.md) | Every config, every teammate | Yes — encrypted in the database | +| Condition | Behavior | +|-----------|----------| +| Value and `--stdin` together | `Pass the value as an argument or via --stdin, not both.`, exit 1 | +| Neither | `Missing secret value. Pass it as an argument, or pipe it in with --stdin.`, exit 1 | +| `--stdin` with empty input | `No secret value on stdin.`, exit 1 | +| No `--config` and no active config | `No config specified and no active config set. Use --config or run "noorm config use <name>".`, exit 1 | +| Config does not exist | `Config "nope" not found.`, exit 1 | +| Your role is `operator` | `Writing a secret to config "staging" requires confirmation (yes-staging). Pass --yes to confirm.`, exit 1 | +| Your role does not allow `secret:write` | The policy message, exit 1 | -If a value should be different for you than for the rest of the team, it belongs in -`noorm secret` (or the TUI's global-local screen). If it should be the same for everyone — -CI included — it belongs in `noorm vault set`. `noorm secret set` on a value the whole team -needs is the mistake this page exists to prevent: it lands in your local `state.enc` only, -so a teammate or a CI runner resolving the same key finds nothing and either fails loudly (a -missing secret now throws — see [Templates](../guide/sql-files/templates.md)) or silently -picks up a stale vault value instead of the one you just set. +`--yes` and `NOORM_YES=1` both confirm. -See [Secrets](../guide/environments/secrets.md) for config-scoped vs. global-local, and -[Vault](../guide/environments/vault.md) for the full three-tier resolution hierarchy and -team-provisioning workflow. +## secret list -## Examples - noorm secret set API_KEY "sk-live-..." - echo "$API_KEY" | noorm secret set API_KEY --stdin - noorm secret set DB_PASSWORD "secret123" --config prod - noorm secret set API_KEY "sk-live-..." --json +``` +noorm secret list [--config <name>] [--json] +``` - noorm secret list - noorm secret list --config staging --json +```bash +noorm secret list --json +``` - noorm secret rm OLD_KEY --yes - noorm secret rm OLD_KEY --yes --config prod +```json +{"success":true,"configName":"prod","keys":["API_KEY","DB_PASSWORD"]} +``` +An empty config prints `No secrets set for config "prod".`. Unknown config, no active config, or a role without `secret:read` exit 1, with the messages above. -## Related -- [`noorm settings secret`](./settings-secret.md) — edits which secrets a stage *requires*, - not their values. -- [Secrets guide](../guide/environments/secrets.md) — full CLI + TUI workflow. -- [Vault guide](../guide/environments/vault.md) — team-shared secrets and CI provisioning. +## secret rm + + +``` +noorm secret rm <key> --yes [--config <name>] [--json] +``` + +```bash +noorm secret rm OLD_KEY --yes --config prod +``` + +| Condition | Behavior | +|-----------|----------| +| No `--yes` and no `NOORM_YES` | `Pass --yes to confirm deletion of secret "OLD_KEY" from config "prod".`, exit 1 | +| Key not on the config | `Secret "OLD_KEY" not found in config "prod".`, exit 2 | +| Removed | `Secret "OLD_KEY" removed from config "prod".`, exit 0 | + + +## Exit codes + + +| Code | Meaning | Remedy | +|------|---------|--------| +| 0 | Done | None | +| 1 | Bad value input, unknown or missing config, state would not load, role denied, or confirmation missing | Read the message. Add `--config`, `--yes`, or `--stdin` as it says | +| 2 | `rm` named a key the config does not hold | `noorm secret list` shows the keys | + + +## Next + + +- [Secrets](/guide/environments/secrets): config-scoped versus global secrets +- [Vault](/guide/environments/vault): team secrets and CI provisioning +- [`noorm settings secret`](/cli/settings-secret): declare required secrets per stage diff --git a/docs/cli/settings-edit.md b/docs/cli/settings-edit.md index 3df3c810..ab2f8014 100644 --- a/docs/cli/settings-edit.md +++ b/docs/cli/settings-edit.md @@ -1,21 +1,32 @@ # noorm settings edit -Interactive editor for `settings.yml` — paths, build rules, strict mode, -logging, stages, conditional rules, teardown. +Edit `.noorm/settings.yml` from a menu instead of by hand. +```bash +noorm settings edit +``` -## Environment +The menu offers Paths, Build, Strict, Logging, Stages, Rules, Teardown, and Done. Each section saves as soon as its prompts finish. Cancelling inside a section drops that section's edit and returns to the menu. Done, or a cancel at the menu, exits 0 with `Settings saved.` -Interactive TTY required. Fails with exit code 2 (usage error) in CI or piped stdin. -`settings.yml` is plain YAML, so `--yes` / `NOORM_YES` has no useful -non-interactive path here. Both produce a redirect hint pointing you at -direct YAML edits, with `noorm settings build` available afterwards to -re-validate and apply defaults. See -[Non-interactive operation](../guide/automation/non-interactive.md). +## Where does it refuse to run? -## Example +It needs a terminal. `settings.yml` is plain YAML, so a script edits the file itself. - noorm settings edit +| Condition | Behavior | +|-----------|----------| +| `--yes`, `-y`, or `NOORM_YES=1` | `Error: noorm settings edit is interactive only.` Edit `settings.yml` directly, then run `noorm settings build` to validate. Exit 2 | +| No terminal on stdin | `Error: noorm settings edit requires an interactive terminal.`, exit 2 | +| No `settings.yml` | `Error: No settings.yml found. Run: noorm settings init`, exit 2 | +| `settings.yml` does not load | `Error: Failed to load settings: …`, exit 1 | + +`--yes` is the only flag, and it exists for the redirect above. See [non-interactive operation](/guide/automation/non-interactive). + + +## Next + + +- [`noorm settings secret`](/cli/settings-secret): declare required secrets per stage +- [Stages](/guide/environments/stages) diff --git a/docs/cli/settings-secret.md b/docs/cli/settings-secret.md index 16c7b8c5..ceda6ead 100644 --- a/docs/cli/settings-secret.md +++ b/docs/cli/settings-secret.md @@ -1,23 +1,42 @@ # noorm settings secret -Interactive editor for the secret *requirement* declarations in -`settings.yml`. Manages which secrets each stage requires — not the -actual values. +Declare which secrets a stage requires, from a menu. It writes requirements to `settings.yml` and never stores a value. -For secret values, see [`noorm secret set`](./secret.md). +```bash +noorm settings secret +``` +It prints the current requirements, then loops over Add, Edit, Remove, List, and Done. A requirement has a scope (`Universal`, or one stage), a key, a type (`string`, `password`, `api_key`, or `connection_string`), a description, and a required flag. It lands in `settings.yml`: -## Environment +```yaml +# .noorm/settings.yml +stages: + prod: + secrets: + - key: PAYMENT_GATEWAY_KEY + type: password + required: true +``` -Interactive TTY required. Fails with exit code 2 (usage error) in CI or piped stdin. +A universal requirement sits in the top-level `secrets:` list, and `required` defaults to true. Values are set with [`noorm secret set`](/cli/secret). -`--yes` / `NOORM_YES` produces a redirect hint here — the requirement -list lives directly in `settings.yml`, so editing the YAML by hand is the -non-interactive path. Use `noorm secret set <key> <value>` for values. -See [Non-interactive operation](../guide/automation/non-interactive.md). +## Where does it refuse to run? -## Example - noorm settings secret +| Condition | Behavior | +|-----------|----------| +| `--yes`, `-y`, or `NOORM_YES=1` | `Error: noorm settings secret is interactive only.` Edit the `secrets` section of `settings.yml` directly, and use `noorm secret set <key> <value>` for values. Exit 2 | +| No terminal on stdin | `Error: noorm settings secret requires an interactive terminal.`, exit 2 | +| No `settings.yml` | `Error: No settings.yml found. Run: noorm settings init`, exit 2 | +| `settings.yml` does not load | `Error: Failed to load settings: …`, exit 1 | + +`--yes` is the only flag. See [non-interactive operation](/guide/automation/non-interactive). + + +## Next + + +- [Stages](/guide/environments/stages#how-do-i-require-a-secret): how a config meets a requirement +- [`noorm secret`](/cli/secret): store the values diff --git a/docs/cli/sql-repl.md b/docs/cli/sql-repl.md index 46b7c2de..d3e046f8 100644 --- a/docs/cli/sql-repl.md +++ b/docs/cli/sql-repl.md @@ -1,28 +1,41 @@ # noorm sql repl -Launch the TUI directly on the SQL Terminal screen. Uses the active config -unless `--config` is provided. +Open the TUI directly on the SQL terminal, against the active config or the one you name. -For one-off non-interactive queries (CI scripts, ad-hoc inspection), use [`noorm sql query`](./sql.md) — `repl` requires an interactive TTY. +```bash +noorm sql repl --config dev +``` + +The terminal opens with the query box focused. `Enter` runs the query and `Esc` backs out. [The TUI page](/tui#sql-terminal) lists the keys, and [the SQL terminal guide](/guide/database/terminal) covers history, result browsing, and cancel. ## Flags -- `--config <name>` / `-c` — switch active config before launching +| Flag | Effect | +|------|--------| +| `--config <name>`, `-c <name>` | Make this config the active one, then launch. The switch is saved, so it outlasts the session | +| `--yes`, `-y` | Accepted only to refuse: see below | + + +## Where does it refuse to run? + + +A REPL is interactive, so `--yes` and `NOORM_YES` cannot make it run unattended. -## Environment +| Condition | Behavior | +|-----------|----------| +| `--yes`, `-y`, or `NOORM_YES=1` | `Error: noorm sql repl is interactive only.` with `noorm sql query "SELECT 1"` and `noorm sql query --file query.sql` as the headless routes. Exit 2 | +| No terminal on stdin | `Error: noorm sql repl requires an interactive terminal.`, exit 2 | +| `--config` names a config that does not exist | `Error: Config not found: <name>`, exit 2 | +| State cannot be decrypted | `Error: <message>`, exit 1 | -Interactive TTY required. Fails with exit code 2 (usage error) in CI or piped stdin. +Use [`noorm sql query`](/cli/sql) for one-shot SQL in scripts and CI. History from the terminal is what [`noorm sql history`](/cli/sql#sql-history) reads. -A REPL is interactive by definition, so `--yes` / `NOORM_YES` cannot make it -succeed. They produce a redirect hint instead, pointing at `noorm sql query` -for one-shot SQL and `noorm sql query --file` for files. See -[Non-interactive operation](../guide/automation/non-interactive.md). +## Next -## Example - noorm sql repl - noorm sql repl --config dev +- [`noorm sql`](/cli/sql): one-shot queries and history +- [The TUI](/tui): keys for every screen diff --git a/docs/cli/sql.md b/docs/cli/sql.md index fe0060a7..fd769ba0 100644 --- a/docs/cli/sql.md +++ b/docs/cli/sql.md @@ -1,77 +1,170 @@ # noorm sql -Execute SQL statements and manage query history from the command line. +Run a SQL statement against a config from the command line, and manage the saved query history. -The `sql` namespace groups four subcommands: +```bash +noorm sql query "SELECT party_id, order_no FROM sales_order LIMIT 2" +``` -| Subcommand | Purpose | -|------------|---------| -| `query` | Run a SQL statement against the active config | -| `history` | Show recent queries (with rows and timings) | -| `clear` | Clear the saved query history | -| `repl` | Launch the TUI directly on the SQL Terminal screen | +``` +Columns: party_id, order_no +{"party_id":1,"order_no":1} +{"party_id":1,"order_no":2} +2 rows returned (3ms) +``` +| Subcommand | Does | +|------------|------| +| `query` | Runs one SQL statement, or a file of them, against the active config | +| `history` | Lists recent queries from the SQL terminal | +| `clear` | Deletes saved history | +| `repl` | Opens the TUI on the SQL terminal: [`noorm sql repl`](/cli/sql-repl) | -## Running a single query +The role your channel holds on the config decides which statements run. The statement is classified as read, write, or DDL; see [access roles](/guide/environments/configs#access-roles). -### Canonical form +## Can I skip the word `query`? -The explicit form is always correct and is what scripts and CI pipelines should use: - noorm sql query "SELECT 1" - noorm sql query "SELECT id, name FROM users LIMIT 10" - noorm sql query --json "SELECT id, name FROM users" - noorm sql query -f reports/monthly.sql +Yes, when the first word after `sql` that is not a flag starts with a SQL verb. noorm inserts `query` for you: +```bash +noorm sql "SELECT 1" # runs: noorm sql query "SELECT 1" +noorm sql --json "SELECT 1 AS one" # runs: noorm sql query --json "SELECT 1 AS one" +``` -### Bare form (convenience) +That word picks one of three outcomes: -The shorter `noorm sql "<SQL>"` form is also supported. Internally, the CLI inspects argv before handing off to citty — if the first non-flag token after `sql` looks like a SQL statement (it starts with one of `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `WITH`, `CREATE`, `DROP`, `ALTER`, `TRUNCATE`, `EXEC`, `EXECUTE`, `CALL`, `SHOW`, `EXPLAIN`, `GRANT`, `REVOKE`, `COMMENT`, `PRAGMA`, `VACUUM`, `ANALYZE`, `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT`, or `USE`), the CLI inserts the `query` subcommand for you: +```mermaid +flowchart TD + A["noorm sql ..."] --> B{"first token after sql<br/>that is not a flag"} + B -->|"query, history,<br/>clear, repl"| C["that subcommand runs"] + B -->|"starts with a SQL verb"| D["query is inserted"] + B -->|"anything else"| E["Unknown command, exit 1"] +``` - noorm sql "SELECT 1" # → noorm sql query "SELECT 1" - noorm sql --json "SELECT 1 AS one" # → noorm sql query --json "SELECT 1 AS one" +The rewrite is positional and never touches the SQL or the flags. A SQL verb is one of `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `WITH`, `CREATE`, `DROP`, `ALTER`, `TRUNCATE`, `EXEC`, `EXECUTE`, `CALL`, `SHOW`, `EXPLAIN`, `GRANT`, `REVOKE`, `COMMENT`, `PRAGMA`, `VACUUM`, `ANALYZE`, `BEGIN`, `COMMIT`, `ROLLBACK`, `SAVEPOINT`, or `USE`, in any case. -The rewrite is purely positional — it never modifies the actual SQL string or flags. The real subcommands (`query`, `history`, `clear`, `repl`) are reserved words that never trigger the rewriter. +A flag that takes a value puts the value first, so the rewrite misses: -The rewriter reads the *first non-flag token* after `sql`, so a flag that takes a value puts something else there and no rewrite happens: +| Typed | Result | Write instead | +|-------|--------|---------------| +| `noorm sql -c prod "SELECT 1"` | `Unknown command prod`, exit 1 | `noorm sql query -c prod "SELECT 1"` | +| `noorm sql -f reports/monthly.sql` | `Unknown command reports/monthly.sql`, exit 1 | `noorm sql query -f reports/monthly.sql` | +| `noorm sql --json "SELECT 1"` | Works. `--json` takes no value | | - noorm sql -f reports/monthly.sql # → "Unknown command reports/monthly.sql" - noorm sql -c prod "SELECT 1" # → "Unknown command prod" -Both need the explicit form (`noorm sql query -f …`, `noorm sql query -c prod "…"`). A leading `--json`, which takes no value, is fine: `noorm sql --json "SELECT 1"` still rewrites. +## sql query -### Flags (on `query`) +``` +noorm sql query [<sql>] [--file <path>] [--config <name>] [--json] +``` -- `--config <name>` / `-c` — switch the active config for this invocation -- `--json` — print the result as JSON instead of human output -- `--file <path>` / `-f` — read SQL from a file +| Flag | Effect | +|------|--------| +| `<sql>` | The statement. Optional when `--file` is given | +| `--file <path>`, `-f <path>` | Read the SQL from a file. It wins over `<sql>` | +| `--config <name>`, `-c <name>` | Run against this config instead of the active one | +| `--json` | Print `{"success":true,"columns":[…],"rows":[…],"durationMs":…}` on stdout | -> `--json` is per-subcommand — place it after `query` (`noorm sql query "…" --json`), -> not before `sql`. See [CLI flag conventions](./flags.md) for the full rule. +Human output is `Columns:`, one JSON line per row, then `N row(s) returned (Xms)`. A statement that changes rows prints `Rows affected: N` instead. `--json` is a per-subcommand flag: write it after `query`, not before `sql`. The [flag rules](/cli/flags) explain why. +```bash +noorm sql query -f reports/monthly.sql --json +``` -## Sibling commands +| Condition | Behavior | +|-----------|----------| +| No `<sql>` and no `--file` | `No query provided. Usage: noorm sql "SELECT ..."`, exit 2 | +| `--file` cannot be read | `Failed to read SQL file: <path>: <reason>`, exit 2 | +| The database rejects the statement | `Query failed: <message>`, exit 1 | +| Your role does not allow that class of statement | The policy message, exit 1 | +| Cannot connect | `Failed to connect: <message>`, exit 1 | -For interactive use and history management: +`sql query` writes nothing to history. It is the headless path, and persisting query text and returned rows on a build agent is an exposure nobody asked for. -- [`noorm sql repl`](./sql-repl.md) — launch the TUI on the SQL terminal screen -- `noorm sql history` — list recent queries (`-n`/`--limit`, default 50). Only the interactive SQL terminal records history; `sql query` deliberately writes none. -- `noorm sql clear --yes` — wipe the saved history (`--older-than <months>` to keep recent entries). Without `--yes` it prints what it would clear and exits without clearing. +## sql history -## Examples - # One-off query against the active config - noorm sql query "SELECT count(*) FROM users" +``` +noorm sql history [--limit <n>] [--config <name>] [--json] +``` - # Same thing, bare form - noorm sql "SELECT count(*) FROM users" +| Flag | Effect | +|------|--------| +| `--limit <n>`, `-n <n>` | Entries to show. Default 50 | +| `--config <name>`, `-c <name>` | History of this config | +| `--json` | `{"configName":…,"entries":[{id, query, executedAt, durationMs, success, errorMessage, rowCount}]}` | - # Run a SQL file (CI-friendly) - noorm sql query -f reports/monthly.sql --json +```bash +noorm sql history -n 2 +``` - # Pick a different config - noorm sql query -c prod "SELECT count(*) FROM orders" +``` +SQL history for 'prod' (2 entries): + + ✓ Oct 21, 01:41:59 12ms · 2 row(s) + SELECT party_id, order_no FROM sales_order LIMIT 2 + + ✗ Oct 21, 01:40:12 4ms · relation "invoice_line" does not exist + SELECT * FROM invoice_line +``` + +Only the interactive SQL terminal (`noorm sql repl`, or `Shift+Q` in the TUI) records history. Query text longer than 80 characters is cut with `…`. History needs a config name but no connection. The name comes from `--config`, then `NOORM_CONFIG`, then the active config. + +| Condition | Behavior | +|-----------|----------| +| `--limit` is not a positive integer | `Invalid limit: <value>. Must be a positive integer.`, exit 2 | +| No config named and none active | `No config specified and no active config set. Use --config or run "noorm config use <name>".`, exit 2 | +| Nothing recorded | `No SQL history found for config 'prod'.`, exit 0 | + + +## sql clear + + +``` +noorm sql clear [--older-than <months>] [--yes] [--config <name>] [--json] +``` + +| Flag | Effect | +|------|--------| +| `--older-than <months>` | Delete only entries older than this many months | +| `--yes`, `-y` | Delete without the preview step | +| `--config <name>`, `-c <name>` | History of this config | +| `--json` | Print `{configName, entriesRemoved, filesRemoved}`. Also skips the preview step | + +```bash +noorm sql clear --older-than 3 --yes +``` + +``` +Cleared 12 entries older than 3 month(s) (4 result files removed). +``` + +| Condition | Behavior | +|-----------|----------| +| Neither `--yes` nor `--json` | Prints `This will clear all entries from SQL history for config 'prod'.` and `Pass --yes to skip this confirmation.`, deletes nothing, exit 0 | +| `--json` without `--yes` | Deletes at once | +| `--older-than` is not a positive integer | `Invalid --older-than value: <value>. Must be a positive integer (months).`, exit 2 | +| No config named and none active | Same message and exit 2 as `history` | + + +## Exit codes + + +| Code | Meaning | Remedy | +|------|---------|--------| +| 0 | Statement ran, or history was listed or cleared | None | +| 1 | Connection failed, the database rejected the statement, role denied, or `Unknown command` | Read the message. Spell out `query` when a flag precedes the SQL | +| 2 | No SQL given, unreadable `--file`, a bad `--limit` or `--older-than`, or no config to read history from | Fix the argument, or pass `--config` | + + +## Next + + +- [`noorm sql repl`](/cli/sql-repl): the interactive SQL terminal +- [The SQL terminal](/guide/database/terminal): history, result browsing, and cancel +- [CLI reference](/headless#sql): every flag in one place diff --git a/docs/design/site-running-example.md b/docs/design/site-running-example.md new file mode 100644 index 00000000..9741ac88 --- /dev/null +++ b/docs/design/site-running-example.md @@ -0,0 +1,438 @@ +# noorm.dev: one running example, less prose + + +This is an audit of the published docs site (`docs/`, deployed to noorm.dev) and a proposal to rebuild it around one example project. The audit covered all 48 user-facing pages (14,339 lines). The contributor pages under `docs/dev/` were read only where the user sidebar links into them. + +The proposal has three parts: + +- one example schema that follows the modeling rules the site teaches +- one source for every snippet, so examples cannot drift +- one page shape that puts the command and its output on the first screen + + +## Problem + + +A developer arriving at noorm.dev hits these problems in roughly this order. + + +### ignatius has no obvious download + + +| Where a reader looks | What they find | +|---|---| +| Hero buttons | "Get started" (noorm install) and "View on GitHub". Nothing for ignatius. | +| Top nav | "Modeling", which opens a narrative overview. None of its 8 "On this page" sections is an install step. | +| Sidebar | "Installation" under "Information Modeling", a group that starts collapsed. | +| Landing page | One install line, mid-page, in "Plan it before you build it". | +| Install URL | `https://raw.githubusercontent.com/noormdev/ignatius/main/install.sh`. noorm gets `noorm.dev/install.sh`; `noorm.dev/ignatius.sh` returns 404. | +| npm | ignatius is not published (`ignatius/package.json` is `"private": true`). | + + +### The modeling pages contradict each other and the rules they teach + + +The site argues for inherited keys, and its own examples break the rules that make inherited keys work. + +| Page | Example | Problem | +|---|---|---| +| `guide/relational-design.md:32,42` | `todos PRIMARY KEY (user_id, created_at)`, `todo_items (user_id, created_at, item_index)` | A timestamp is a key component. The per-parent sequence (`todo_no`) is the rule. | +| `models/todo-list/` (embedded in that page) | Same keys | The diagram readers see repeats the defect. | +| `modeling/entities.md` | `Party → SalesInvoice → SI_Line` with `line_seq` | Correct, but a different domain from the page that makes the argument. | +| `getting-started/first-build.md` | `CREATE TABLE IF NOT EXISTS users (id SERIAL ...)`, `posts (id SERIAL, user_id REFERENCES users(id))` | `IF NOT EXISTS` in `sql/`, surrogate key on a child table, unnamed FK. | +| `guide/changes/overview.md:147` | `verification_tokens (id SERIAL ...)` | Surrogate key on a child of `users`. | +| `guide/database/explore.md:248-254` | `posts_user_id_fkey`, `comments_post_id_fkey` | Default FK names instead of predicate names. | +| `examples/todo-db/sql/00_tables/03_task.sql:26`, `04_association.sql:20` | `todo PRIMARY KEY (user_id, category_id, created_at)`, FK `todo_has_tag` | Timestamp and a reclassifiable column in the key, ORM-verb FK name. | +| `tapes/demo-project/sql/01_tables/003_task.sql:9` | `task (user_id, created_at, task_index)` | Source of every TUI screenshot. | + +`modeling/entities.md` adds a second kind of confusion. It opens on IDEF1X theory and an optional skill, and reaches the file format in its second section. It ends in viewer internals (the lineage spotlight's subset test, classification precedence) that belong in the ignatius repo's reference docs. Nothing on the site shows how an entity file becomes a `CREATE TABLE`, which is the handoff the whole modeling section exists to support. + + +### At least six example domains, none shared + + +| Domain | Where | +|---|---| +| Blog: `users`, `posts`, `comments` | First Build, Building Your SDK, 9 guide pages, `headless.md`, `cli/sql.md`. `users` alone has 4+ incompatible definitions. | +| Todo: `users/todos/todo_items` | `relational-design.md`, `models/todo-list` | +| Todo variant: `app_user/project/task` | TUI screenshots (`tapes/demo-project`) | +| Order-to-cash: `Party/SalesInvoice/SI_Line` | `modeling/entities.md` | +| LLM memory: `Memory`, `Sentinels`, `sp_Memory_Create` | ignatius demo video, `cli/run.md`, `reference/sdk.md` | +| Social graph: `posts/photos/tags/groups` | `relational-design.md`, `models/social`, `models/polymorphic` | + +`reference/sdk.md` alone draws from about seven: blog, accounts and invoices, departments, orders, task batches, memory, and abstract helpers (`add_numbers`, `simple_proc`). Eleven guide pages show no schema at all, only connection strings or secret names. + + +### Prose before the command + + +| Page | First command | Notes | +|---|---|---| +| `getting-started/first-build.md` | line 27 | Four TUI detours (config, create, new change, test config). Each has a headless equivalent: `noorm config import`, `noorm db create`, `noorm change add`. | +| `getting-started/building-your-sdk.md` | line 399 (`createContext`) | Steps 1-4 are monorepo scaffolding. Several samples do not compile (`app`, `emailService` undefined). | +| `guide/changes/history.md` | line 113 | | +| `guide/environments/vault.md` | line 91 | Lines 60-84 are X25519/HKDF internals. | +| `guide/database/explore.md` | line 69 | 13 ASCII TUI mockups before the CLI form. | +| `guide/sql-files/templates.md`, `guide/environments/stages.md`, `guide/sql-files/organization.md`, `guide/relational-design.md` | none | No `noorm` command block on the page. | + + +### Navigation + + +- Nine of the ten `cli/*` pages are in no nav or sidebar. `cli/run.md` and `cli/settings-edit.md` have no inbound links at all. +- `headless.md` (1,870 lines) puts its 70 commands at `####`, and `outline.level` is `[2, 3]`, so the right-hand TOC lists group names only. There is no command index. +- `dev/sdk.md` (1,159 lines) is an older copy of the user SDK reference, reachable from the user sidebar as "Contributor Documentation". +- The same content lives in several places: the access-role matrix (configs, teardown, mcp, terminal), the CI bootstrap script (create, non-interactive, troubleshooting), the env-var table (configs, ci, deployment), and dry-run (changes overview, forward-revert). + + +### Correctness bugs found along the way + + +These ship regardless of the redesign. + +| Bug | Evidence | +|---|---| +| `noorm -c prod run build` runs in a directory named `prod` | `src/cli/index.ts:179` reads `-c` before the subcommand as `--cwd`. `guide/environments/configs.md:86,169,189,375` use it as `--config`. | +| `https://noorm.dev/install.sh` verifies nothing | The served copy, `docs/public/install.sh` (116 lines), has no checksum code and ignores `NOORM_INSECURE`. The repo-root `install.sh` (200 lines, the GitHub-mirror URL) verifies against `checksums.txt`. The two copies drifted, so `getting-started/installation.md`'s "every install path verifies the downloaded binary" is false for the primary URL. Fix: deploy the root script at build time and delete the copy. | +| `NOORM_INSTALL_DIR=~/my-tools curl … \| sh` never reaches the installer | A prefix assignment applies to `curl` only. `getting-started/installation.md:38,64` (`NOORM_INSECURE` too). Fix: `curl … \| NOORM_INSTALL_DIR=~/my-tools sh`. | +| SDK install line omits the `kysely` peer | `reference/sdk.md:16-18` vs `packages/sdk/package.json` peers. | +| "Four permissions require confirmation even for admin" | `changes.delete` is a fifth (`src/core/policy/matrix.ts:17`). | +| `secret set` shown without `--stdin`/`--yes`; `init` shown without `--here` | `headless.md` vs `src/cli/secret/set.ts`, `src/cli/init.ts`. | +| Settings link loops | `guide/sql-files/organization.md:351` → `stages.md`, which has no `build.include` content and links back. | + + +## Goals / Non-goals + + +Goals: + +- A reader can install either tool from the first screen of noorm.dev. +- Every code sample on the site comes from one example project, and that project's CI proves the samples run. +- The example follows every modeling rule the site teaches. +- Every page shows its command and real output within the first screen. +- Fewer pages: one home per topic, everything else links to it. + +Non-goals: + +- Visual redesign beyond the landing and Download pages. The landing replaces the stock VitePress hero, so its hero components and their CSS go; the rest of the theme and brand CSS stay. +- New CLI features. A `noorm init --example` scaffold would help, but it is a public contract and a separate decision. +- Rewriting `docs/dev/`, apart from removing `dev/sdk.md`'s duplicate of the user reference. +- Re-running the 1M-row benchmark in `relational-design.md`. + + +## Approaches: which running example + + +| # | Example | Wins | Loses | +|---|---|---|---| +| A | **Order-to-cash**, trimmed from ignatius's `models/key-inherited` | Already follows the rules (`line_seq`, `places / is placed by`). Has a real subtype cluster (Party → Person, Business). Payments that can only settle the same party's invoice show what inherited keys buy, in one FK. Every noorm feature has a natural use. ignatius already ships the `order-to-cash` flow. | New example project to build. Business vocabulary (invoice, allocation) is one notch past a toy. | +| B | Todo, remodeled correctly (`todo_no`, `item_no`) | Smallest. Every reader knows the domain. `examples/todo-db` exists. | Subtypes, stored procedures, transfer, and vault all feel staged in a todo app. `todo-db` needs a full remodel anyway. | +| C | LLM memory (`examples/llm-memory-db-*`) | Exists with 230 tests, matches the MCP pitch. | 38 tables, 140 SQL files. The domain needs explaining before any example lands. | +| D | Blog (`users/posts`), cleaned up | Already the most-used domain. | No subtype story, no procedure story. Every ORM tutorial uses it, so it does nothing to show why noorm is different. | + + +## Recommendation + + +**A, order-to-cash.** It is the only candidate that is already correct, already modeled in ignatius, and large enough to carry every feature without staging. Its central FK is the site's argument in one line. `payment_allocation` carries `party_id` from both parents, so a payment from one customer cannot settle another customer's invoice. The database enforces that without a trigger. + + +### The schema + + +Eleven tables. The quick start uses two, and later pages add tables as the topic needs them. + +The transactional spine. Every child inherits `party_id`, which is what lets one constraint stop cross-customer payments: + +```mermaid +erDiagram + party ||--o{ sales_order : "places" + sales_order ||--|{ order_line : "contains" + product ||--o{ order_line : "is sold via" + party ||--o{ invoice : "owes on" + invoice ||--|{ invoice_line : "contains" + order_line ||--o{ invoice_line : "is billed by" + party ||--o{ payment : "makes" + payment ||--|{ payment_allocation : "is applied as" + invoice_line ||--o{ payment_allocation : "is settled by" +``` + +The party cluster. `party_type` is the discriminator, so a party is exactly one of person or business: + +```mermaid +erDiagram + party_type ||--o{ party : "classifies" + party ||--o| person : "is realized as" + party ||--o| business : "is realized as" +``` + +| Table | Primary key | Rule it demonstrates | +|---|---|---| +| `party` | `party_id` | Surrogate allowed at a root: nothing above it supplies identity. | +| `person`, `business` | `party_id` | Exclusive subtypes, one-to-one with the basetype. | +| `product` | `product_id`, AK `sku` | Root entity with an alternate key named for its rule. | +| `sales_order` | `party_id, order_no` | Per-parent sequence, assigned by `place_order()`. No timestamp in the key. | +| `order_line` | `party_id, order_no, line_no` | Full lineage in the key; the owner is readable without a join. | +| `invoice`, `invoice_line` | `party_id, invoice_no[, line_no]` | Same shape, second spine. | +| `payment` | `party_id, payment_no` | | +| `payment_allocation` | `party_id, payment_no, invoice_no, line_no` | Two identifying parents share `party_id`, which makes cross-party allocation unrepresentable. | + +FKs follow `<table>_<reverse predicate>_<target>`: `sales_order_isplacedby_party`, `order_line_ispartof_sales_order`, `payment_allocation_settles_invoice_line`. `sql/` declares every column and constraint once, inline in `CREATE TABLE`. Every `ALTER`, backfill, and drop lives in `changes/`. + + +### How the example grows across the site + + +| Page | Tables and objects it uses | Feature it shows | +|---|---|---| +| Landing, Quick start | `party`, `sales_order` | `run build`, then a change that adds `order_line` | +| Concepts | same | SQL files are current state, changes are how existing databases catch up | +| SQL files & build order | all tables, `01_tables/` … `04_views/` | Ordering, `build.include` | +| Templates & seeds | `products.yml` → `product` rows, `roles.yml` → `GRANT`s | `.sql.tmpl`, data loaders | +| Changes | `order-status` change: `ALTER TABLE sales_order ADD COLUMN status` + backfill + `SET NOT NULL` | Why history lives in `changes/` and never in `sql/` | +| Configs & stages | `dev`, `staging`, `prod` | Stages, access roles | +| Secrets & vault | `PAYMENT_GATEWAY_KEY`, read in a `.sql.tmpl` as `$.secrets.PAYMENT_GATEWAY_KEY` | Sensitive values stay out of committed SQL; the template fills them in when the SQL runs | +| Explore, SQL terminal | the built schema | Real `noorm db explore --json` output | +| Transfer | staging → qa: `party` → `product` → `sales_order` → `order_line` | FK-ordered copy | +| SDK | `ctx.proc('place_order', …)` with a TVP of lines, `ctx.func('party_balance', …)` | Typed procs, functions, TVPs | +| Testing | `requireTest`, truncate between tests | Test guards | +| CI | the example's own workflow job | Headless build + change + tests | +| MCP | an agent with `viewer` reads `sales_order` | Per-channel roles | +| Modeling | the same 11 entities plus the `order-to-cash` flow | IDEF1X, DFDs, model → SQL | +| Relational design | `payment_allocation` | The case for inherited keys, with this schema | + + +### One source for every snippet + + +The example lives in `examples/order-to-cash/` and holds both halves: the ignatius model and the noorm project. Docs import files from it with VitePress's `<<<` syntax, which resolves paths relative to the page with no restriction to `srcDir` (checked in VitePress 2.0.0-alpha.15's snippet plugin). A sample on the site is therefore a file CI has run. + +Each artifact the site shows is built from the example project, so a schema change reaches every page on the next deploy: + +```mermaid +flowchart TD + EX["examples/order-to-cash/"] --> CI["CI: build, change ff, tests"] + EX --> SNIP["docs pages via snippet import"] + EX --> EXP["ignatius export"] + EXP --> HTML["public/models/order-to-cash.html"] + EX --> TAPE["tapes/*.tape"] + TAPE --> GIF["public/image/*.gif"] +``` + +``` +examples/order-to-cash/ + model/ ignatius root (ignatius.yml, data/, flows/) + sql/ + 01_tables/ + 02_views/ + 03_functions/ + 09_seeds/ products.yml, roles.yml, *.sql.tmpl + changes/ + scripts/quickstart.sh the landing quick start, run in CI + tests/ SDK integration tests +``` + +Until checkpoint 3 creates the example project, the model lives at `docs/models/order-to-cash/` beside the site's other models, and moves into `examples/order-to-cash/model/` with it. + +`examples/todo-db` stays as an SDK test target and drops out of the docs. `docs/models/todo-list` is replaced by the order-to-cash export. `tapes/demo-project` is replaced by the example project, and the tapes re-record from it. + + +### Before and after: the landing quick start + + +Before (`docs/index.md`): + +```bash +mkdir -p sql/01_tables +echo "CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT);" > sql/01_tables/001_users.sql + +noorm run build +``` + +After, imported from `examples/order-to-cash`. The exact config commands get settled when `scripts/quickstart.sh` runs green in CI: + +```sql +-- sql/01_tables/01_party.sql +CREATE TABLE party ( + party_id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + name text NOT NULL +); + +-- sql/01_tables/02_sales_order.sql +CREATE TABLE sales_order ( + party_id integer NOT NULL, + order_no integer NOT NULL, + CONSTRAINT sales_order_pk PRIMARY KEY (party_id, order_no), + CONSTRAINT sales_order_isplacedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) +); +``` + +```bash +noorm run build # fresh database: runs sql/ in order +noorm change add order-lines # add sql/01_tables/03_order_line.sql, list it in the change +noorm change ff # existing database: runs pending changes +``` + +The reader sees a correct key, a named FK, and both halves of the model (build and change) before the first heading. + + +### Before and after: model to SQL + + +The site never shows this handoff today. A new "From model to SQL" page puts the entity file and the DDL side by side, using a VitePress `code-group`: + +```markdown +--- +entity: sales_order +group: sales +pk: [party_id, order_no] +columns: + party_id: { type: integer } + order_no: { type: integer, desc: "Per-party sequence, assigned by place_order()." } +relationships: + - target: party + on: { party_id: party_id } + predicate: { fwd: places, rev: is placed by } +--- +``` + +| Entity file | SQL | +|---|---| +| `pk:` | `PRIMARY KEY (…)` | +| `relationships[].on` | `FOREIGN KEY (…) REFERENCES …` | +| `predicate.rev` + target | constraint name: `sales_order_isplacedby_party` | +| `type: integer` | the physical type, chosen per dialect in `sql/` | +| `ak:` rule | `CONSTRAINT product_isidentifiedby_sku UNIQUE (sku)` | +| `subtypes:` | one table per member, PK = FK to the basetype | + + +### Site structure + + +48 user pages become 34. Each topic gets one page, and everything that repeated it links there instead. + +``` +Docs + Start + Quick start installation + first-build + Concepts + The example project new: tour of examples/order-to-cash + Schema + SQL files & build order organization + execution + Templates & seeds + Changes overview + forward-revert + history + Environments + Configs & stages configs + stages + Secrets & vault secrets + vault + Identity cli/identity + Data + Explore CLI output first, TUI second + Transfer + Create, reset & teardown create + teardown + SQL terminal terminal + cli/sql + cli/sql-repl + App code + SDK in 5 minutes top of building-your-sdk + Procedures, functions, TVPs + Testing against a real DB + Deploying deployment + Automation + CI ci + non-interactive + Agents (MCP) + Troubleshooting symptom | cause | fix table + Background + Why noorm + Relational design rewritten on order-to-cash +Modeling (ignatius) + Install first page of the section + Model the example serve the order-to-cash model, edit one entity + Entity files format only; internals move to the ignatius repo + Data flows + From model to SQL new + Reverse-engineering + Modeling skill absorbs branding + best practices +Reference + CLI headless + cli/flags, help, init, run, secret, settings-* + SDK API + settings.yml + TUI keys tui.md +Download new: both tools, every platform, both skills +``` + +Top nav becomes `Docs · Modeling · Reference · Download`. "Contributor Documentation" moves from the user sidebar to the footer, and `dev/sdk.md` shrinks to internals that `reference/sdk.md` does not cover. + + +### Page shape + + +Every guide page follows the same order, so a reader learns the layout once: + +``` +# <Task, verb first> "Add a column to a live database" +<one sentence: when you need this> + +<<< the example file or command +<its real output> + +## How it works diagram or table; prose only for reasoning +## Options flag table, linking to Reference +## What bites symptom | cause | fix +Next: <one link> +``` + +Budgets: first code block before line 15, no paragraph over four sentences, internals (crypto, classifiers, driver quirks) go to `docs/dev/`. + +The reference pages get one entry shape. CLI: synopsis, flag table, example, JSON output, exit codes, with a command index at the top and `outline.level` raised to `[2, 4]`. SDK: signature, params table, returns, example, with an API index at the top. + + +### Download page + + +One page, linked from the hero and the nav. + +| | noorm | ignatius | +|---|---|---| +| macOS / Linux | `curl -fsSL https://noorm.dev/install.sh \| sh` | `curl -fsSL https://noorm.dev/ignatius.sh \| sh` | +| npm | `npm i -g @noormdev/cli` | none | +| Windows | `noorm-windows-x64.exe` from releases | `ignatius-windows-x64.exe` from releases | +| Agent skill | `npx skills add noormdev/noorm/skills` | `npx skills add https://github.com/noormdev/ignatius --skill ignatius-modeling` | +| Verify | `noorm --version` | `ignatius --help` | + +`noorm.dev/ignatius.sh` is a two-line file in `docs/public/` that pipes the ignatius repo's `install.sh` into `sh`. It needs no copy of the installer, so it cannot go stale. `IGNATIUS_*` variables set on the outer `sh` (`curl … | IGNATIUS_VERSION=v0.23.0 sh`) reach the inner one. The hero's secondary button changes from "View on GitHub" (already in the nav icons) to "Download". + + +## What will bite + + +| Risk | Cause | Mitigation | +|---|---|---| +| A moved example file blanks a code block on the site | VitePress renders `Code snippet path not found: …` in place of the block and does not fail the build | Grep `.vitepress/dist` for that string in `docs.yml` and fail the job | +| The docs deploy misses an example change | `docs.yml` triggers on `docs/**` only | Add `examples/order-to-cash/**` to its path filter | +| The model drifts from the SQL | ignatius and noorm read different files | CI runs `ignatius validate model/` and a check that every entity has a table | +| ignatius format changes break the example | ignatius releases on its own cadence | Pin `IGNATIUS_VERSION` in CI; bump deliberately | +| `docs/models/key-inherited` style models multiply | ignatius repo keeps its own 24-entity reference model | Out of scope here; the site uses only the order-to-cash model | +| The page shape conflicts with the docs rule | `.claude/rules/documentation.md` says "code follows the explanation that motivates it" | Change the rule to "one-line problem, then the example, then the explanation" in the same PR as the first rewritten page | + + +## Decisions needed + + +1. **Running example.** Order-to-cash (recommended), remodeled todo, LLM memory, or blog. +2. **Tutorial database.** Postgres in Docker for every page (recommended: one dialect keeps procs, TVPs, and transfer real) or SQLite for the quick start only (zero install, but the reader switches dialects at the SDK page). +3. **Relational-design essay.** Rewrite it on order-to-cash and drop `models/social` and `models/polymorphic` (recommended), or keep the social-graph section and its 1M-row benchmark as is. +4. **`examples/todo-db`.** Keep as an SDK test target outside the docs (recommended), or retire it once order-to-cash covers its tests. + + +## Rollout + + +| # | Checkpoint | Depends on | +|---|---|---| +| 1 | Correctness fixes from the bug table above | nothing; ships first | +| 2 | Download page, `noorm.dev/ignatius.sh`, hero button, nav rename | nothing | +| 3 | `examples/order-to-cash`: model, `sql/`, `changes/`, seeds, `quickstart.sh`, tests, CI job, `ignatius export` | decisions 1, 2 | +| 4 | Landing, Quick start, Concepts, Example project page, doc-rule change | 3 | +| 5 | Modeling section: Install first, Model the example, From model to SQL, internals moved out | 3 | +| 6 | Guides rewritten to the page shape, merges and redirects | 4 | +| 7 | CLI and SDK reference restructure, orphan `cli/*` folded in, `dev/sdk.md` trimmed | 6 | +| 8 | Tapes re-recorded from the example, `tapes/demo-project` and `models/todo-list` removed | 3 | diff --git a/docs/dev/change.md b/docs/dev/change.md index 41ff3279..cca682e5 100644 --- a/docs/dev/change.md +++ b/docs/dev/change.md @@ -20,7 +20,7 @@ noorm's change system solves this with versioned folders containing forward and A change is a folder with a specific structure: ``` -changes/ +db/changes/ └── 2025-01-15-add-email-verification/ ├── change/ # Forward changes (required) │ ├── 001_create-tokens-table.sql diff --git a/docs/dev/ci.md b/docs/dev/ci.md index 60c5cf6a..6e300a7d 100644 --- a/docs/dev/ci.md +++ b/docs/dev/ci.md @@ -106,8 +106,8 @@ a partial one needs a human, so the two must never share a code. | Variable | Required | Description | |----------|----------|-------------| -| `NOORM_PATHS_SQL` | No | Schema directory (default: `./sql`) | -| `NOORM_PATHS_CHANGES` | No | Changes directory (default: `./changes`) | +| `NOORM_PATHS_SQL` | No | Schema directory (default: `./db/sql`) | +| `NOORM_PATHS_CHANGES` | No | Changes directory (default: `./db/changes`) | ## GitHub Actions Example (Test CI) diff --git a/docs/dev/headless.md b/docs/dev/headless.md index 67499a55..ea5f736a 100644 --- a/docs/dev/headless.md +++ b/docs/dev/headless.md @@ -163,7 +163,7 @@ string, `success` is derived from it (`partial` is never reported as `true`). Execute a single SQL file. ```bash -noorm run file sql/01_tables/001_users.sql +noorm run file db/sql/01_tables/001_users.sql ``` **JSON output:** @@ -182,7 +182,7 @@ noorm run file sql/01_tables/001_users.sql Execute all SQL files in a directory. ```bash -noorm run dir sql/01_tables/ +noorm run dir db/sql/01_tables/ ``` @@ -660,7 +660,7 @@ src/cli/ ├── run/ ├── secret/ ├── settings/ -├── sql/ +├── db/sql/ └── vault/ ``` diff --git a/docs/dev/runner.md b/docs/dev/runner.md index 36d66048..b45577b3 100644 --- a/docs/dev/runner.md +++ b/docs/dev/runner.md @@ -71,9 +71,9 @@ The `--dry-run` flag renders all SQL files and writes them to a local `tmp/` dir ``` Source: Dry run output: -sql/02_views/001_my_view.sql → tmp/sql/02_views/001_my_view.sql -sql/03_seeds/001_users.sql.tmpl → tmp/sql/03_seeds/001_users.sql -sql/04_auth/002_permissions.sql.tmpl → tmp/sql/04_auth/002_permissions.sql +db/sql/02_views/001_my_view.sql → tmp/db/sql/02_views/001_my_view.sql +db/sql/03_seeds/001_users.sql.tmpl → tmp/db/sql/03_seeds/001_users.sql +db/sql/04_auth/002_permissions.sql.tmpl → tmp/db/sql/04_auth/002_permissions.sql ``` Templates are fully rendered with the current config context. The `.tmpl` extension is stripped from output files. @@ -115,7 +115,7 @@ noorm run build --preview --output build.sql Files ending in `.sql.tmpl` are processed through the template engine before execution: ```sql --- sql/03_seeds/001_users.sql.tmpl +-- db/sql/03_seeds/001_users.sql.tmpl {% for (const user of $.users) { %} INSERT INTO users (email, name) VALUES ({%~ $.quote(user.email) %}, {%~ $.quote(user.name) %}); {% } %} @@ -165,16 +165,16 @@ console.log(`Ran ${result.filesRun} files in ${result.durationMs}ms`) console.log(`Skipped ${result.filesSkipped} unchanged files`) // Execute a single file -const fileResult = await runFile(context, '/project/sql/001_users.sql') +const fileResult = await runFile(context, '/project/db/sql/001_users.sql') // Execute all files in a directory -const dirResult = await runDir(context, '/project/sql/02_views') +const dirResult = await runDir(context, '/project/db/sql/02_views') // Execute specific files (selective execution) const filesResult = await runFiles(context, [ - '/project/sql/01_tables/001_users.sql', - '/project/sql/01_tables/002_posts.sql', - '/project/sql/02_views/001_active_users.sql', + '/project/db/sql/01_tables/001_users.sql', + '/project/db/sql/01_tables/002_posts.sql', + '/project/db/sql/02_views/001_active_users.sql', ]) // Dry run - render to tmp/ without executing @@ -184,13 +184,13 @@ const dryResult = await runBuild(context, '/project/sql', { // Preview rendered SQL const previews = await preview(context, [ - '/project/sql/seed.sql.tmpl', + '/project/db/sql/seed.sql.tmpl', ]) console.log(previews[0].renderedSql) // Preview to a file (optional 3rd parameter) const previewsToFile = await preview(context, [ - '/project/sql/seed.sql.tmpl', + '/project/db/sql/seed.sql.tmpl', ], '/project/output.sql') ``` @@ -248,7 +248,7 @@ When `context.dialect === 'mssql'`, the runner splits SQL content on `^\s*GO\s*$ ```typescript -const result = await ctx.noorm.run.file('sql/procs/checkout.sql'); +const result = await ctx.noorm.run.file('db/sql/procs/checkout.sql'); if (result.status === 'failed') { @@ -402,18 +402,18 @@ Filter files by include/exclude path patterns: import { filterFilesByPaths } from './core/shared' const files = [ - '/project/sql/01_tables/001_users.sql', - '/project/sql/02_views/001_active.sql', - '/project/sql/archive/old.sql', + '/project/db/sql/01_tables/001_users.sql', + '/project/db/sql/02_views/001_active.sql', + '/project/db/sql/archive/old.sql', ] const filtered = filterFilesByPaths( files, '/project', - ['sql/01_tables', 'sql/02_views'], // include - ['sql/archive'] // exclude + ['db/sql/01_tables', 'db/sql/02_views'], // include + ['db/sql/archive'] // exclude ) -// ['/project/sql/01_tables/001_users.sql', '/project/sql/02_views/001_active.sql'] +// ['/project/db/sql/01_tables/001_users.sql', '/project/db/sql/02_views/001_active.sql'] ``` This utility is used internally by build operations when applying settings rules. The CLI passes the SQL directory (`paths.sql`) as the base directory, so settings patterns like `01_tables` resolve relative to the SQL path, not the project root. Exclude patterns take precedence when both match. @@ -427,8 +427,8 @@ The `executeFiles` function provides low-level file execution with full tracking import { executeFiles, type FileInput } from './core/runner' const files: FileInput[] = [ - { path: '/project/sql/001_users.sql', type: 'sql' }, - { path: '/project/sql/002_posts.sql', type: 'sql' }, + { path: '/project/db/sql/001_users.sql', type: 'sql' }, + { path: '/project/db/sql/002_posts.sql', type: 'sql' }, ] // Four arguments: run options and execution options are separate objects diff --git a/docs/dev/sdk.md b/docs/dev/sdk.md index 880554f6..89719e0f 100644 --- a/docs/dev/sdk.md +++ b/docs/dev/sdk.md @@ -449,7 +449,7 @@ await ctx.noorm.run.dir('seeds/') Discover SQL files in a directory. Defaults to the configured SQL directory. ```typescript -const files = await ctx.noorm.run.discover('sql/') +const files = await ctx.noorm.run.discover('db/sql/') ``` ##### `run.preview(filepaths, output?)` @@ -457,7 +457,7 @@ const files = await ctx.noorm.run.discover('sql/') Preview SQL files — render templates without executing. ```typescript -const results = await ctx.noorm.run.preview(['sql/001.sql', 'sql/002.sql']) +const results = await ctx.noorm.run.preview(['db/sql/001.sql', 'db/sql/002.sql']) ``` @@ -693,7 +693,7 @@ const { released, holder } = await ctx.noorm.lock.forceRelease() Render a template file without executing. ```typescript -const result = await ctx.noorm.templates.render('sql/001_users.sql.tmpl') +const result = await ctx.noorm.templates.render('db/sql/001_users.sql.tmpl') console.log(result.sql) ``` @@ -801,7 +801,7 @@ const result = await ctx.noorm.vault.copy(destConfig, ['API_KEY'], privateKey) Compute SHA-256 checksum for a file. ```typescript -const checksum = await ctx.noorm.utils.checksum('sql/001_users.sql') +const checksum = await ctx.noorm.utils.checksum('db/sql/001_users.sql') ``` ##### `utils.testConnection()` diff --git a/docs/dev/settings.md b/docs/dev/settings.md index 4257f9ca..b08c383c 100644 --- a/docs/dev/settings.md +++ b/docs/dev/settings.md @@ -164,8 +164,11 @@ Defaults: | Path | Default | |------|---------| -| `sql` | `./sql` | -| `changes` | `./changes` | +| `sql` | `./db/sql` | +| `changes` | `./db/changes` | + +These apply when `settings.yml` has no `paths` key. `noorm init` writes them into the file +explicitly. A project laid out elsewhere, such as `./sql`, sets `paths` to say so. ```typescript const paths = settings.getPaths() @@ -716,7 +719,7 @@ import { DEFAULT_SETTINGS, createDefaultSettings } from './core/settings' console.log(DEFAULT_SETTINGS) // { // build: { include: [], exclude: [] }, // empty include = all files under paths.sql -// paths: { sql: './sql', changes: './changes' }, +// paths: { sql: './db/sql', changes: './db/changes' }, // rules: [], // stages: {}, // strict: { enabled: false, stages: [] }, diff --git a/docs/dev/template.md b/docs/dev/template.md index 75dd4ae2..37f6d2b3 100644 --- a/docs/dev/template.md +++ b/docs/dev/template.md @@ -69,7 +69,7 @@ INSERT INTO users (id) VALUES ('{%~ $.uuid() %}'); Data files in the same directory as the template are automatically loaded into the context: ``` -sql/users/ +db/sql/users/ ├── 001_seed.sql.tmpl # Template ├── users.json5 # → $.users ├── roles.yml # → $.roles @@ -132,7 +132,7 @@ strings work', Helper files named `$helpers.ts` or `$helpers.js` are automatically loaded. They're inherited up the directory tree - child helpers override parent helpers. ``` -sql/ +db/sql/ ├── $helpers.ts # Project-wide helpers ├── users/ │ ├── $helpers.ts # Overrides/extends parent @@ -152,7 +152,7 @@ sql/ ### Writing Helpers ```typescript -// sql/$helpers.ts +// db/sql/$helpers.ts export function padId(id: number, length = 6): string { return String(id).padStart(length, '0') @@ -234,7 +234,7 @@ secret, a bad lookup), never an intentional value. `null` is unaffected and stil Define users, roles, and permissions in YAML - generate all the grants from config: ```yaml -# sql/Auth/auth.yml +# db/sql/Auth/auth.yml users: - username: __web_app password: $DB_WEB_APP_PASSWORD @@ -267,7 +267,7 @@ permissions: ``` ```sql --- sql/Auth/02-Permissions.sql.tmpl +-- db/sql/Auth/02-Permissions.sql.tmpl {% for (const role in $.auth.permissions) { %} {% const perms = $.auth.permissions[role] %} @@ -290,7 +290,7 @@ GRANT EXECUTE ON [{%~ proc %}] TO [{%~ role %}]; Manage cron jobs from a config file - no more clicking through SSMS: ```yaml -# sql/Crons/cron.yml +# db/sql/Crons/cron.yml Jobs: Sync_Entrata: Description: 'Sync property data from Entrata API' @@ -321,7 +321,7 @@ Schedules: ``` ```sql --- sql/Crons/Crons.sql.tmpl +-- db/sql/Crons/Crons.sql.tmpl -- Create jobs {% for (const job in $.cron.Jobs) { %} @@ -353,7 +353,7 @@ EXEC msdb.dbo.sp_attach_schedule ### Seeding from CSV ```sql --- sql/seed/users.sql.tmpl +-- db/sql/seed/users.sql.tmpl INSERT INTO users (email, name, role) VALUES {% $.users.forEach((user, i) => { %} ({%~ $.quote(user.email) %}, {%~ $.quote(user.name) %}, {%~ $.quote(user.role) %}){% if (i < $.users.length - 1) { %},{% } %} @@ -372,7 +372,7 @@ bob@example.com,Bob,user ### Using Secrets ```sql --- sql/config/api_keys.sql.tmpl +-- db/sql/config/api_keys.sql.tmpl INSERT INTO config (key, value) VALUES ('stripe_key', '{%~ $.secrets.STRIPE_KEY %}'), ('api_url', '{%~ $.env.API_URL || 'https://api.example.com' %}'); @@ -398,7 +398,7 @@ INSERT INTO config (key, value) VALUES ### Dialect-Specific SQL ```sql --- sql/indexes/create.sql.tmpl +-- db/sql/indexes/create.sql.tmpl CREATE INDEX idx_users_email ON users(email); {% if ($.config.connection.dialect === 'postgres') { %} diff --git a/docs/download.md b/docs/download.md new file mode 100644 index 00000000..6d6eafb4 --- /dev/null +++ b/docs/download.md @@ -0,0 +1,109 @@ +--- +title: Download +description: Install noorm and ignatius on macOS, Linux, or Windows, plus the agent skills and the SDK. +sidebar: false +aside: false +pageClass: page-download +prev: false +next: false +--- + +# Download + +noorm builds and changes databases. ignatius models them before they exist. Each is one binary with no runtime to install. + +<div class="dl-grid"> +<div class="dl-tool"> + +## noorm + +<p class="dl-what">CLI, terminal UI, and MCP server. Builds schemas from SQL files, applies changes, and manages configs, secrets, and data.</p> + +::: code-group + +```bash [macOS / Linux] +curl -fsSL https://noorm.dev/install.sh | sh +``` + +```bash [npm] +npm install -g @noormdev/cli +``` + +```powershell [Windows] +npm install -g @noormdev/cli +# or noorm-windows-x64.exe from the GitHub releases page +``` + +::: + +<p class="dl-label">Agent skill</p> + +```bash +npx skills add noormdev/noorm/skills +``` + +<p class="dl-label">SDK, for application code</p> + +```bash +npm install @noormdev/sdk kysely pg +``` + +<p class="dl-small">Swap <code>pg</code> for <code>mysql2</code>, <code>better-sqlite3</code>, or <code>tedious tarn</code>. Verify with <code>noorm --version</code>; update with <code>noorm update</code>.</p> + +</div> +<div class="dl-tool"> + +## ignatius + +<p class="dl-what">Data modeler. Turns a folder of markdown into an IDEF1X entity diagram, a data dictionary, and data flow diagrams.</p> + +::: code-group + +```bash [macOS / Linux] +curl -fsSL https://noorm.dev/ignatius.sh | sh +``` + +```powershell [Windows] +# ignatius-windows-x64.exe from the GitHub releases page +``` + +```bash [From source] +git clone https://github.com/noormdev/ignatius +cd ignatius && bun install && bun run build:cli +``` + +::: + +<p class="dl-label">Agent skill</p> + +```bash +npx skills add https://github.com/noormdev/ignatius --skill ignatius-modeling +``` + +<p class="dl-label">Try it on a model</p> + +```bash +ignatius serve path/to/model -o +``` + +<p class="dl-small">Verify with <code>ignatius version</code>; update with <code>ignatius update</code>. See <a href="/modeling/installation">ignatius commands</a> for serve, validate, export, and the rest.</p> + +</div> +</div> + +## Both at once + +```bash +curl -fsSL https://noorm.dev/install.sh | sh && curl -fsSL https://noorm.dev/ignatius.sh | sh +``` + +## Install options + +| Need | noorm | ignatius | +|---|---|---| +| Install somewhere else | `curl … \| NOORM_INSTALL_DIR=~/bin sh` | `curl … \| IGNATIUS_INSTALL_DIR=~/bin sh` | +| Pin a version | `npm install -g @noormdev/cli@<version>` | `curl … \| IGNATIUS_VERSION=v0.23.0 sh` | +| `noorm.dev` is blocked | `raw.githubusercontent.com/noormdev/noorm/master/install.sh` | `raw.githubusercontent.com/noormdev/ignatius/main/install.sh` | +| Releases and checksums | [noorm releases](https://github.com/noormdev/noorm/releases) | [ignatius releases](https://github.com/noormdev/ignatius/releases/latest) | + +The variable goes on `sh`, after the pipe. Set before `curl`, it reaches only `curl` and the installer never sees it. diff --git a/docs/getting-started/building-your-sdk.md b/docs/getting-started/building-your-sdk.md index 147aa04e..392c5e4b 100644 --- a/docs/getting-started/building-your-sdk.md +++ b/docs/getting-started/building-your-sdk.md @@ -1,273 +1,94 @@ -# Building Your SDK +# Building your SDK -You've built your schema. Now you want to use it. This guide shows how to wrap noorm in a TypeScript SDK that provides type-safe database access across your workers, servers, and CLI tools. +Wrap `createContext` in one workspace package, so every app and test queries the database through the same typed client. +```typescript +// apps/api/src/orders.ts +import { connect } from '@acme/db'; -## What You'll Build - -A dedicated database package in your monorepo: +const db = await connect({ config: 'dev' }); -``` -packages/ -└── db/ - ├── package.json - ├── src/ - │ ├── index.ts # Public API - │ ├── client.ts # Database client - │ ├── testing.ts # Test utilities - │ ├── types.ts # Database types - │ └── domains/ # Domain classes - │ ├── users.ts - │ └── posts.ts - └── tests/ # Integration tests +const orders = await db.orders.listByParty(2); +// [ { party_id: 2, order_no: 1 }, { party_id: 2, order_no: 2 } ] ``` -This structure gives you: -- **Type safety** - Kysely types mirror your database -- **Testability** - Integration tests use real databases -- **Portability** - Same SDK works in workers, servers, CLI +## How is the package laid out? -## Step 1: Set Up the Monorepo +Apps and tests share one client, and only the test entry point sets `requireTest`. -Create a pnpm workspace: - -```bash -mkdir my-project && cd my-project -pnpm init +```mermaid +flowchart TD + APPS["apps/api<br/>apps/worker"] --> CONNECT["connect()"] + TESTS["tests/orders.test.ts"] --> SETUP["setupTestDb()<br/>requireTest: true"] + SETUP --> CREATE["createClient()"] + CONNECT --> CREATE + CREATE --> CTX["createContext()"] + CTX --> KYSELY["ctx.kysely"] + CTX --> OPS["ctx.noorm"] + KYSELY --> DB[(database)] + OPS --> DB ``` -Create `pnpm-workspace.yaml`: - -```yaml -packages: - - 'packages/*' - - 'apps/*' +``` +packages/db/ +├── src/ +│ ├── index.ts # re-exports connect, createClient, Client, Database +│ ├── client.ts +│ ├── testing.ts +│ ├── types.ts +│ └── domains/orders.ts +└── tests/orders.test.ts ``` - -## Step 2: Create the Database Package +In `packages/db`, install the SDK, Kysely, and your driver ([Installation](/getting-started/installation#install-the-sdk-optional) lists the driver per dialect), plus the test tools: ```bash -mkdir -p packages/db -cd packages/db -pnpm init +pnpm add @noormdev/sdk kysely pg +pnpm add -D typescript vitest ``` -Update `packages/db/package.json`: - -```json -{ - "name": "@my-project/db", - "version": "1.0.0", - "type": "module", - "main": "./dist/index.js", - "types": "./dist/index.d.ts", - "exports": { - ".": "./dist/index.js", - "./testing": "./dist/testing.js" - }, - "scripts": { - "build": "tsc", - "test": "vitest run", - "test:watch": "vitest" - }, - "dependencies": { - "@noormdev/sdk": "*", - "kysely": "^0.28.0", - "pg": "^8.16.0", - "zod": "^3.24.0" - }, - "devDependencies": { - "typescript": "^5.9.0", - "vitest": "^4.0.0" - } -} -``` +## Describe the schema -## Step 3: Define Your Types -Create `src/types.ts` with types that match your schema: +One interface per table, mirroring `db/sql/`. Update an interface in the same commit that changes its table. ```typescript -import { z } from 'zod'; - -/** - * Database type definitions matching SQL schema. - * - * These types are used by Kysely for type-safe queries. - * Keep them in sync with your SQL files. - */ +// packages/db/src/types.ts +import type { Generated } from 'kysely'; -export interface UsersTable { - id: number; +export interface PartyTable { + party_id: Generated<number>; name: string; - email: string | null; - bio: string | null; - created_at: Date; -} - -export interface PostsTable { - id: number; - user_id: number; - title: string; - body: string | null; - created_at: Date; } -export interface RecentPostsView { - id: number; - user_id: number; - title: string; - body: string | null; - created_at: Date; - author_name: string; +export interface SalesOrderTable { + party_id: number; + order_no: number; + placed_at: Generated<Date>; } -/** - * Complete database schema for Kysely. - */ export interface Database { - users: UsersTable; - posts: PostsTable; - recent_posts: RecentPostsView; + party: PartyTable; + sales_order: SalesOrderTable; } - -// Domain types (what comes OUT of the database) -export type User = UsersTable; -export type Post = PostsTable; -export type RecentPost = RecentPostsView; - -// ───────────────────────────────────────────────────────────── -// Input Schemas (what goes INTO the database) -// ───────────────────────────────────────────────────────────── - -/** - * Define inputs as Zod schemas, then infer types from them. - * - * Input types differ from table types—they don't include - * auto-generated fields like `id` or `created_at`. - */ - -export const CreateUserSchema = z.object({ - name: z.string().min(1).max(255), - email: z.string().email().optional(), - bio: z.string().max(1000).optional(), -}); - -export const CreatePostSchema = z.object({ - user_id: z.number().int().positive(), - title: z.string().min(1).max(255), - body: z.string().max(10000).optional(), -}); - -// Infer input types from schemas—no duplication -export type CreateUserInput = z.infer<typeof CreateUserSchema>; -export type CreatePostInput = z.infer<typeof CreatePostSchema>; -``` - -Zod gives you runtime validation that TypeScript can't. Never trust user input—validate it before it touches your database. This protects against malformed data, provides clear error messages, and catches mistakes that even you might make. - -::: tip Type Generation -Kysely provides [kysely-codegen](https://github.com/RobinBlomberg/kysely-codegen) to automatically generate types from your database: - -```bash -pnpm add -D kysely-codegen -npx kysely-codegen --dialect postgres --out-file src/types.ts ``` -This introspects your database and generates accurate TypeScript types. -::: - - -## Step 4: Create Domain Classes - -Create classes for each domain that hold a reference to the database context. - -`src/domains/users.ts`: - -```typescript -import type { Context } from '@noormdev/sdk'; -import type { Database, User, CreateUserInput } from '../types.js'; -import { CreateUserSchema } from '../types.js'; - -export class Users { - - #ctx: Context<Database>; - - constructor(ctx: Context<Database>) { - - this.#ctx = ctx; - - } - - async findById(id: number): Promise<User | null> { - - const result = await this.#ctx.kysely - .selectFrom('users') - .selectAll() - .where('id', '=', id) - .executeTakeFirst(); - - return result ?? null; - - } - - async findByEmail(email: string): Promise<User | null> { - - const result = await this.#ctx.kysely - .selectFrom('users') - .selectAll() - .where('email', '=', email) - .executeTakeFirst(); - - return result ?? null; - - } - - async create(input: CreateUserInput): Promise<User> { - - // Validate input before touching the database - const validated = CreateUserSchema.parse(input); - - const result = await this.#ctx.kysely - .insertInto('users') - .values({ - name: validated.name, - email: validated.email ?? null, - bio: validated.bio ?? null, - }) - .returningAll() - .executeTakeFirstOrThrow(); - - return result; - - } - - async list(): Promise<User[]> { - - return this.#ctx.kysely - .selectFrom('users') - .selectAll() - .orderBy('created_at', 'desc') - .execute(); - } +## Write a domain class -} -``` -`src/domains/posts.ts`: +Each domain holds the context and exposes the queries for its tables: ```typescript +// packages/db/src/domains/orders.ts import type { Context } from '@noormdev/sdk'; -import type { Database, Post, RecentPost, CreatePostInput } from '../types.js'; -import { CreatePostSchema } from '../types.js'; +import type { Database } from '../types.js'; -export class Posts { +export class Orders { #ctx: Context<Database>; @@ -277,41 +98,13 @@ export class Posts { } - async create(input: CreatePostInput): Promise<Post> { - - const validated = CreatePostSchema.parse(input); - - const result = await this.#ctx.kysely - .insertInto('posts') - .values({ - user_id: validated.user_id, - title: validated.title, - body: validated.body ?? null, - }) - .returningAll() - .executeTakeFirstOrThrow(); - - return result; - - } - - async getRecent(): Promise<RecentPost[]> { - - return this.#ctx.kysely - .selectFrom('recent_posts') - .selectAll() - .orderBy('created_at', 'desc') - .execute(); - - } - - async getByUser(userId: number): Promise<Post[]> { + listByParty(partyId: number) { return this.#ctx.kysely - .selectFrom('posts') - .selectAll() - .where('user_id', '=', userId) - .orderBy('created_at', 'desc') + .selectFrom('sales_order') + .select(['party_id', 'order_no']) + .where('party_id', '=', partyId) + .orderBy('order_no') .execute(); } @@ -320,103 +113,72 @@ export class Posts { ``` -## Step 5: Create the Database Client +## Assemble the client -The client assembles all domains and exposes them as properties. -`src/client.ts`: +`createContext` returns an unconnected context. `connect()` creates and connects it: ```typescript -import { createContext, type Context } from '@noormdev/sdk'; +// packages/db/src/client.ts +import { createContext, type Context, type CreateContextOptions } from '@noormdev/sdk'; import type { Database } from './types.js'; -import { Users } from './domains/users.js'; -import { Posts } from './domains/posts.js'; - -export interface ClientOptions { - config?: string; - requireTest?: boolean; - projectRoot?: string; - yes?: boolean; -} +import { Orders } from './domains/orders.js'; export class Client { #ctx: Context<Database>; - readonly users: Users; - readonly posts: Posts; + readonly orders: Orders; constructor(ctx: Context<Database>) { this.#ctx = ctx; - this.users = new Users(ctx); - this.posts = new Posts(ctx); + this.orders = new Orders(ctx); } - async connect(): Promise<void> { + get kysely() { - await this.#ctx.connect(); + return this.#ctx.kysely; } - async disconnect(): Promise<void> { + connect() { - await this.#ctx.disconnect(); + return this.#ctx.connect(); } - /** - * Reset database (teardown + build). - */ - async reset(): Promise<void> { + disconnect() { - await this.#ctx.noorm.db.reset(); + return this.#ctx.disconnect(); } - /** - * Truncate all tables (keep schema). - */ - async truncate(): Promise<void> { + reset() { - await this.#ctx.noorm.db.truncate(); + return this.#ctx.noorm.db.reset(); + + } + + truncate() { + + return this.#ctx.noorm.db.truncate(); } } -/** - * Create a database client. - * - * @example - * const db = await createClient({ config: 'dev' }); - * await db.connect(); - * const user = await db.users.findById(1); - */ -export async function createClient(options: ClientOptions = {}): Promise<Client> { - - const ctx = await createContext<Database>({ - config: options.config, - requireTest: options.requireTest, - projectRoot: options.projectRoot, - yes: options.yes, - }); - - return new Client(ctx); +export async function createClient(options: CreateContextOptions = {}) { + + return new Client(await createContext<Database>(options)); } -/** - * Create a connected client. - * - * @example - * const db = await connect({ config: 'dev' }); - * const user = await db.users.findById(1); - */ -export async function connect(options: ClientOptions = {}): Promise<Client> { +export async function connect(options: CreateContextOptions = {}) { const client = await createClient(options); + await client.connect(); return client; @@ -424,87 +186,28 @@ export async function connect(options: ClientOptions = {}): Promise<Client> { } ``` +`reset()` tears the schema down and rebuilds it from `db/sql/`. `truncate()` empties every table and keeps the schema. -## Step 6: Create the Public API -`src/index.ts`: +## Test against a real database -```typescript -// Client -export { Client, createClient, connect, type ClientOptions } from './client.js'; - -// Types -export type { - Database, - User, - Post, - RecentPost, - CreateUserInput, - CreatePostInput, -} from './types.js'; -``` +The test client refuses any config that is not marked as a test database: -## Step 7: Add Testing Utilities +```typescript +// packages/db/src/testing.ts +import { createClient } from './client.js'; -Create `src/testing.ts` for integration test helpers: +export function createTestClient(projectRoot?: string) { -```typescript -import { createClient, Client } from './client.js'; + return createClient({ config: 'test', requireTest: true, yes: true, projectRoot }); -export interface TestClientOptions { - projectRoot?: string; } -/** - * Create a test database client. - * - * requireTest: true refuses any config that isn't marked as a test database. - * yes: true pre-confirms the operations noorm would otherwise stop on. - * db:truncate needs confirmation even on an admin config, so without it - * every truncate() throws ProtectedConfigError. - * - * @example - * const db = await createTestClient(); - * await db.connect(); - * await db.reset(); - */ -export async function createTestClient( - options: TestClientOptions = {}, -): Promise<Client> { - - return createClient({ - config: 'test', - requireTest: true, - yes: true, - projectRoot: options.projectRoot, - }); +export async function setupTestDb(projectRoot?: string) { -} + const db = await createTestClient(projectRoot); -/** - * Setup hook for test suites. - * - * Returns connected client with clean database. - * - * @example - * describe('Users', () => { - * let db: Client; - * - * beforeAll(async () => { - * db = await setupTestDb(); - * }); - * - * afterAll(async () => { - * await db.disconnect(); - * }); - * }); - */ -export async function setupTestDb( - options: TestClientOptions = {}, -): Promise<Client> { - - const db = await createTestClient(options); await db.connect(); await db.reset(); @@ -513,265 +216,107 @@ export async function setupTestDb( } ``` +Export it as its own entry, so production code never imports it. In `packages/db/package.json`: -## Step 8: Write Integration Tests - -By testing your SDK against a real database, you're testing your entire data layer—tables, views, stored procedures, constraints, and all. If someone changes a column type, renames a table, or breaks a view, your tests will catch it. This is how you detect schema drift before it reaches production. +```json +{ + "type": "module", + "exports": { + ".": { "types": "./dist/index.d.ts", "import": "./dist/index.js" }, + "./testing": { "types": "./dist/testing.d.ts", "import": "./dist/testing.js" } + }, + "scripts": { "build": "tsc", "test": "vitest run" } +} +``` -Create `tests/users.test.ts`: +Then write tests against the real schema: ```typescript -import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; +// packages/db/tests/orders.test.ts +import { afterAll, beforeAll, beforeEach, expect, it } from 'vitest'; import { setupTestDb } from '../src/testing.js'; import type { Client } from '../src/client.js'; -describe('User Queries', () => { - let db: Client; - - beforeAll(async () => { - db = await setupTestDb(); - }); - - afterAll(async () => { - await db.disconnect(); - }); - - beforeEach(async () => { - await db.truncate(); - }); - - describe('create', () => { - it('creates a user with required fields', async () => { - const user = await db.users.create({ name: 'Alice' }); - - expect(user.id).toBeDefined(); - expect(user.name).toBe('Alice'); - expect(user.email).toBeNull(); - expect(user.created_at).toBeInstanceOf(Date); - }); - - it('creates a user with all fields', async () => { - const user = await db.users.create({ - name: 'Bob', - email: 'bob@example.com', - bio: 'Hello world', - }); - - expect(user.name).toBe('Bob'); - expect(user.email).toBe('bob@example.com'); - expect(user.bio).toBe('Hello world'); - }); - }); - - describe('findByEmail', () => { - it('finds existing user', async () => { - await db.users.create({ name: 'Alice', email: 'alice@example.com' }); - - const found = await db.users.findByEmail('alice@example.com'); - - expect(found).not.toBeNull(); - expect(found?.name).toBe('Alice'); - }); - - it('returns null for non-existent email', async () => { - const found = await db.users.findByEmail('nobody@example.com'); - - expect(found).toBeNull(); - }); - }); - - describe('list', () => { - it('returns users in descending order by created_at', async () => { - await db.users.create({ name: 'First' }); - await db.users.create({ name: 'Second' }); - - const list = await db.users.list(); - - expect(list).toHaveLength(2); - expect(list[0].name).toBe('Second'); - expect(list[1].name).toBe('First'); - }); - }); -}); -``` - +let db: Client; -## Step 9: Configure Test Database - -The SDK uses noorm configs. You'll need a `test` config that points to your test database. Add one via the noorm TUI wherever your noorm project lives: - -```bash -noorm ui -``` +beforeAll(async () => { -Press `c` for config, `a` to add: + db = await setupTestDb(); -- **Config Name**: `test` -- **Database**: `my_project_test` -- **Test Database (skipped in production builds)**: checked - -The checkbox is what `requireTest: true` reads. A context created with `requireTest: true` against a config without it throws `RequireTestError` before it ever connects. - - -## Using Your SDK +}); -Now you can use your SDK anywhere in the monorepo. +afterAll(async () => { + await db.disconnect(); -### In a Server +}); -```typescript -// apps/api/src/routes/users.ts -import { connect } from '@my-project/db'; +beforeEach(async () => { -const db = await connect({ config: 'prod' }); + await db.truncate(); -app.get('/users', async (req, res) => { - const list = await db.users.list(); - res.json(list); }); -app.get('/users/:id', async (req, res) => { - const user = await db.users.findById(Number(req.params.id)); - if (!user) { - return res.status(404).json({ error: 'User not found' }); - } - res.json(user); -}); -``` - - -### In a Worker +it('lists a party\'s orders by order_no', async () => { -```typescript -// apps/worker/src/jobs/send-welcome-email.ts -import { connect } from '@my-project/db'; + const { party_id } = await db.kysely + .insertInto('party') + .values({ name: 'Ada Lovelace' }) + .returning('party_id') + .executeTakeFirstOrThrow(); -export async function sendWelcomeEmail(userId: number) { - const db = await connect(); + await db.kysely + .insertInto('sales_order') + .values([{ party_id, order_no: 1 }, { party_id, order_no: 2 }]) + .execute(); - const user = await db.users.findById(userId); - if (!user?.email) { - console.log('User has no email, skipping'); - return; - } + const orders = await db.orders.listByParty(party_id); - await emailService.send({ - to: user.email, - subject: 'Welcome!', - body: `Hi ${user.name}, welcome aboard!`, - }); + expect(orders.map((o) => o.order_no)).toEqual([1, 2]); - await db.disconnect(); -} +}); ``` +Add the `test` config once with `noorm ui`: press `c`, then `a`, name it `test`, and check Test Database. -### In a CLI Tool -```typescript -// apps/cli/src/commands/list-users.ts -import { connect } from '@my-project/db'; +## Where does the connection come from? -export async function listUsersCommand() { - const db = await connect({ config: process.env.DB_CONFIG }); - const list = await db.users.list(); +| Where it runs | Connection source | Call | +|---|---|---| +| Your machine | A stored config in `.noorm/state/state.enc` | `connect({ config: 'dev' })` | +| Tests | A stored config marked as a test database | `createTestClient()` | +| CI | `noorm ci init` writes the `test` config from env vars | `createTestClient()` | +| Production | `NOORM_CONNECTION_*` env vars, no stored config | `connect()` | - console.table(list.map(u => ({ - id: u.id, - name: u.name, - email: u.email ?? '(none)', - }))); +In CI, create the config before the tests run: - await db.disconnect(); -} +```bash +noorm -c packages/db ci init --name test --test +pnpm --filter @acme/db test ``` +Both steps need `NOORM_IDENTITY_PRIVATE_KEY`, `NOORM_IDENTITY_NAME`, and `NOORM_IDENTITY_EMAIL`. `ci init` also needs `NOORM_CONNECTION_*`. [CI](/guide/automation/ci) has the full workflow, and [Deploying](/guide/deployment) covers production. -### In CI/CD - -```yaml -# .github/workflows/test.yml -name: Tests -on: [push] - -jobs: - test: - runs-on: ubuntu-latest - services: - postgres: - image: postgres:15 - env: - POSTGRES_DB: test - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - ports: - - 5432:5432 - - steps: - - uses: actions/checkout@v4 - - uses: pnpm/action-setup@v2 - - uses: actions/setup-node@v4 - with: - node-version: '20' - cache: 'pnpm' - - - run: pnpm install - - # createTestClient asks for a stored config named `test`, and a - # fresh runner has no .noorm/state/state.enc. `ci init` writes one - # from the env vars and marks the config active and isTest. - # -c runs it in packages/db, which is where vitest's cwd will be, - # and createContext resolves state against cwd. - - run: npx noorm -c packages/db ci init --name test - env: - NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} - NOORM_IDENTITY_NAME: "GitHub CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: localhost - NOORM_CONNECTION_DATABASE: test - NOORM_CONNECTION_USER: postgres - NOORM_CONNECTION_PASSWORD: postgres - - - run: pnpm --filter @my-project/db test - env: - NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} - NOORM_IDENTITY_NAME: "GitHub CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" -``` -Generate `NOORM_CI_KEY` once with `noorm ci identity new --name "GitHub CI" --email ci@example.com`, which prints the env block to stdout, then paste it into your provider's secret store. If your templates read vault secrets, use `noorm ci identity enroll` instead so the key also gets vault access. See [CI/CD](/guide/automation/ci) for the full walkthrough. +## What bites -## Final Structure +| Symptom | Cause | Fix | +|---|---|---| +| `RequireTestError: Config "test" does not have isTest: true` | The config is not marked as a test database. `ci init` leaves it unmarked | Check Test Database in `noorm ui`, or run `ci init --test` | +| PostgreSQL: `db.truncate()` fails with `permission denied to set parameter "session_replication_role"` | Truncate turns off foreign-key triggers with a setting only superusers, or roles granted `SET` on it, may change | Use a superuser login for the test database, or [grant the setting](/guide/database/teardown#what-bites) | +| `ProtectedConfigError` from `truncate()` | `db:truncate` asks for confirmation even on an `admin` config, and the SDK cannot ask | Pass `yes: true` to `createContext`, or set `NOORM_YES=1` | +| `Config "dev" not found` | Stored configs resolve against `projectRoot`, which defaults to the process's cwd | Run from the directory that holds `.noorm/`, or pass `projectRoot` | +| `NotConnectedError: Not connected` | `createContext` returns an unconnected context | Call `connect()` first | +| A query fails: column does not exist | The interface in `types.ts` drifted from `db/sql/` | Update the interface with the table | -``` -my-project/ -├── pnpm-workspace.yaml -├── packages/ -│ └── db/ -│ ├── package.json -│ ├── tsconfig.json -│ ├── src/ -│ │ ├── index.ts -│ │ ├── client.ts -│ │ ├── testing.ts -│ │ ├── types.ts -│ │ └── domains/ -│ │ ├── users.ts -│ │ └── posts.ts -│ └── tests/ -│ └── users.test.ts -└── apps/ - ├── api/ # Uses @my-project/db - ├── worker/ # Uses @my-project/db - └── cli/ # Uses @my-project/db -``` +## Next -## What's Next? -- [SDK Reference](/reference/sdk) - Full API documentation -- [Templates](/guide/sql-files/templates) - Dynamic SQL generation +- [SDK reference](/reference/sdk): every method on `ctx` and `ctx.noorm` +- [Deploying](/guide/deployment): env-only connections and one context per process +- [CI](/guide/automation/ci): `ci init` and the workflow shapes +- [Templates](/guide/sql-files/templates): generate SQL from data diff --git a/docs/getting-started/concepts.md b/docs/getting-started/concepts.md index 8f43ec77..ce687ebb 100644 --- a/docs/getting-started/concepts.md +++ b/docs/getting-started/concepts.md @@ -1,358 +1,183 @@ # Concepts -This page explains how noorm thinks about your database. Understanding these concepts makes everything else click. - - -## SQL Files Are Your Source of Truth - -Your SQL files define what your schema **is**. They represent the ideal, current state of your database structure. +A noorm project keeps its schema in `db/`. `db/sql/` holds the schema as it stands today, and `db/changes/` holds the steps that bring databases that already exist up to it. ``` -sql/ -├── 01_tables/ -│ ├── 001_users.sql # CREATE TABLE users... -│ └── 002_posts.sql # CREATE TABLE posts... -└── 02_views/ - └── 001_recent_posts.sql # CREATE VIEW recent_posts... +db/ +├── sql/ +│ ├── 01_tables/ +│ │ ├── 01_party.sql +│ │ └── 02_sales_order.sql +│ └── 02_views/ +│ └── 01_open_orders.sql +└── changes/ + └── 2026-10-20-order-status/ + ├── change/001_order-status.sql + └── revert/001_order-status.sql ``` -Need a fresh database? Run your SQL files. They contain everything needed to build your schema from scratch—no need to replay years of migrations. - -This is different from migration-based tools where the "current state" only exists by running every migration in order. With noorm, your SQL files **are** the current state. - - -## Execution Order Is Alphanumeric +A new database and an old one reach the same schema by different commands: -Everything runs in alphanumeric order. Directories, files within directories, change scripts—all sorted the same way. +```mermaid +flowchart TD + SQL["db/sql/"] -->|noorm run build| NEW[(new database)] + CH["db/changes/"] -->|noorm change ff| OLD[(existing database)] + NEW --> SAME[same schema] + OLD --> SAME +``` -This matters because dependencies have order. Tables must exist before views reference them. Functions must exist before triggers call them. Without numeric prefixes, you get alphabetical order: +Every edit to `db/sql/` comes with the change that brings an already-built database to the same state. -``` -sql/functions/ ← runs first (f < t < v) -sql/tables/ ← runs second -sql/views/ ← runs third -``` -That's broken—views can't reference tables that don't exist yet. Use numeric prefixes: +## SQL files are the current state -``` -sql/01_tables/ ← runs first -sql/02_views/ ← runs second -sql/03_functions/ ← runs third (functions can reference tables/views) -``` -The same rule applies to files within directories: +Each file in `db/sql/` declares one object the way it should exist now. Every column and constraint is declared once, inline in `CREATE TABLE`. A test run or a new developer gets a database with `noorm run build`, and no history is replayed. -``` -01_tables/ -├── 01_users.sql ← runs first -├── 02_posts.sql ← runs second (can reference users) -└── 03_comments.sql ← runs third (can reference posts) -``` +History never goes in `db/sql/`. `DROP`, an `ALTER` of an existing table, an `IF NOT EXISTS` guard, and a backfill all belong in a change. -And to change scripts: +| | `db/sql/` | `db/changes/` | +|---|---|---| +| Holds | the schema today | how existing databases catch up | +| Runs on | new databases | databases that already exist | +| Command | `noorm run build` | `noorm change ff` | +| Undo | none | each change has a `revert/` folder | +| Order | path, alphanumeric | date prefix | -``` -change/ -├── 001_create_table.sql -├── 002_add_indexes.sql -└── 003_seed_data.sql -``` +[Add a column](/guide/changes/add-a-column) walks one edit through both folders. -One rule, everywhere. Look at the filesystem and you know the order. +## Order is the sort order of the path -## Changes Are Your Changelog -Changes document what you did to evolve an existing database. They answer: who changed what, when, and why. +noorm sorts every path alphanumerically: folders, the files inside them, and the scripts inside a change. A view can only reference a table that already exists, so the folders need number prefixes: ``` -changes/ -└── 2024-01-15-add-user-roles/ - ├── change/ - │ └── 001_add_role_column.sql - └── revert/ - └── 001_remove_role_column.sql +db/sql/tables/ runs 2nd db/sql/01_tables/ runs 1st +db/sql/functions/ runs 1st db/sql/02_views/ runs 2nd +db/sql/views/ runs 3rd db/sql/03_functions/ runs 3rd ``` -When you need to add a column to an existing production database, you don't just edit the SQL file—you also write a change that performs the `ALTER TABLE`. - - -## Templates Turn Configuration Into SQL - -Sometimes you need more than static SQL. Templates let you generate SQL from structured data—define your rules in YAML, JSON, or CSV, and let noorm write the repetitive SQL for you. +Without prefixes, `functions/` sorts first and fails on tables that do not exist yet. `build.include` in settings picks which folders take part in a build. It does not reorder them. -Add `.tmpl` to any SQL file to make it a template: -``` -sql/ -├── tables/ -│ └── users.sql # Static SQL -└── security/ - ├── rls.yml # Configuration data - └── rls.sql.tmpl # Template that reads it -``` +## Templates turn data into SQL -Templates can read adjacent data files. Place a `rls.yml` next to `rls.sql.tmpl`, and the data is available as `rls` in your template. -Here's a real example: PostgreSQL Row-Level Security. Without templates, you'd write dozens of similar `CREATE POLICY` statements. With templates, you define policies in YAML: +Add `.tmpl` to a SQL file and noorm renders it before running it. A data file in the same folder becomes a variable named after the file, so `products.yml` is `$.products`: ```yaml -# sql/security/rls.yml -policies: - users: - - name: users_own_data - for: ALL - check: user_id = current_user_id() - - posts: - - name: posts_read_public - for: SELECT - check: is_public = true - - - name: posts_owner_write - for: ALL - check: author_id = current_user_id() - - comments: - - name: comments_read_if_post_visible - for: SELECT - check: | - EXISTS ( - SELECT 1 FROM posts - WHERE posts.id = comments.post_id - AND (posts.is_public OR posts.author_id = current_user_id()) - ) +# db/sql/09_seeds/products.yml +- { sku: HW-DONGLE, name: USB security key, list_price: 49.00 } +- { sku: SW-SEAT, name: "Desktop seat, yearly", list_price: 89.00 } ``` -Then generate all the SQL: - ```sql --- sql/security/rls.sql.tmpl -{% for (const [table, policies] of Object.entries($.rls.policies)) { %} - -ALTER TABLE {%~ table %} ENABLE ROW LEVEL SECURITY; - -{% for (const policy of policies) { %} -DROP POLICY IF EXISTS {%~ policy.name %} ON {%~ table %}; -CREATE POLICY {%~ policy.name %} ON {%~ table %} - FOR {%~ policy.for %} - USING ({%~ policy.check %}); -{% } %} - -{% } %} +-- db/sql/09_seeds/01_products.sql.tmpl +-- {% for (const p of $.products) { %} +INSERT INTO product (sku, name, list_price) +VALUES ({%~ $.quote(p.sku) %}, {%~ $.quote(p.name) %}, {%~ p.list_price %}); +-- {% } %} ``` -This generates correct, complete SQL for every table. Add a new policy? Edit the YAML. The template handles the boilerplate. +A template reads from four sources, and renders to plain SQL that the database runs: -You can also define **helper functions** in `$helpers.ts` files. Helpers are inherited up the directory tree—define utilities at the root, specialize them in subdirectories: - -```typescript -// sql/$helpers.ts -export function grantAll(role: string, tables: string[]) { - return tables - .map(t => `GRANT SELECT, INSERT, UPDATE, DELETE ON ${t} TO ${role};`) - .join('\n'); -} -``` - -```sql --- sql/security/grants.sql.tmpl -{%~ $.grantAll('app_user', ['users', 'posts', 'comments']) %} +```mermaid +flowchart TD + D["products.yml<br/>(same folder)"] --> T["01_products.sql.tmpl"] + H["$helpers.ts<br/>(this folder and above)"] --> T + S["$.secrets"] --> T + C["$.config"] --> T + T --> R[rendered SQL] --> DB[(database)] ``` -Templates are powerful for: -- **Row-level security policies** - Define access rules, generate USING/CHECK clauses -- **Role permissions** - List roles and grants in YAML, generate GRANT/REVOKE statements -- **Scheduled jobs** - Define cron schedules in YAML, generate pg_cron or SQL Agent setup -- **Seed data** - Store reference data in CSV, generate INSERT statements -- **Custom types** - Define domain constraints in YAML, generate CREATE DOMAIN statements - -The pattern is always the same: human-readable configuration goes in a data file, repetitive SQL comes out of the template. - +Data files can be YAML, JSON, JSON5, or CSV. A `$helpers.ts` applies to its folder and every folder below it, and a deeper one overrides a shallower one. [Templates](/guide/sql-files/templates) covers the syntax. -## The Workflow -1. **Update your SQL files first.** They should always reflect the ideal schema. -2. **Write a change** to evolve existing databases to match. +## Configs, stages, and roles -Your SQL files stay in sync with reality. Changes track how you got there. -::: tip Fresh vs Existing -- **Fresh database** → Run SQL files with `noorm run build` -- **Existing database** → Apply changes with `noorm change ff` -::: +A **config** is a saved connection: `dev`, `staging`, `prod`. Commands use the active one unless `--config` names another: - -## Configs - -A **config** is a saved database connection. You can have multiple configs for different environments: - -- `dev` - Your local database -- `test` - Test database (gets wiped between runs) -- `staging` - Staging server -- `prod` - Production (guarded — see Access Roles below) - -``` -┌─ Configs ─────────────────────────────────┐ -│ • dev postgres localhost:5432 │ -│ staging postgres db.staging.com │ -│ prod postgres db.prod.com │ -└───────────────────────────────────────────┘ -``` - -The dot (•) shows which config is active. All commands use the active config unless you specify otherwise. - -Switch configs: ```bash noorm config use staging -``` - -Or specify per-command: -```bash noorm run build --config prod ``` -`--config` is a per-subcommand flag, so it goes *after* the subcommand. Before the subcommand, `-c` means `--cwd` (the only root-level flag), so `noorm -c prod run build` runs the build in a directory named `prod` instead. - - -## Access Roles - -Every config declares an access role — `viewer`, `operator`, or `admin` — separately for two **channels**: `user` (you, at the CLI or TUI) and `agent` (an AI agent, whether it reaches noorm over MCP or by running the CLI). This is how you prevent accidental destructive operations: - -- `viewer` - Read-only. `db teardown`, writes, and changes are all blocked. -- `operator` - Reads and writes are fine; builds, changes, and `db create` require typing a confirmation phrase. Raw DDL, `db teardown`, and `db drop` are denied outright. -- `admin` - Builds and changes run without prompting. The operations that cannot be undone still ask: `db drop`, `db teardown`, `db truncate`, `change rm`, `config` edits, `vault propagate`, and `lock force`. -- Accidental `DROP TABLE` via raw SQL - Still executes if the role allows SQL DDL, which only `admin` does (be careful!) - -Roles are a safety net, not a security boundary. They catch mistakes like running teardown against the wrong database — they won't stop a determined user or malicious script. - - -## Secrets - -Secrets are sensitive values that get inserted **into** your database via SQL templates. They're for application configuration you store in the database: - -- AWS credentials (access key, secret key) -- SMTP credentials for sending emails -- API keys for third-party services - -noorm stores secrets encrypted in your local state. When you run SQL templates, secrets are available but never written to disk in plain text. - -```sql --- sql/config/aws.sql.tmpl -INSERT INTO app_config (key, value) VALUES - ('aws_access_key', '{%~ $.secrets.AWS_ACCESS_KEY %}'), - ('aws_secret_key', '{%~ $.secrets.AWS_SECRET_KEY %}'); -``` - -```bash -# Set a secret for the current config -noorm secret set AWS_ACCESS_KEY "AKIAEXAMPLE" - -# Or pipe the value in so it never reaches the process table or shell history -printf %s "$AWS_SECRET_KEY" | noorm secret set AWS_SECRET_KEY --stdin -``` - -When you rebuild your database, your credentials are always there—but never exposed in your SQL files or version control. - -Secrets are **config-scoped**. Each config has its own set of secrets (dev AWS vs prod AWS). There are also **global secrets** shared across all configs. - - -## State and Identity - -Your **identity** lives in `~/.noorm/`, split across three files: `identity.key` (private key, owner-readable only), `identity.pub` (public key), and `identity.json` (metadata such as name, email, and machine). These are machine-local, and the private key is never shared. - -Your **state** lives in `.noorm/state/state.enc` and contains: - -- Your saved configs (database credentials) -- Secrets (encrypted with your keys) - -State is encrypted with your identity keys. It's not portable between machines—each developer has their own configs and secrets. - -::: warning Don't Commit State -Add `.noorm/state/` to `.gitignore`. It contains machine-specific encrypted data that won't work on other machines anyway. -::: - - -## Settings - -Settings live in `.noorm/settings.yml` and **should be committed**. They define project-wide behavior: - -- Schema and changes directory paths -- Build include/exclude patterns -- Stage definitions -- Secret requirements +A **stage** in `.noorm/settings.yml` applies to the config with the same name. Its `defaults` seed that config when the TUI first creates it, `protected: true` caps its access, and `secrets` lists what it needs: ```yaml # .noorm/settings.yml -paths: - sql: ./sql - changes: ./changes - -build: - include: - - tables - - views - - functions - stages: prod: defaults: + dialect: postgres protected: true secrets: - - key: AWS_SECRET_KEY + - key: PAYMENT_GATEWAY_KEY type: password required: true ``` -The `include` array filters which folders are included in builds. Execution order is alphanumeric—use numeric prefixes (`01_tables`, `02_views`) to control the sequence. Paths are relative to your `paths.sql` directory. +Each config gives a **role** to two channels: `user` (you, in the CLI or TUI) and `agent` (an AI agent over MCP or the CLI). The role decides what runs, what asks for a typed confirmation, and what is refused: +| Role | Runs | Asks first | Refused | +|---|---|---|---| +| `viewer` | explore, read-only SQL | | writes, builds, changes, teardown | +| `operator` | explore, SQL reads and writes | build, change, `db create`, truncate | raw DDL, `db drop`, `db teardown` | +| `admin` | everything else | `db drop`, `db teardown`, `db truncate`, `change rm`, config edits, `vault propagate`, `lock force` | | -## Stages +A stage with `protected: true` caps its configs at `operator` for the user and `viewer` for agents. Roles catch mistakes, such as a teardown against the wrong database. They do not stop someone who has the credentials and connects with another client. [Stages](/guide/environments/stages) covers the rest. -A **stage** is a template for configs. When you create a config and assign it to a stage, it inherits the stage's defaults: -```yaml -stages: - development: - defaults: - dialect: sqlite - isTest: false +## Secrets stay out of committed SQL - production: - defaults: - dialect: postgres - protected: true - secrets: - - key: AWS_SECRET_KEY - type: password - required: true + +A secret is a value a template reads when the SQL runs. The value is stored encrypted and never written into a file: + +```sql +-- db/sql/09_seeds/02_gateway.sql.tmpl +INSERT INTO app_setting (name, value) +VALUES ('payment_gateway_key', {%~ $.quote($.secrets.PAYMENT_GATEWAY_KEY) %}); ``` -Now when you create a config with stage `production`, its resolved access is capped to at most `operator`/`viewer` (it can be stricter, never looser), and it requires an `AWS_SECRET_KEY` secret. +```bash +printf %s "$KEY" | noorm secret set PAYMENT_GATEWAY_KEY --stdin +``` + +When a template asks for `$.secrets.PAYMENT_GATEWAY_KEY`, noorm looks in the config's own secrets first, then in the [vault](/guide/environments/vault), which is stored encrypted in the database and shared with the team. A secret set on your config overrides the team's value for you alone. + + +## What lives where + + +| Path | Holds | Commit it | +|---|---|---| +| `db/sql/`, `db/changes/` | the schema and its changes | yes | +| `db/AGENTS.md`, `db/CLAUDE.md` | instructions for AI agents working in `db/`, written by `noorm init` | yes | +| `.noorm/settings.yml` | paths, `build.include`, stages, required secrets | yes | +| `.noorm/state/state.enc` | your configs, credentials, and secrets, encrypted with your identity key | no, `noorm init` adds it to `.gitignore` | +| `~/.noorm/identity.key`, `identity.pub`, `identity.json` | your keypair, name, and email | never leaves your machine | -Connection values go under `defaults`. Required secrets go under `secrets`, and each entry needs a `type` (`string`, `password`, `api_key`, or `connection_string`). Settings fail to load without it. +State is encrypted with your own key, so a copy is useless on another machine. Each developer keeps their own. -## SQL Files vs Changes +## What bites -| Aspect | SQL Files | Changes | -|--------|-----------|---------| -| Purpose | Define ideal schema | Evolve existing databases | -| Location | `sql/` | `changes/` | -| When to run | Fresh database, tests | Existing database | -| Rollback | No | Yes, with revert scripts | -| Order | Alphanumeric by full path | By date prefix | -**SQL files** are your source of truth. They define what your schema looks like today. Execution order is alphanumeric by full path. `build.include` picks which folders take part; it does not reorder them, so the numeric prefixes are what control the sequence. +| Mistake | What happens | Fix | +|---|---|---| +| Edit a table in `db/sql/` with no change | new databases get the edit, existing ones never do | add a change: `noorm change add <slug>` | +| A change that is not safe to run twice | a database built from `db/sql/` has no change records, so `change ff` runs the change against a table that already has it | guard it: `ADD COLUMN IF NOT EXISTS`, `WHERE status IS NULL` | +| Folders without number prefixes | alphabetical order runs `functions/` before `tables/` | `01_tables/`, `02_views/` | +| `noorm -c prod run build` fails: `--cwd path is not a directory` | before the subcommand, `-c` is `--cwd`, so noorm looks for a project folder named `prod` | `noorm run build --config prod` | -**Changes** are your changelog. They document how to get an existing database to match your SQL files. +## Next -## What's Next? -- [Building Your SDK](/getting-started/building-your-sdk) - Wrap noorm in a TypeScript SDK for your apps -- [TUI Quick Reference](/tui) - Navigate the terminal interface -- [SQL Templates](/guide/sql-files/templates) - Dynamic SQL generation -- [Changes Guide](/guide/changes/overview) - Evolving existing databases +- [First build](/getting-started/first-build): init a project and build it +- [Add a column](/guide/changes/add-a-column): one edit, both folders +- [Templates](/guide/sql-files/templates): the full syntax +- [Building your SDK](/getting-started/building-your-sdk): call the database from TypeScript diff --git a/docs/getting-started/first-build.md b/docs/getting-started/first-build.md index e882cb9c..a52f1456 100644 --- a/docs/getting-started/first-build.md +++ b/docs/getting-started/first-build.md @@ -1,28 +1,57 @@ -# First Build +# First build -This tutorial takes about 5 minutes. By the end, you'll understand how noorm separates **schema definition** from **schema evolution**. +Create a project, connect it to a PostgreSQL server, and build a schema from SQL files. -![Building a schema from SQL files, then fast-forwarding two pending changes](/image/build-and-change.gif) +```sql +-- db/sql/01_tables/02_sales_order.sql +CREATE TABLE sales_order ( + party_id integer NOT NULL, + order_no integer NOT NULL, + placed_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT sales_order_pk + PRIMARY KEY (party_id, order_no), + CONSTRAINT sales_order_isplacedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) +); +``` -That is the whole loop: `run build` applies the SQL files, `change ff` walks an existing database forward, and `change history` records what ran. The rest of this page does it step by step. +```bash +noorm run build +``` +``` +• log Build completed status=success filesRun=2 filesSkipped=0 filesFailed=0 durationMs=31.208 +``` + +Real runs prefix each line with a timestamp. -## What You'll Do -1. Initialize a project and configure a database -2. Create SQL files and build your schema -3. Evolve the schema with a change -4. Create a fresh test database to see you don't need migrations +## What do the commands touch? -## Prerequisites +In this walkthrough, only three commands change the database: `db create`, `run build`, and `change ff`. + +```mermaid +sequenceDiagram + participant You + participant noorm + participant Database + You->>noorm: noorm init + noorm-->>You: .noorm/ and db/ + You->>noorm: noorm ui, then c and a + noorm->>Database: test the server connection + You->>noorm: noorm db create + noorm->>Database: create it, add tracking tables + You->>noorm: noorm run build + noorm->>Database: run each db/sql/ file + You->>noorm: noorm change ff + noorm->>Database: run each pending change +``` -- noorm installed ([Installation Guide](/getting-started/installation)) -- A PostgreSQL database to connect to +## Create the project -## Step 1: Initialize Your Project ```bash mkdir noorm-tutorial @@ -30,316 +59,157 @@ cd noorm-tutorial noorm init ``` -`noorm init` sets up your identity (or reuses the one already in `~/.noorm/`) and creates the project scaffolding. When it finishes, you'll have this layout: +`noorm init` asks for a display name and email unless `~/.noorm/` already holds an identity. It creates: ``` noorm-tutorial/ +├── .gitignore # gets a .noorm/state/ line ├── .noorm/ -│ ├── .gitignore # ignores state/ -│ ├── settings.yml # Project settings (commit this) +│ ├── .gitignore +│ ├── settings.yml # commit this │ └── state/ -│ └── state.enc # Encrypted state (don't commit) -├── sql/ # Your SQL files go here -└── changes/ # Versioned changes go here +│ └── state.enc # your configs and secrets, encrypted +└── db/ + ├── sql/ + ├── changes/ + ├── AGENTS.md + └── CLAUDE.md ``` -Configs are added separately, from the TUI. That's the next step. +`db/AGENTS.md` sends AI agents to the `/noorm` skill, and `db/CLAUDE.md` imports it. [What lives where](/getting-started/concepts#what-lives-where) covers the rest. -## Step 2: Add a Database Config +## Add a config -Launch the TUI: + +A config is a saved connection. Creating one is interactive: ```bash noorm ui ``` -Press `c` for config, then `a` to add a new config. Fill out the form: - -- **Config Name**: `dev` -- **Database Type**: PostgreSQL (use arrow keys to select) -- **Host**: `localhost` -- **Port**: `5432` -- **Database**: `noorm_dev` -- **Username**: `postgres` -- **Password**: your password -- **User Role (CLI/TUI access)**: leave at `admin` -- **Agent Role (MCP/CLI access)**: leave at `viewer` -- **Test Database**: leave unchecked - -Submit the form to save your config. SQL and changes paths are not part of a config. They live in `.noorm/settings.yml` under `paths`, and default to `./sql` and `./changes`. - - -## Step 3: Create the Database - -Back at the home screen, press `d` for Database. You'll see the Database Operations screen: - -``` -Database Operations - -Config: dev -Connection: NOT CREATED - -Available Actions - -[c] Create - Create database and tracking tables -[d] Destroy - Drop all managed objects -[x] Explore - Browse database schema -[w] Wipe - Truncate table data (keep schema) -[t] Teardown - Drop user objects (keep noorm) -[r] Transfer - Copy data to another database -``` - -Press `c` to create. noorm will create the `noorm_dev` database on your PostgreSQL server and initialize its tracking tables. - -::: tip No SQL Files Yet? -That's fine! At this point we're just creating the empty database and noorm's internal tracking tables. We'll add SQL files next. -::: +Press `c`, then `a`. Enter name `dev`, type PostgreSQL, host `localhost`, port `5432`, database `noorm_dev`, and your server's username and password. Leave User Role at `admin`, Agent Role at `viewer`, and Test Database unchecked. +The form tests the server connection before it saves, and the first config you save becomes the active one. Quit with `ctrl + c`. -## Step 4: Configure Build Settings -Open `.noorm/settings.yml` and define which folders to include: - -```yaml -paths: - sql: ./sql - changes: ./changes - -build: - include: - - 01_tables -``` +## Create the database -The `include` array filters which folders are included in builds. Folder names are relative to your `paths.sql` directory. -::: warning Execution Order -Files and directories run in **alphanumeric order**. Without numeric prefixes: - -``` -sql/functions/ ← runs first (alphabetically) -sql/tables/ ← runs second -sql/views/ ← runs third +```bash +noorm db create ``` -That's wrong—tables should exist before views reference them. Use numeric prefixes: - ``` -sql/01_tables/ ← runs first -sql/02_views/ ← runs second -sql/03_functions/ ← runs third +Database "noorm_dev" created and initialized. ``` -This rule applies everywhere: directories, files within directories, change scripts. One mental model for everything. -::: +It also adds noorm's tracking tables. -## Step 5: Create Your First SQL File +## Build the schema -Create the users table: ```bash -mkdir -p sql/01_tables +mkdir db/sql/01_tables ``` -Create `sql/01_tables/001_users.sql`: +Save the `sales_order` file from the top of this page in that folder, and add the table it points at: ```sql -CREATE TABLE IF NOT EXISTS users ( - id SERIAL PRIMARY KEY, - name TEXT NOT NULL, - email TEXT UNIQUE, - created_at TIMESTAMP DEFAULT NOW() +-- db/sql/01_tables/01_party.sql +CREATE TABLE party ( + party_id integer GENERATED ALWAYS AS IDENTITY, + name text NOT NULL, + CONSTRAINT party_pk PRIMARY KEY (party_id) ); ``` +Then run `noorm run build`. It prints the block at the top of this page: two files run in path order, so `party` exists before the foreign key needs it. -## Step 6: Run Build - -```bash -noorm run build -``` - -Output: - -``` -• build:start Starting schema build (1 files) -• file:after Executed sql/01_tables/001_users.sql (12ms) -• build:complete Build complete: 1 run, 0 skipped (18ms) -• log Build completed status=success filesRun=1 filesSkipped=0 filesFailed=0 durationMs=18 -``` - -Your users table now exists. - - -## Step 7: Evolve the Schema - -Now you want to expand the schema: add a `bio` column to users, create a posts table, and add a view. -First, **update your SQL files** to reflect the new ideal state. +## Catch an existing database up -Edit `sql/01_tables/001_users.sql`: -```sql -CREATE TABLE IF NOT EXISTS users ( - id SERIAL PRIMARY KEY, - name TEXT NOT NULL, - email TEXT UNIQUE, - bio TEXT, - created_at TIMESTAMP DEFAULT NOW() -); -``` - -Create `sql/01_tables/002_posts.sql`: +Treat `dev` as a database you cannot rebuild. Declare a new table in `db/sql/`, then add a change so `dev` gets it too. ```sql -CREATE TABLE IF NOT EXISTS posts ( - id SERIAL PRIMARY KEY, - user_id INTEGER REFERENCES users(id), - title TEXT NOT NULL, - body TEXT, - created_at TIMESTAMP DEFAULT NOW() +-- db/sql/01_tables/03_invoice.sql +CREATE TABLE invoice ( + party_id integer NOT NULL, + invoice_no integer NOT NULL, + issued_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT invoice_pk + PRIMARY KEY (party_id, invoice_no), + CONSTRAINT invoice_isowedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) ); ``` -Create `sql/02_views/001_recent_posts.sql`: - -```bash -mkdir -p sql/02_views -``` - -```sql -CREATE OR REPLACE VIEW recent_posts AS -SELECT p.*, u.name as author_name -FROM posts p -JOIN users u ON p.user_id = u.id -WHERE p.created_at > NOW() - INTERVAL '7 days'; -``` - -Since you added views, **update your settings** to include the new folder: - -```yaml -build: - include: - - 01_tables - - 02_views -``` - -The `02_` prefix ensures views run after tables—important since views reference tables. - -Your SQL files now represent the complete, ideal schema. But the dev database still only has the old users table. - - -## Step 8: Create a Change - -Use the TUI to create a change: - ```bash -noorm ui -``` - -Press `g` for changes, then `a` to add a new change. Enter a name like `add-posts-and-bio`. - -The TUI creates the folder structure and shows: - -``` -✓ Change "2024-01-15-add-posts-and-bio" created! -Created folder structure with template files. - -[e] Edit in editor [any] Back to list +noorm change add invoice-table ``` -Press `e` to open the change folder in your editor, or navigate to it manually. Add these files: - -`changes/2024-01-15-add-posts-and-bio/change/001_add_bio.sql`: +This scaffolds `db/changes/2026-10-21-invoice-table/` (your folder carries the day you run it) with comment-only stubs in `change/` and `revert/`. Replace both. The change carries its own guarded copy of the table: ```sql -ALTER TABLE users ADD COLUMN IF NOT EXISTS bio TEXT; +-- db/changes/2026-10-21-invoice-table/change/001_invoice-table.sql +CREATE TABLE IF NOT EXISTS invoice ( + party_id integer NOT NULL, + invoice_no integer NOT NULL, + issued_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT invoice_pk + PRIMARY KEY (party_id, invoice_no), + CONSTRAINT invoice_isowedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) +); ``` -Changes should be idempotent—safe to run on any database state. PostgreSQL's `IF NOT EXISTS` ensures this won't error if the column already exists. - -`changes/2024-01-15-add-posts-and-bio/change/002_schema.txt`: - -``` -01_tables/002_posts.sql -02_views/001_recent_posts.sql +```sql +-- db/changes/2026-10-21-invoice-table/revert/001_invoice-table.sql +DROP TABLE IF EXISTS invoice; ``` -The `.txt` file references your SQL files. When the change runs, it executes those files from your `sql/` directory. - - -## Step 9: Apply the Change - ```bash noorm change ff ``` -Output: - ``` -• change:start Starting change for 2024-01-15-add-posts-and-bio (3 files) -• change:complete change 2024-01-15-add-posts-and-bio: success (29ms) -• log Fast-forward success executed=1 skipped=0 failed=0 -• log 2024-01-15-add-posts-and-bio (success) +• log Fast-forward success executed=1 skipped=0 failed=0 +• log 2026-10-21-invoice-table (success) ``` -Three files, because the `002_schema.txt` manifest expands to the two SQL files it names. Per-file lines are `change:file` events, which sit at debug level. Raise `logging.level` to `verbose` in `.noorm/settings.yml` to see them. +Adding a column works the same way: see [Add a column](/guide/changes/add-a-column). -Your dev database now matches your SQL files. +## Build a second database -## Step 10: Fresh Database (No Migrations Needed) -Now create a test database to see the other path. In the TUI: - -1. Press `c` for Configurations, then `a` to add a new config named `test` pointing to `noorm_test` -2. Use arrow keys to select the `test` config and press `Enter` to switch to it -3. Press `d` for Database, then `c` to create the database -4. Press `r` for Run SQL, then `b` for Build - -If you already have a test config and database set up, the same flow runs from the non-interactive CLI: +A fresh database takes the schema from `db/sql/`. The change exists for databases that predate the table. Point one command at another database name with `NOORM_CONNECTION_DATABASE`: ```bash -noorm config use test -noorm run build +NOORM_CONNECTION_DATABASE=noorm_test noorm db create +NOORM_CONNECTION_DATABASE=noorm_test noorm run build ``` -Output: - -``` -• build:start Starting schema build (3 files) -• file:after Executed sql/01_tables/001_users.sql (14ms) -• file:after Executed sql/01_tables/002_posts.sql (11ms) -• file:after Executed sql/02_views/001_recent_posts.sql (7ms) -• build:complete Build complete: 3 run, 0 skipped (32ms) -• log Build completed status=success filesRun=3 filesSkipped=0 filesFailed=0 durationMs=32 -``` - -The test database has the complete schema from your SQL files. No changes needed—they're only for evolving existing databases. - +The build runs 3 files, `invoice` included. The variable overrides the stored database name for that command, and the `dev` config is unchanged. -## What Just Happened? -You experienced both paths: +## What bites -| Scenario | Command | What Runs | -|----------|---------|-----------| -| Fresh database | `noorm run build` | SQL files | -| Existing database | `noorm change ff` | Changes | -Your SQL files are the source of truth. Changes are the changelog of how you evolved existing databases to match them. +| Symptom | Cause | Fix | +|---|---|---| +| `noorm init` exits: requires an interactive terminal | `init` prompts, and a script has no terminal | Run `noorm identity init --name "Ada" --email ada@example.com` once, then `noorm init --yes` | +| `noorm config add` prints: Interactive only | The CLI cannot prompt for connection details | Use `noorm ui`, or `noorm config import <file.json>` in scripts | +| `"run:build" is not allowed on config "dev" (role: viewer)` | An AI agent is driving the CLI, so the config's agent role applies, and it defaults to `viewer` | Run the command in your own terminal, or raise Agent Role on the config | +| `change ff` on a freshly built database fails: relation "invoice" already exists | The change has no guard, and a built database has no change records, so `ff` runs every change | Guard it: `CREATE TABLE IF NOT EXISTS` | -::: tip The Workflow -1. Update your SQL files to reflect the ideal schema -2. Create a change to evolve existing databases -3. Fresh databases just run build—no migration history needed -::: +## Next -## What's Next? -- [Concepts](/getting-started/concepts) - Understand the full mental model -- [Building Your SDK](/getting-started/building-your-sdk) - Wrap noorm in a TypeScript SDK for your apps -- [TUI Quick Reference](/tui) - Navigate the terminal interface -- [Changes Guide](/guide/changes/overview) - Versioned changes in depth +- [Concepts](/getting-started/concepts): the two folders and the commands that apply them +- [Add a column](/guide/changes/add-a-column): one edit, both folders, with a backfill +- [Building your SDK](/getting-started/building-your-sdk): query the schema from TypeScript +- [Terminal UI](/tui): keys and screens diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 054f0077..053f70d8 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -1,174 +1,111 @@ # Installation -![Installing noorm: the install script, creating an identity, and bootstrapping a project](/image/install.gif) - - -## Requirements - -- A supported database: - - PostgreSQL - - MySQL - - SQLite - - SQL Server (MSSQL) - - -## Install the CLI - -`noorm` is a non-interactive CLI by default. Every command emits structured output suitable for scripting or piping into CI/CD. Running `noorm ui` launches the optional Ink/React terminal wizard for when you'd rather point-and-click. - -### macOS / Linux +Install the noorm CLI, then the SDK and the agent skill if you need them. ```bash curl -fsSL https://noorm.dev/install.sh | sh ``` -This downloads a self-contained binary for your platform. The installer looks for a user-writable bin directory already in your PATH (`~/.local/bin`, `~/bin`, etc.) so you don't need sudo. No runtime dependencies needed. - -::: tip Corporate network? -If `noorm.dev` is blocked, use the GitHub mirror: -```bash -curl -fsSL https://raw.githubusercontent.com/noormdev/noorm/master/install.sh | sh ``` -::: - -You can also override the install location: - -```bash -NOORM_INSTALL_DIR=~/my-tools curl -fsSL https://noorm.dev/install.sh | sh +Installing noorm v1.5.1 (darwin-arm64) to /Users/ada/.local/bin... +Installed noorm v1.5.1 to /Users/ada/.local/bin/noorm ``` -### npm - -If you already work in a Node project, npm installs the same binary and puts it on your PATH for you: - ```bash -npm install -g @noormdev/cli +noorm --version ``` -The package is a thin wrapper: installing it downloads the prebuilt binary for your platform and verifies it, exactly as the shell installer does. `npx @noormdev/cli` works too if you'd rather not install globally. +``` +1.5.1 +``` -This is the most convenient route on Windows, and the one to reach for when you want noorm pinned per-project rather than per-machine. +The binary needs no runtime and includes the database drivers for PostgreSQL, MySQL, SQLite, and SQL Server. For npm, Windows, and the GitHub mirror, see [Download](/download). -### Windows +`noorm` on its own prints the command list. `noorm ui` opens the [terminal UI](/tui). -Use the npm install above, or download [noorm-windows-x64.exe](https://github.com/noormdev/noorm/releases) from the latest `@noormdev/cli` release, rename it to `noorm.exe`, and add it to your PATH. -### Integrity verification +## Install the SDK (optional) -Every install path verifies the downloaded binary before it runs. The release publishes a `checksums.txt` alongside the binaries, and the installer (`install.sh`), the npm `postinstall` step, and `noorm update` each recompute the binary's SHA-256 and compare it against that file before making the binary executable. A mismatch aborts — a corrupted download or a tampered release asset never gets run. -Verification is fail-closed: if `checksums.txt` can't be fetched (an offline mirror, a network block, an older release without it), the install stops rather than trusting an unverified binary. To override that one case — and only that case — set `NOORM_INSECURE=1`: +The SDK needs Node 22.13 or newer, `kysely`, and the driver for your dialect: ```bash -NOORM_INSECURE=1 curl -fsSL https://noorm.dev/install.sh | sh - -# self-update equivalent -noorm update --insecure +npm install @noormdev/sdk kysely pg ``` -`NOORM_INSECURE` skips verification only when the checksums can't be reached. A confirmed hash mismatch always fails, escape hatch or not — the flag can't wave through a binary that has actually been altered. +| Dialect | Driver | +|---|---| +| PostgreSQL | `pg` | +| MySQL | `mysql2` | +| SQLite | `better-sqlite3` | +| SQL Server | `tedious` and `tarn` | -### Verify +Drivers are optional peer dependencies, so install only the one you use. -```bash -noorm --version -``` -### Update +## Install the agent skill (optional) -noorm can update itself: -```bash -noorm update +The skill tells your coding agent where SQL goes in a noorm project, how SDK calls report errors, and which CLI flags CI needs, so it stops guessing. -# Or launch the TUI (`noorm ui`) and press [u] from the home screen +```bash +npx skills add noormdev/noorm/skills ``` - -## Install the SDK (Optional) - -For programmatic access in scripts, tests, or custom tooling: +Without Node.js, GitHub CLI 2.90 or newer installs the same skill: ```bash -npm install @noormdev/sdk kysely +gh skill install noormdev/noorm skills/noorm ``` -The SDK uses peer dependencies. Install the driver for your database: - -| Dialect | Install | -|---------|---------| -| PostgreSQL | `npm install pg` | -| MySQL | `npm install mysql2` | -| SQLite | `npm install better-sqlite3` | -| SQL Server | `npm install tedious tarn` | -::: tip CLI Includes Drivers -The CLI bundles all drivers — no extra installation needed. These are only required if you're using the SDK directly. -::: +## Update noorm -## Install the Agent Skill (Optional) - -If you work with an AI coding agent — Claude Code, Codex, Cursor, Gemini CLI — install the noorm skill: - ```bash -npx skills add noormdev/noorm/skills +noorm update ``` -Agents otherwise guess at noorm's conventions, and several of them are non-obvious enough to guess wrong: SDK methods throw named errors rather than returning them, `attempt()` is used deliberately rather than everywhere, the codebase uses no `try`/`catch`, and test helpers carry safety guards that exist for a reason. The skill covers all of it, plus the headless CLI flags used in CI. +After a successful update, noorm offers the skill, using the same commands: + +| Where `noorm update` runs | Skill step | +|---|---| +| A terminal, with `npx` or `gh` on `PATH` | Asks before it installs or updates the skill | +| Piped output, with `npx` or `gh` on `PATH` | Prints the command to run | +| Neither `npx` nor `gh` on `PATH` | Prints both commands and where to get Node.js and the GitHub CLI | +| `--json` | Skips the offer | -It installs four references the agent reads on demand: +A declined or failed skill install does not change the exit code. From the TUI, press `u` on the home screen. -| Reference | Covers | -|-----------|--------| -| `sdk.md` | `createContext()`, Kysely patterns, error handling, testing | -| `cli.md` | Headless commands, flags, exit codes, CI usage | -| `templates.md` | `.sql.tmpl` syntax and data loaders | -| `config.md` | `settings.yml`, paths, stages, secrets | -::: tip Include paths are relative to `paths.sql` -The single most common mistake — by people and agents alike — is writing `build.include` entries relative to the project root. They resolve against `paths.sql`. The skill states this explicitly; `noorm run build` also warns when an include entry matches nothing. -::: +## How is the download verified? -## Initialize a Project +The npm package, `noorm update`, and the script at `raw.githubusercontent.com/noormdev/noorm/master/install.sh` compare the binary's SHA-256 with the `checksums.txt` published beside it. A mismatch always aborts. The short script at `noorm.dev/install.sh` does not check yet; install with the GitHub URL when you want the check. -Run `noorm init` in your project root. It prompts for the identity to create (skipped if you already have one) and scaffolds the project: +If `checksums.txt` cannot be fetched, they stop. `NOORM_INSECURE=1` overrides that case only: ```bash -cd your-project -noorm init +curl -fsSL https://raw.githubusercontent.com/noormdev/noorm/master/install.sh | NOORM_INSECURE=1 sh +noorm update --insecure ``` -The init wizard creates: - -``` -your-project/ -├── .gitignore # gets a .noorm/state/ entry appended -├── .noorm/ -│ ├── .gitignore # ignores state/ -│ ├── settings.yml # Project settings (commit this) -│ └── state/ -│ └── state.enc # Encrypted state (don't commit) -├── sql/ # Your SQL files -│ └── .gitkeep -└── changes/ # Versioned changes - └── .gitkeep -``` -`noorm init` is strictly interactive and exits with an error on a non-TTY unless you pass `--yes` and already have an identity at `~/.noorm/identity.{key,pub,json}`. +## What bites -Adding the first database config comes next, from the TUI (`noorm ui`, then `c` for config and `a` to add). -::: warning Git Ignore -`noorm init` adds `.noorm/state/` to your `.gitignore` for you. That folder holds encrypted configs and secrets specific to each developer's machine. The `.noorm/settings.yml` should be committed — it's the shared project configuration. -::: +| Symptom | Cause | Fix | +|---|---|---| +| `noorm: command not found` | The install directory is not on `PATH` | Add the `export PATH="<dir>:$PATH"` line the installer prints to your shell profile | +| `NOORM_INSTALL_DIR` has no effect | The variable was set before `curl`, so only `curl` sees it | Set it after the pipe: `curl … \| NOORM_INSTALL_DIR=~/bin sh` | +| Install stops with "could not verify checksum" | `checksums.txt` is unreachable or has no entry for your platform | Retry, or set `NOORM_INSECURE=1` as shown above | +| The SDK cannot connect and reports a missing driver | The driver is a peer dependency | Install the driver for your dialect | -## Next Steps +## Next -You're ready to go! Continue to: -- [First Build](/getting-started/first-build) - Complete the 5-minute tutorial -- [Concepts](/getting-started/concepts) - Understand the mental model +- [First build](/getting-started/first-build): `noorm init`, a config, and a built schema +- [Concepts](/getting-started/concepts): the two folders and how databases reach the same schema +- [Download](/download): every install method and option diff --git a/docs/guide/automation/ci.md b/docs/guide/automation/ci.md index f08ddc33..f8961f3c 100644 --- a/docs/guide/automation/ci.md +++ b/docs/guide/automation/ci.md @@ -1,194 +1,117 @@ # CI/CD Automation -Running noorm from an automated pipeline feels a lot like running it on your laptop — the same `run build`, `change ff`, and `db transfer` commands do the same things — but a CI runner has no `.noorm/` directory, no interactive TTY, and no identity on disk. The `noorm ci` namespace exists to close that gap: bootstrap an ephemeral state from environment variables, enroll a long-lived identity once from a developer machine, and let the same commands work in both places. - -This guide walks through the two common shapes of CI and the commands that connect them. - - -## Two Shapes of CI - -Not every pipeline needs the same setup. Before writing YAML, decide which of these describes your job: - -| | Test CI | Prod CI | -|---|---|---| -| **Database** | Ephemeral (Postgres container, SQLite file, etc.) spun up in the job | Real, shared, already has data | -| **Vault-backed secrets?** | No — templates don't reference `secrets.*` from the vault | Yes — templates decrypt keys at render time | -| **Identity source** | Not required (no vault to decrypt, no audit trail worth sharing) | Required (enrolled once, keeps the audit trail of who deployed what) | -| **`noorm ci init`?** | Optional | **Required** | -| **Where to start** | [Test CI section](#test-ci) | [Prod CI section](#prod-ci) | - -When in doubt: if any change or template calls `secrets.KEY` and that key lives in the team vault, you need Prod CI. - - -## Test CI - -The goal is to verify the schema builds and changes apply against a throwaway database. Nothing shared, nothing persisted, nothing secret. - -The flow is three env vars and two commands: - -```bash -export NOORM_CONNECTION_DIALECT=postgres -export NOORM_CONNECTION_HOST=localhost -export NOORM_CONNECTION_DATABASE=test_db -export NOORM_CONNECTION_USER=postgres -export NOORM_CONNECTION_PASSWORD=test - -noorm run build -noorm change ff -``` - -No identity, no `ci init`, no vault. If the schema compiles and all changes apply cleanly, exit 0. - - -### GitHub Actions (Test CI) +Run `noorm run build` and `noorm change ff` from a pipeline that has no saved state, no terminal, and no identity file. ```yaml +# .github/workflows/database.yml name: Database CI on: pull_request: - branches: [main] jobs: - test: + schema: runs-on: ubuntu-latest services: postgres: image: postgres:16 env: POSTGRES_PASSWORD: test - POSTGRES_DB: test_db + POSTGRES_DB: order_to_cash_ci ports: - 5432:5432 options: >- --health-cmd pg_isready - --health-interval 10s - --health-timeout 5s + --health-interval 5s --health-retries 5 + env: + NOORM_CONNECTION_DIALECT: postgres + NOORM_CONNECTION_HOST: localhost + NOORM_CONNECTION_DATABASE: order_to_cash_ci + NOORM_CONNECTION_USER: postgres + NOORM_CONNECTION_PASSWORD: test steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '22' - - run: npm ci - - - name: Apply schema and changes - env: - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: localhost - NOORM_CONNECTION_DATABASE: test_db - NOORM_CONNECTION_USER: postgres - NOORM_CONNECTION_PASSWORD: test - run: | - npx noorm run build - npx noorm change ff - + - run: npm install -g @noormdev/cli + - run: noorm run build + - run: noorm change ff - run: npm test ``` +A database built from `db/sql/` has no change records, so `change ff` replays every change in `db/changes/`. A change that is not safe to run twice fails here instead of in production. -## Prod CI - -When templates render vault-backed secrets or the target config's `access.user` role isn't `admin`, the runner needs two things the test flow skipped: - -1. **An identity** — an X25519 keypair whose public key is enrolled in the target database, giving it a slot in the `encrypted_vault_key` column. -2. **A bootstrapped `.noorm/state/state.enc`** — the ephemeral state that every later command expects to find. - -The provisioning (step 1) happens once, on a developer machine. The runtime (step 2) happens every job. - +Tests that open the database through the SDK with `requireTest` need a stored test config. Add the `NOORM_IDENTITY_*` variables and run `noorm ci init --name test --test` before `npm test`, as [Building your SDK](/getting-started/building-your-sdk#where-does-the-connection-come-from) shows. -### One-Time: Provision the CI Identity -Pick the flow that matches your trust model: +## Which shape does my pipeline need? -- **In-place** — mint the keypair right on the developer's laptop and grant it vault access. Simplest; the private key never leaves the developer's terminal. -- **Air-gapped** — mint the keypair on one machine, hand the public key to a developer with vault access, and have them enroll just the public half. Use when the machine that will hold the CI private key is different from the one with vault access. +Test CI builds a throwaway database and replays every change. Prod CI applies only the pending changes to a shared database: -#### In-Place - -Run this on a machine that already has vault access on the target config: - -```bash -noorm ci identity enroll \ - --config prod \ - --name "GitHub CI" \ - --email ci@example.com \ - --yes +```mermaid +flowchart TD + subgraph TEST["Test CI"] + T1["noorm run build"] --> T2["noorm change ff"] + T2 --> T3["npm test"] + end + subgraph PROD["Prod CI"] + P1["noorm ci init"] --> P2["noorm change ff"] + end ``` -`--yes` is required, not optional. Enrollment ends in a `vault:propagate` grant, which the matrix marks `confirm` for every role that holds it, and sealing the vault key to a public key cannot be undone. Without `--yes` (or `NOORM_YES=1`) the command prints the identity it was about to grant to and exits 1 so you can check it first. - -The command: - -1. Mints a new X25519 keypair. -2. Decrypts the vault key using your own private key. -3. Inserts the new identity row and re-encrypts the vault key for it. -4. **Prints the private key exactly once** — stdout contains `NOORM_IDENTITY_PRIVATE_KEY=<96-hex>`. - -Copy the printed block into your CI provider's secret store (GitHub Actions Secrets, GitLab CI Variables, etc.) under whatever name you like. You'll reference it later as `NOORM_IDENTITY_PRIVATE_KEY`. +| | Test CI | Prod CI | +|---|---|---| +| Database | Created in the job | Shared, holds data | +| Identity | None, unless SDK tests need a stored test config | Enrolled once, read from `NOORM_IDENTITY_*` | +| Vault secrets | Unavailable | Decrypted when templates render | +If a change or template reads `$.secrets.KEY` and the key lives in the team vault, you need prod CI. -#### Air-Gapped -On the machine that will hold the CI secret: +## How do I set up prod CI? -```bash -noorm ci identity new --name "GitHub CI" --email ci@example.com --json > ci-identity.json -``` -This prints the same env block as `ci identity enroll`, but does not touch any database. Extract the `publicKey` from `ci-identity.json` and send it to someone with vault access on the target config. They run: +A developer with vault access enrolls the CI identity once. Every job then builds its own state from environment variables: -```bash -noorm ci identity enroll \ - --config prod \ - --name "GitHub CI" \ - --email ci@example.com \ - --public-key <hex-from-ci-identity.json> \ - --yes +```mermaid +sequenceDiagram + participant Dev as Developer + participant DB as Database + participant Store as CI secrets + participant Job as Runner + Dev->>DB: ci identity enroll --yes + DB-->>Dev: identity added, vault key sealed to it + Dev->>Store: save the identity variables + Note over Dev,Store: once per CI identity + Job->>Store: read identity and connection variables + Job->>Job: ci init writes .noorm/state/state.enc + Job->>DB: change ff + DB-->>Job: pending changes applied ``` -Only the public half is enrolled. The private key never leaves the first machine. The public key is 88 hex characters; anything else is rejected before a row is written. - -`ci identity enroll` is **idempotent on identityHash** — re-running just ensures vault access, so you can replay it safely if anything fails partway through. It refuses outright when a row already exists under that hash carrying a *different* public key, since granting there would seal the vault key to someone else's key. - - -### Every Job: Runtime Bootstrap - -Inside the CI job, set the three identity env vars plus the connection and call `ci init`: +Enroll from a machine that has vault access on `prod`. The command prints `NOORM_IDENTITY_PRIVATE_KEY` once. Save it, with the name and email, in your CI secret store: ```bash -noorm ci init --name prod -noorm change ff +noorm ci identity enroll --config prod --name "GitHub CI" --email ci@example.com --yes ``` -`ci init` reads `NOORM_IDENTITY_*` and `NOORM_CONNECTION_*`, writes a fresh `.noorm/state/state.enc`, creates a config (default name `ci`, override via `--name` or `NOORM_CI_CONFIG_NAME`), and marks it active. After it runs, every other `noorm` command in the job works exactly as if a developer had bootstrapped the project manually. - -If the env vars are missing or malformed, `ci init` exits 2 (bad invocation) before doing anything else. If `state.enc` already exists and `--force` was not passed, it exits 1. With `--force` it backs the old file up next to it first, and from an interactive terminal it also demands `--yes`, since overwriting destroys every config and config-scoped secret in the project. - - -### Loading Per-Job Secrets - -When your templates need secrets that shouldn't live in the vault — one-off job tokens, ephemeral API keys — use `ci secrets` to batch-load them into the active config's vault from a dotenv file: +If the CI secret lives on a different machine than the vault access, mint the keypair there and enroll only the public half: ```bash -cat > ./ci-secrets.env <<EOF -API_KEY=$API_KEY -STRIPE_KEY=$STRIPE_KEY -EOF - -noorm ci secrets --file ./ci-secrets.env -rm ./ci-secrets.env +noorm ci identity new --name "GitHub CI" --email ci@example.com --json > ci-identity.json +noorm ci identity enroll --config prod --name "GitHub CI" --email ci@example.com \ + --public-key <publicKey from ci-identity.json> --yes ``` -Existing keys are skipped by default (so a rerun is safe). Pass `--overwrite` to replace them. Exit codes: `0` clean load, `1` total failure, `2` bad invocation (unknown config, unreadable or malformed `--file`), `3` partial (some keys set, some errored). - - -### GitHub Actions (Prod CI) +Each job runs `ci init`, then `change ff`: ```yaml -name: Deploy Database Changes +# .github/workflows/deploy-db.yml +name: Deploy database changes on: push: @@ -197,158 +120,60 @@ on: jobs: deploy: runs-on: ubuntu-latest + env: + NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} + NOORM_IDENTITY_NAME: GitHub CI + NOORM_IDENTITY_EMAIL: ci@example.com + NOORM_CONNECTION_DIALECT: postgres + NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} + NOORM_CONNECTION_DATABASE: order_to_cash + NOORM_CONNECTION_USER: ${{ secrets.DB_USER }} + NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_PASSWORD }} + # the prod stage is protected, so change ff asks for confirmation + NOORM_YES: "1" steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '22' - - run: npm ci - - - name: Bootstrap CI state - env: - NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} - NOORM_IDENTITY_NAME: "GitHub CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} - NOORM_CONNECTION_DATABASE: ${{ secrets.DB_NAME }} - NOORM_CONNECTION_USER: ${{ secrets.DB_USER }} - NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_PASSWORD }} - run: npx noorm ci init --name prod - - - name: Apply changes - run: npx noorm change ff + - run: npm install -g @noormdev/cli + - run: noorm ci init --name prod + - run: noorm change ff ``` +GitLab CI takes the same variables under `variables:` and the same three commands. `NOORM_CI_CONFIG_NAME` renames the config when `--name` is absent (default `ci`). The connection variables are listed under [environment-only mode](/headless#environment-only-mode). -### GitHub Actions (Prod CI with job-scoped secrets) - -If your changes read vault keys the runner doesn't already have, load them after `ci init`: - -```yaml -- name: Bootstrap CI state - env: - NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} - NOORM_IDENTITY_NAME: "GitHub CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} - NOORM_CONNECTION_DATABASE: ${{ secrets.DB_NAME }} - NOORM_CONNECTION_USER: ${{ secrets.DB_USER }} - NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_PASSWORD }} - run: npx noorm ci init --name prod - -- name: Load job secrets into vault - env: - API_KEY: ${{ secrets.API_KEY }} - STRIPE_KEY: ${{ secrets.STRIPE_KEY }} - run: | - cat > ./ci-secrets.env <<EOF - API_KEY=$API_KEY - STRIPE_KEY=$STRIPE_KEY - EOF - npx noorm ci secrets --file ./ci-secrets.env - rm ./ci-secrets.env - -- name: Apply changes - run: npx noorm change ff -``` +## How do I load secrets for one job? -### GitLab CI (Prod) -```yaml -migrate: - stage: deploy - image: node:22 - script: - - npm ci - - npx noorm ci init --name prod - - npx noorm change ff - variables: - NOORM_IDENTITY_PRIVATE_KEY: $NOORM_CI_KEY - NOORM_IDENTITY_NAME: "GitLab CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: $DB_HOST - NOORM_CONNECTION_DATABASE: $DB_NAME - NOORM_CONNECTION_USER: $DB_USER - NOORM_CONNECTION_PASSWORD: $DB_PASSWORD -``` - - -### Shell Script (Prod, with locking) +`ci secrets` reads `KEY=value` lines from a dotenv file into the active config's secrets, in the runner's state file. They outrank a vault secret with the same key. Existing keys are skipped unless you pass `--overwrite`: ```bash -#!/bin/bash -set -e - -export NOORM_IDENTITY_PRIVATE_KEY=$NOORM_CI_KEY -export NOORM_IDENTITY_NAME="CI Bot" -export NOORM_IDENTITY_EMAIL="ci@example.com" -export NOORM_CONNECTION_DIALECT=postgres -export NOORM_CONNECTION_HOST=$DB_HOST -export NOORM_CONNECTION_DATABASE=$DB_NAME -export NOORM_CONNECTION_USER=$DB_USER -export NOORM_CONNECTION_PASSWORD=$DB_PASSWORD - -noorm ci init --name prod - -# Prevent concurrent migrations. `change ff` releases the lock itself once -# the last change lands, so the trap tolerates an already-released lock. -noorm lock acquire -trap 'noorm lock release || true' EXIT - -noorm change ff -noorm db explore --json +printf 'STRIPE_KEY=%s\n' "$STRIPE_KEY" > ci-secrets.env +noorm ci secrets --file ci-secrets.env +rm ci-secrets.env ``` -## Reference +## What bites -### Environment Variables - -| Variable | Required for | Description | -|---|---|---| -| `NOORM_IDENTITY_PRIVATE_KEY` | Prod CI | X25519 private key, hex PKCS8 DER (96 hex chars) | -| `NOORM_IDENTITY_NAME` | Prod CI | Display name (e.g. `"GitHub CI"`) | -| `NOORM_IDENTITY_EMAIL` | Prod CI | Email address | -| `NOORM_CI_CONFIG_NAME` | Optional | Default config name for `ci init` | -| `NOORM_YES` | Optional | Truthy value stands in for `--yes` on every command. Truthy is any non-empty value except `0` and `false` | -| `NOORM_CONNECTION_DIALECT` | Always | `postgres`, `mysql`, `sqlite`, or `mssql` | -| `NOORM_CONNECTION_HOST` | Dialect-specific | Host | -| `NOORM_CONNECTION_PORT` | Dialect-specific | Port | -| `NOORM_CONNECTION_DATABASE` | Always | Database name / file path | -| `NOORM_CONNECTION_USER` | Dialect-specific | Username | -| `NOORM_CONNECTION_PASSWORD` | Dialect-specific | Password | - - -### `ci` Commands at a Glance - -| Command | Runs on | Purpose | +| Symptom | Cause | Fix | |---|---|---| -| `noorm ci identity new` | Developer machine | Mint a CI keypair + print env block. No DB contact. | -| `noorm ci identity enroll --yes` | Developer machine (with vault access) | Enroll a CI identity in a target DB; grant vault access. Idempotent. `--yes` required. | -| `noorm ci init` | CI runner | Bootstrap ephemeral `state.enc` from env; create & activate a config. | -| `noorm ci secrets --file <path>` | CI runner (after `ci init`) | Batch-load secrets from dotenv into the active config. | - - -### Exit Codes - -| Code | Meaning | -|---|---| -| `0` | Success — everything the caller asked for happened | -| `1` | Total failure — the operation ran and no unit of work succeeded | -| `2` | Bad invocation — a missing or malformed flag or env var, or a named target (config, file, change) that does not exist. Nothing was attempted | -| `3` | Partial — some units of work succeeded and some failed. The target is in a mixed state and re-running is not automatically safe | +| `ci init` exits 2: `Missing or invalid environment variables` | A secret is not passed to the step, or the key is not 96 hex characters | Map the secret into the step's `env`. Copy the full value `enroll` printed | +| `ci init` exits 1: `State already exists at … Use --force to overwrite.` | A cached or committed `.noorm/state/state.enc` is in the workspace | `ci init --force` backs it up first. At a terminal it also needs `--yes` | +| `ci identity enroll` exits 1: `About to grant irrevocable vault access` | `--yes` is missing, and sealing a vault key cannot be undone | Check the fingerprint it prints, then rerun with `--yes` | +| `change ff` exits 1: `requires confirmation` | A stage named like the config (`prod`) has `protected: true`, which caps `user` at `operator` | Set `NOORM_YES=1` on the deploy step | +| Tests fail: `Config "ci" does not have isTest: true` | `ci init` leaves `isTest` false, and an environment-only config has it false too | `ci init --test`. `ci identity new` mints a throwaway identity for it | +| `Secret "STRIPE_KEY" not found (searched: config-local, global-local, vault)` | A template reads a secret this job does not have | Use prod CI for vault secrets, or `ci secrets` for job-scoped ones | +| `ci secrets` exits 3 | Some keys were stored and some failed | Fix the errors it lists, then rerun. Stored keys are skipped | -These four codes are shared by every noorm command, not just `ci`. `2` and `3` are what let a pipeline tell "you named something that isn't there" (retry is pointless) from "half of it landed" (a human needs to look). +## Next -## See Also -- [Identity Management](../../cli/identity.md) — keypair model and env-var identity -- [Vault](../environments/vault.md) — how team secrets are encrypted and shared -- [Secrets](../environments/secrets.md) — the three-layer secret resolution hierarchy -- [CI Architecture](../../dev/ci.md) — developer-focused notes on the CI flow +- [Non-interactive operation](/guide/automation/non-interactive): what `--yes` answers and what still refuses +- [Deploying an application](/guide/deployment): schema once per release, the app on every replica +- [Vault](/guide/environments/vault): how team secrets are encrypted and shared +- [Exit codes](/headless#exit-codes): the 0, 1, 2, 3 contract every command shares diff --git a/docs/guide/automation/mcp.md b/docs/guide/automation/mcp.md index 7e6fa823..b234f542 100644 --- a/docs/guide/automation/mcp.md +++ b/docs/guide/automation/mcp.md @@ -1,140 +1,95 @@ # MCP (AI Agent Integration) -noorm runs an [MCP](https://modelcontextprotocol.io/) server over stdio. AI coding agents connect to it and get access to schema inspection, queries, changes, and config management. - - -## Quick Start +Let an AI agent explore and query a database through noorm, limited to what the `agent` role on each config allows. ```bash -# Generate config for your agent (default: Claude Code) noorm mcp init - -# Or for Cursor -noorm mcp init --agent cursor ``` -This writes (or extends) the agent's MCP configuration file. The agent picks up the noorm server on next launch. - -Pair it with the skill, which teaches the agent noorm's conventions rather than its tools: - -```bash -npx skills add noormdev/noorm/skills ``` - -MCP and the skill solve different halves of the same problem — MCP is what the agent can *do*, the skill is what it *knows*. An agent with MCP but no skill can query your schema and still write SDK code that wraps everything in `try`/`catch` this codebase never uses. See [Installation](/getting-started/installation#install-the-agent-skill-optional). - - -## How It Works - -The MCP server wraps noorm's internal RPC registry (the same command set behind the TUI) over stdio transport. Your agent connects, calls `noorm_help` to list available commands, then executes them through `run_noorm_cmd`. - -``` -┌─────────────┐ stdio ┌─────────────────┐ RPC ┌──────────┐ -│ AI Agent │ ◄──────────► │ noorm mcp serve │ ◄──────────► │ Database │ -└─────────────┘ └─────────────────┘ └──────────┘ +Created /home/dev/order-to-cash/.mcp.json ``` -The server stays alive as long as the agent's process keeps stdin open. - - -## Available Tools - -The server exposes two MCP tools: - -| Tool | Purpose | -|------|---------| -| `run_noorm_cmd` | Execute a noorm command (connect, sql, overview, list_configs, etc.) | -| `noorm_help` | List available commands and their parameter schemas | - -Commands load from the RPC registry at runtime, so the tool list matches whatever noorm version you have installed. - - -## Setup by Agent - -### Claude Code - -```bash -noorm mcp init -``` - -Writes to `.mcp.json` in the project root. Claude Code reads this on startup. - -### Cursor - -```bash -noorm mcp init --agent cursor -``` - -Writes to `.cursor/mcp.json`. - - -## Manual Configuration - -If you need to configure the server manually (e.g., for a custom agent), add this to your MCP config: - ```json +// .mcp.json { "mcpServers": { "noorm": { "command": "noorm", - "args": ["mcp", "serve"] + "args": [ + "mcp", + "serve" + ] } } } ``` -This is the exact entry `noorm mcp init` writes, so the two routes produce the same server. +That is Claude Code's file. `--agent cursor` writes `.cursor/mcp.json`. Other MCP clients take the same entry by hand. An existing file keeps its other entries, and `init` prints `Extended`. Pair it with the [agent skill](/getting-started/installation#install-the-agent-skill-optional): `npx skills add noormdev/noorm/skills`. + -The server uses stdio transport — no ports, no HTTP. +## How does a request reach the database? -## What the Agent Can Do +An agent holds `viewer` on a config named `orders`. Every command passes a policy check first: -Once connected, your agent can: +```mermaid +sequenceDiagram + participant Agent + participant Server + participant Policy + participant DB as Database + Agent->>Server: connect orders + Server-->>Agent: connected, role viewer + Agent->>Server: sql SELECT from sales_order + Server->>Policy: SELECT is sql:read + Policy-->>Server: allow + Server->>DB: SELECT from sales_order + DB-->>Server: rows + Server-->>Agent: rows as JSON + Agent->>Server: sql DELETE from sales_order + Server->>Policy: DELETE is sql:write + Policy-->>Server: deny + Server-->>Agent: error, database untouched +``` -- **Explore schemas** — `overview`, `list`, `detail` for tables, columns, foreign keys, indexes -- **Run SQL** — `sql` executes a query and returns structured results -- **Manage connections** — `list_configs`, `connect`, `disconnect`, `status` -- **Apply changes** — `change_run`, `change_ff`, `change_revert`, `change_history` -- **Build schemas** — `run_build`, `run_file` +The server offers two tools: `noorm_help` lists commands and their parameters, and `run_noorm_cmd` runs one as `{ command, config?, payload? }`. -Config resolution and identity attribution work the same as the CLI. Access control does not: every config-scoped command is gated by the config's **`agent` role** before its handler ever runs, and there is no confirmation flow — the agent gets an answer, not a prompt. +## Which commands can the agent run? -## Access Roles -Every config declares a role per channel: `access: { user, agent }`. The `agent` role decides what an AI agent can do to that config — independently of what a human gets in the CLI/TUI. +Each command needs one permission, and the agent's role on the config decides if it has it: -The channel names *who is driving*, not which binary was invoked. An agent that gets refused here and shells out to `noorm` on the command line is still an agent, and gets the same `agent` role: the CLI detects the harness it was spawned from. Raising or lowering `agent` therefore governs both routes at once. See [Access Roles](/guide/environments/configs#access-roles) for the detection rules and the `NOORM_CHANNEL` override. +| Commands | Permission | `viewer` | `operator` | `admin` | +|---|---|---|---|---| +| `list_configs`, `connect`, `disconnect`, `status` | none | yes | yes | yes | +| `overview`, `list`, `detail`, `change_history` | `explore` | yes | yes | yes | +| `sql` with `SELECT`, or `SHOW` and `DESCRIBE` on MySQL | `sql:read` | yes | yes | yes | +| `sql` with `INSERT`, `UPDATE`, `DELETE` | `sql:write` | no | yes | yes | +| `sql` with DDL | `sql:ddl` | no | no | yes | +| `run_build`, `run_file`, `change_run`, `change_ff`, `change_revert` | `run:*`, `change:*` | no | no | yes | -A config that never declared `access` gets `{ user: 'admin', agent: 'viewer' }`. Agents can explore and read on a fresh project without any setup; anything that writes needs you to raise the `agent` role on purpose. +`sql` is classified by what the statement does, and `EXPLAIN` takes the class of the statement it wraps. A config with no `access` gives agents `viewer`. -| Role | What the agent can do | -|------|------------------------| -| `viewer` | Explore schema, run read-only SQL (`SELECT`, `SHOW`, `DESCRIBE`) | -| `operator` | Everything `viewer` can, plus SQL writes (`INSERT`/`UPDATE`/`DELETE`) — DDL and destructive commands (`change_run`, `run_build`, etc.) are still out of reach | -| `admin` | Writes, DDL, changes, builds, all frictionless. Permissions the matrix marks `confirm` even for `admin` (`db:teardown`, `db:destroy`, `config:write`) still deny on this channel | -| `false` | Invisible — the config does not exist on this channel | +`access.agent: false` hides a config: `list_configs` omits it, and `connect` fails with the error an unknown name produces, `Config "orders" not found`. -Raw SQL is classified by what the statement actually does (`sql:read` / `sql:write` / `sql:ddl`), not by which command the agent called — an agent with `agent: viewer` gets a `SELECT` through but a same-shaped `INSERT` denied. `EXPLAIN` takes the class of the statement it wraps, because `EXPLAIN ANALYZE` runs that statement: `EXPLAIN SELECT` is a read, `EXPLAIN DELETE` is a write. +An agent that shells out to `noorm` after a refusal stays on the `agent` channel ([how the channel is chosen](/guide/environments/configs#which-channel-is-a-process-on)). -The matrix's `confirm` cells never prompt on this channel — they resolve straight to **deny**. An agent typing its own confirmation phrase would be theater, not a safeguard, and on the CLI it would need only `--yes`. If an agent legitimately needs to run changes on a database, give that config `agent: 'admin'` — reserve it for configs where that's an acceptable risk (a disposable dev database, say), not production. -```yaml -# Shape of the config's `access` field — set via `noorm ui` → Config → Edit, -# or by editing the JSON before `config import` (see Config Sharing) -access: - user: admin # what you get in the CLI/TUI - agent: viewer # what any connected agent gets -``` +## What bites -## Invisible Configs +| Symptom | Cause | Fix | +|---|---|---| +| `"sql:write" is not allowed on config "orders" (role: viewer).` | The default agent role is `viewer` | Raise `access.agent` for that config in the TUI | +| `"change:ff" on config "orders" requires confirmation from a human.` | A `confirm` cell is deny for agents, and `--yes` does not change that | Give a disposable database `agent: admin`, or run the command yourself | +| Agent still refused after you raised its role | A session keeps the config it loaded at `connect` | Have the agent call `connect` again, or restart the server | -Set `access.agent: false` to hide a config from agents entirely — over MCP it never appears in `list_configs` or `status`, and `connect` fails with the same error an unknown config name would produce. `noorm config list` filters it the same way when an agent runs it. An agent enumerating configs cannot tell the difference between "doesn't exist" and "exists but is off-limits." +## Next -## Security -The MCP server uses the identity and config of the shell session that spawned it. It has no way to escalate privileges beyond what `noorm` itself can do, and it cannot escalate past the `agent` role a config was given — there is no `--force` override on this channel, and shelling out to the CLI reaches the same role rather than the human's. +- [Access roles](/guide/environments/configs#access-roles): the full role matrix +- [MCP commands](/headless#mcp): `mcp init` and `mcp serve` flags diff --git a/docs/guide/automation/non-interactive.md b/docs/guide/automation/non-interactive.md index 9d5b0743..f6368818 100644 --- a/docs/guide/automation/non-interactive.md +++ b/docs/guide/automation/non-interactive.md @@ -1,138 +1,67 @@ # Non-interactive operation -CI runners, deploy scripts, and AI subagents don't get to "press Enter". -noorm's CLI defaults to assuming a human is at the keyboard for a -handful of commands; the universal `--yes` / `NOORM_YES` opt-in tells -those commands to stop asking. +Run noorm in CI, a deploy script, or an agent subprocess, where nobody can press Enter. -`--yes` does not mean "do anything regardless." It means: "use defaults, -don't prompt, fail loudly if a non-interactive path doesn't exist for -this command." +```bash +# bootstrap.sh +set -euo pipefail +# identity init exits 1 when an identity exists, so guard it +[ -f ~/.noorm/identity.key ] || noorm identity init --name "$CI_USER" --email "$CI_EMAIL" +# init exits 1 when .noorm/ exists, which a CI checkout of a project has +[ -d .noorm ] || noorm init --yes +noorm run build # connection comes from NOORM_CONNECTION_* variables +noorm change ff +``` -## The opt-in +This scaffolds a project if there is none, then builds it. -### Flag form +## What does `--yes` answer? -Every TTY-gated command accepts `--yes` (alias `-y`): - noorm init --yes +`--yes` (or `-y`) answers a confirmation for one command. `NOORM_YES` answers for every command: any non-empty value counts except `0` and `false`, in any letter case. The flag beats `NOORM_YES=0`. +A run with no terminal stops at the first gate it cannot answer. The role cells come from [Access roles](/guide/environments/configs#access-roles), and on the `agent` channel `confirm` is `deny`: -### Environment form +```mermaid +flowchart TD + A["noorm command"] --> B{"TTY-only command?<br/>sql repl, settings edit"} + B -->|yes| X2["exit 2<br/>needs a terminal"] + B -->|no| C{"role cell"} + C -->|deny| X1["exit 1<br/>no flag opens it"] + C -->|allow| R["runs"] + C -->|confirm| D{"--yes or<br/>NOORM_YES?"} + D -->|yes| R + D -->|no| X3["exit 1<br/>requires confirmation"] +``` -`NOORM_YES` is equivalent and applies to every command at once. Useful in -CI where you set it once at the top of the pipeline. - export NOORM_YES=1 - noorm init - noorm change ff - noorm db reset +## Which prompts have a flag? -Truthy values: `1`, `true`, `yes`, `TRUE`, any non-empty string. Falsy -values: `0`, `false`, empty string, unset. The comparison for `false` is -case-insensitive. -The `--yes` flag wins over `NOORM_YES=0`. Use that if you want to override -a globally-set falsy default. +| Command | Without a terminal | +|---|---| +| `noorm init` | `--yes`, after `noorm identity init` has written `~/.noorm/identity.*` | +| `change run`, `revert`, `rm`, `edit`, `add` | Pass the name or description as the argument, or exit 1 | +| `db transfer` with a `.dtzx` file | `--passphrase`, or exit 2 | +| `sql repl` | `--yes` exits 2. Use `noorm sql query "SELECT 1"` or `--file query.sql` | +| `settings edit` | `--yes` exits 2. Edit `.noorm/settings.yml`, then `noorm settings build` | +| `settings secret` | `--yes` exits 2. Edit `secrets:`, and set values with `noorm secret set <key> --stdin` | -## What `--yes` does per command +## What bites -Some commands have a genuine non-interactive equivalent — for those, -`--yes` succeeds. The rest are interactive by design; for them, `--yes` -errors fast and points at the headless alternative. +| Symptom | Cause | Fix | +|---|---|---| +| `identity init --force` still exits 1 | Replacing needs the `--yes` flag. `NOORM_YES` does not count, because a new keypair orphans every `state.enc` | Pass `--yes` | +| `db reset` exits 1: `This is a destructive operation. Pass --yes to confirm.` | It needs `--yes` or `NOORM_YES` even for `admin` | Pass it | -### Accepts `--yes` (proceeds) -| Command | Behavior with `--yes` | -|---------|-----------------------| -| `noorm init` | Skips all prompts. Requires an existing identity at `~/.noorm/identity.{key,pub,json}`. Errors with a bootstrap hint if missing. | +## Next -### Refuses `--yes` (redirect hint) - -| Command | Redirect | -|---------|----------| -| `noorm sql repl` | Use `noorm sql query "<SQL>"` or `noorm sql query --file query.sql`. | -| `noorm settings edit` | Edit `settings.yml` directly; run `noorm settings build` to validate. | -| `noorm settings secret` | Edit the `secrets:` section of `settings.yml` directly. For values, use `noorm secret set <key> <value>`. | - -These commands exit 2 (bad invocation) with the redirect message on -stderr. Same exit code as their TTY refusal, so a pipeline sees one -code whichever way it hit the gate. - - -### Already non-interactive (no `--yes` needed) - -These accept explicit flags and don't gate on a TTY: - -- `noorm identity init --name "..." --email "..."` — bootstraps the identity used by `noorm init --yes`. -- `noorm config import <path>` — wizard-free config setup. -- `noorm ci init` / `ci secrets` / `ci identity new` — the dedicated CI bootstrap surface. - -One exception inside that surface: `noorm ci identity enroll` **does** need -`--yes` (or `NOORM_YES=1`). It ends in a `vault:propagate` grant, which the -matrix marks `confirm` for every role that holds it, and sealing the vault -key to a public key cannot be undone. - - -## CI bootstrap pipeline - -End-to-end happy path: from a clean machine to a usable noorm project, -with no prompts: - - #!/usr/bin/env bash - set -euo pipefail - - # 1. Identity (once per machine; exits 1 if one already exists). - noorm identity init \ - --name "$CI_USER" \ - --email "$CI_EMAIL" - - # 2. Project scaffold (sql/, changes/, settings.yml, state.enc). - cd "$PROJECT_ROOT" - noorm init --yes - - # 3. Connection config — env-only, no wizard. - export NOORM_CONNECTION_DIALECT=postgres - export NOORM_CONNECTION_HOST="$DB_HOST" - export NOORM_CONNECTION_DATABASE="$DB_NAME" - export NOORM_CONNECTION_USER="$DB_USER" - export NOORM_CONNECTION_PASSWORD="$DB_PASSWORD" - - # 4. Build schema + apply pending changes. - noorm run build - noorm change ff - -Step 1 is not idempotent, and under `set -e` it aborts the script if an -identity already exists. Replacing one takes `--force --yes` together, -mints a fresh keypair, and leaves any state encrypted under the old key -unreadable. Guard it instead of passing `--force` blind: - - [ -f ~/.noorm/identity.key ] || noorm identity init \ - --name "$CI_USER" --email "$CI_EMAIL" - -The raw `--yes` flag is required for a replacement; an ambient -`NOORM_YES` deliberately does not satisfy it. - -If you need a stored config (rather than env-only), use `noorm config -import <path>` with a pre-built JSON file. See -[`docs/headless.md`](../../headless.md) for the full env-only matrix. - - -## When `--yes` is NOT enough - -`--yes` covers prompts. It does NOT relax safety gates that exist for -destructive operations: - -- `noorm db drop`, `noorm db reset`, `noorm db teardown` still require `--yes` per their own contracts. `NOORM_YES=1` does count toward those gates. -- Every destructive command is also gated by the config's access role (`viewer`/`operator`/`admin` — see [Access Roles](../../headless.md#access-roles)). `--yes`/`NOORM_YES=1` only satisfies a `confirm` cell; it cannot open a `deny` cell. A `viewer`- or `operator`-role config still refuses `db teardown` no matter how many flags you pass. -- Vault and identity operations that need a private key still need the key — either on disk at `~/.noorm/identity.key` or in `$NOORM_IDENTITY_PRIVATE_KEY`. - -If a command refuses with a useful error in interactive mode, it refuses -with the same error in `--yes` mode. The flag turns prompts off, not -preconditions off. +- [CI/CD Automation](/guide/automation/ci): test CI and prod CI pipelines +- [Exit codes](/headless#exit-codes): the 0, 1, 2, 3 contract diff --git a/docs/guide/changes/add-a-column.md b/docs/guide/changes/add-a-column.md new file mode 100644 index 00000000..2f0018fc --- /dev/null +++ b/docs/guide/changes/add-a-column.md @@ -0,0 +1,120 @@ +--- +title: Add a column to a live database +description: Declare the column once in db/sql/, then add a change that brings existing databases to the same state, backfill included, and that is safe to run twice. +--- + +# Add a column to a live database + +Declare the column in `db/sql/` so fresh databases get it. Add a change so databases that already exist catch up, backfill included. + +The table's file holds the current state. The column is declared once, inline: + +```sql +-- db/sql/01_tables/02_sales_order.sql +CREATE TABLE sales_order ( + party_id integer NOT NULL, + order_no integer NOT NULL, + placed_at timestamptz NOT NULL DEFAULT now(), + status text NOT NULL DEFAULT 'open', -- [!code ++] + CONSTRAINT sales_order_pk + PRIMARY KEY (party_id, order_no), + CONSTRAINT sales_order_isplacedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) +); +``` + +Scaffold the change, then fill in its two files: + +```bash +noorm change add order-status +``` + +```sql +-- db/changes/2026-10-20-order-status/change/001_order-status.sql +ALTER TABLE sales_order ADD COLUMN IF NOT EXISTS status text; + +-- Invoiced orders become 'invoiced', the rest 'open'. Paid orders are +-- marked by the payment change that follows this one. +UPDATE sales_order so +SET status = CASE WHEN EXISTS ( + SELECT 1 FROM invoice_line il + WHERE il.party_id = so.party_id AND il.order_no = so.order_no + ) THEN 'invoiced' ELSE 'open' END +WHERE status IS NULL; + +ALTER TABLE sales_order + ALTER COLUMN status SET DEFAULT 'open', + ALTER COLUMN status SET NOT NULL; +``` + +```sql +-- db/changes/2026-10-20-order-status/revert/001_order-status.sql +ALTER TABLE sales_order DROP COLUMN IF EXISTS status; +``` + +Apply it to every database that predates the column: + +```bash +noorm change ff +``` + +``` +• log Fast-forward success executed=1 skipped=0 failed=0 +• log 2026-10-20-order-status (success) +``` + + +## How it works + + +Each database reaches the same table by one of two paths: + +```mermaid +flowchart TD + F[fresh database] -->|noorm run build| S["db/sql/01_tables/<br/>02_sales_order.sql"] + E[existing database] -->|noorm change ff| C["change/<br/>001_order-status.sql"] + S --> T[sales_order with status] + C --> T +``` + +| File | Runs on | Says | +|---|---|---| +| `db/sql/01_tables/02_sales_order.sql` | fresh databases, through `run build` | what the table is now | +| `change/001_order-status.sql` | each database where the change is pending, through `change ff` | how to get there from the old table, backfill included | +| `revert/001_order-status.sql` | `change revert` | how to undo it | + +`ALTER`, `UPDATE`, and `DROP` live only in `db/changes/`. The file in `db/sql/` never says how the table got to its current shape. + +A database built from `db/sql/` has no record of any change, so its first `change ff` runs every change it finds, this one included. The `IF NOT EXISTS` and the `WHERE status IS NULL` make that second run a no-op. + + +## Options + + +| Flag | Effect | +|---|---| +| `--dry-run` | Render the change to `tmp/` without touching the database | +| `--json` | Print the result as JSON | +| `--config`, `-c` | Run against another config: `noorm change ff -c staging` | +| `--force`, `-f` | Skip the checksum check and run every file of each pending change | +| `NOORM_YES=1` | Allow operations that need confirmation, for scripts and CI | + +Full entry: [`change ff`](/headless#change-ff). + + +## What bites + + +| Symptom | Cause | Fix | +|---|---|---| +| `change ff` fails on a freshly built database: column "status" already exists | The build created the column, and the change ran again because nothing recorded it | Write the change so a second run changes nothing: `IF NOT EXISTS`, and backfill only rows that still need it | +| `run build` on a fresh database fails: column "status" already exists | An `ALTER TABLE` landed in `db/sql/` | Declare the column inline in `CREATE TABLE`; the `ALTER` belongs in the change | +| `SET NOT NULL` fails: the column contains null values | The backfill missed rows | Backfill in the same change, before `SET NOT NULL` | +| `change ff` exits 1: requires confirmation | Your role on that config is `operator` | Set `NOORM_YES=1`, or confirm in `noorm ui` | + + +## Next + + +- [Forward and revert](/guide/changes/forward-revert): apply, revert, and the status of each change +- [Changes overview](/guide/changes/overview): folder anatomy and manifests diff --git a/docs/guide/changes/forward-revert.md b/docs/guide/changes/forward-revert.md index e10837ed..c7abdf2d 100644 --- a/docs/guide/changes/forward-revert.md +++ b/docs/guide/changes/forward-revert.md @@ -1,301 +1,188 @@ -# Forward and Revert Lifecycle +# Forward and revert -Database changes move in two directions: forward when you apply them, backward when you need to undo. Understanding this lifecycle helps you deploy with confidence and recover gracefully when things go wrong. +Apply outstanding changes in order, and roll one back when it goes wrong. - -## Applying Changes - -noorm provides three ways to apply changes, each suited to different situations. - - -### Fast-Forward All Pending - -Apply every pending change in chronological order: - -**TUI:** Navigate to Changes (`g` from home), then press `f` for fast-forward. - -**Headless:** ```bash noorm change ff -noorm change ff --json -``` - -Fast-forward is the workhorse for deployments. It finds every outstanding change (`pending`, `reverted`, or `stale` after a teardown), sorts them by folder name so the `YYYY-MM-DD` prefix puts them in chronological order, and executes each one in sequence. If any change fails, execution stops immediately. - -**JSON output:** -```json -{ - "success": true, - "status": "success", - "executed": 2, - "skipped": 0, - "failed": 0, - "durationMs": 77, - "changes": [ - {"name": "2024-02-01-add-notifications", "direction": "change", "status": "success", "durationMs": 45}, - {"name": "2024-02-15-user-preferences", "direction": "change", "status": "success", "durationMs": 32} - ] -} ``` -Each entry in `changes` also carries a `files` array with one record per SQL file: relative path, checksum, status, duration, and the error message when the file failed. If the `changes/` directory is missing, the payload gains a `warnings` array saying so, because an absent directory otherwise looks identical to an up-to-date database. - - -### Apply a Specific Change - -Run one change by name: - -**TUI:** From the Changes list, select a change and press `r` for run. - -**Headless:** -```bash -noorm change run 2024-02-01-add-notifications ``` - -This is useful when you need to apply changes out of order during development, or re-run a failed change after fixing the underlying issue. - - -### Apply the Next Pending Change - -Apply only the next change in sequence: - -**TUI:** From Changes, press `n` for next. - -**Headless:** -```bash -noorm change next +• log Fast-forward success executed=2 skipped=0 failed=0 +• log 2026-10-22-open-orders (success) +• log 2026-10-23-payment (success) ``` -Useful for stepping through changes one at a time, perhaps while monitoring system behavior between each. - - -## Reverting Changes - -Rolling back requires that your change includes revert scripts in its `revert/` folder. - - -### Revert a Specific Change - -Undo one change: - -**TUI:** From Changes, select an applied change and press `v` for revert. - -**Headless:** ```bash -noorm change revert 2024-02-01-add-notifications +noorm change revert 2026-10-23-payment ``` -Revert scripts execute in forward sequence order, just like change scripts. You design them to undo in reverse—if your change had `001_create-table.sql` and `002_add-indexes.sql`, your revert should have `001_drop-indexes.sql` (undoes the last thing) and `002_drop-table.sql` (undoes the first thing). - - -### Rewind Multiple Changes - -Revert the last N applied changes: - -**TUI:** From Changes, press `w` for rewind, then specify how many. - -**Headless:** -```bash -noorm change rewind 3 ``` - -Rewind walks through your applied changes starting with the newest, reverting each one before moving to the next oldest. It considers only changes currently recorded as `success`, and orders them by when they were applied. Changes applied in the same second, which is everything a single `change ff` applies, tie on that timestamp, so the tracking table's row id breaks the tie and preserves true apply order. - -Pass a change name instead of a count to rewind back to and including that change: - -```bash -noorm change rewind 2024-02-01-add-notifications +• log 2026-10-23-payment reverted (success) ``` +`ff` ran both outstanding changes, oldest first. `revert` ran the `revert/` files of `payment` and set it to `reverted`, so the next `ff` applies it again. -## What Happens During Apply - -When you apply a change, noorm: - -1. **Reads the change folder** - Loads all files from `change/` and determines their types (`.sql`, `.sql.tmpl`, or `.txt` manifest). - -2. **Calculates a checksum** - Creates a combined hash of all files to detect future modifications. - -3. **Checks the database record** - Looks up this change's status to determine if it should run. - -4. **Acquires the config lock** - Stops a second operation on the same config from running concurrently. The lock is released when the change finishes, whatever the outcome. -5. **Resolves manifest references** - For `.txt` files, replaces file references with actual SQL content from your schema directory. +## Which statuses can a change have? -6. **Processes templates** - For `.sql.tmpl` files, runs them through the Eta templating engine. -7. **Executes SQL in sequence** - Runs each file in order (`001_`, `002_`, etc.) against the database. A file whose checksum matches a still-standing success from an earlier attempt is skipped, so a retry only re-runs what actually needs re-running. +A change has five statuses, and `ff` and `next` pick up three of them. -8. **Records the result** - Stores the execution status, checksum, and timestamp in the change tracking tables. - -If any file fails, execution stops, and the files that were never reached are recorded as skipped. On PostgreSQL the whole change runs inside one transaction, so the failure rolls back the DDL and the history rows together and the change leaves no record at all. On MySQL, SQL Server, and SQLite there is no wrapping transaction: whatever already committed stays committed, and the change is recorded as `failed`. - - -## What Happens During Revert - -Reverting follows a similar process but with key differences: - -1. **Verifies the change can be reverted** - A change recorded as `success` or `failed` can be reverted. One that was never applied, is already `reverted`, or went `stale` after a teardown cannot. `--force` skips this check. - -2. **Reads the revert folder** - Loads files from `revert/` instead of `change/`. +```mermaid +stateDiagram-v2 + [*] --> pending : change add + pending --> success : files pass + pending --> failed : a file fails + failed --> success : change run + success --> reverted : revert + reverted --> success : ff or run + success --> stale : db teardown + stale --> success : ff or run +``` -3. **Executes in sequence order** - Files run from lowest to highest (`001_`, `002_`, etc.), same as change scripts. +| Status | Means | `ff` and `next` | +|---|---|---| +| `pending` | no record yet | run it | +| `success` | applied | skip it | +| `failed` | last run failed and was recorded | skip it | +| `reverted` | applied, then rolled back | run it | +| `stale` | `db teardown` dropped what it built | run it | -4. **Updates the records** - Writes its own history record with direction `revert`, then flips the original forward record's status to `reverted`. The revert record's timestamp is what surfaces as `revertedAt`. +`revert` runs from `success` or `failed`, and `db teardown` turns both into `stale`. A revert from `reverted` or `stale` runs nothing and still reports success. A change with no record fails with `not applied`. -After reverting, the change returns to a state where it can be applied again. This is the `reverted` status, which noorm treats as "needs to run" during the next fast-forward. +## How do I apply changes? -## Dry Run Mode -Preview what would happen without touching the database: +| Command | Applies | TUI | +|---|---|---| +| `noorm change ff` | every outstanding change, oldest first | `f` | +| `noorm change next [n]` | the next `n` outstanding changes, default 1 | `n` | +| `noorm change run <name>` | one named change, in or out of order | `r` | -**TUI:** Press `Shift+D` to turn on dry-run mode. It is a global toggle, so every apply and revert stays a dry run until you press it again. +`ff` and `next` stop at the first failing change. Exit `0` is success, `1` failed, `3` partial: some applied before one failed. [CI](/guide/automation/ci) runs these in a pipeline. -**Headless:** - noorm change ff --dry-run - noorm change run 2024-02-01-add-notifications --dry-run - noorm change revert 2024-02-01-add-notifications --dry-run - noorm change next --dry-run - noorm change rewind 3 --dry-run +## What happens when a change runs? -Dry run writes rendered SQL to a `tmp/` folder so you can inspect exactly what would execute. Templates are processed and manifests are resolved, but nothing from your change files runs against the database and nothing is written to the change tracking tables: the dry-run path skips the status lookup, takes no lock, and creates no history record. `ff`, `next`, and `rewind` still read the tracking tables to decide which changes to render. -In human mode the CLI opens with `Dry run: rendering changes to tmp/ (no DB writes)`, labels the summary line `(dry-run)`, and tags each change `dry-run`, so log scrapers and operators can tell it apart from a real apply. With `--json` those lines are suppressed and the result payload carries a `dryRun: true` field instead (`rewind --json` is the exception and omits it): +A change takes the config lock, runs its files in order, and records the outcome. The first failing file stops it. - $ noorm change ff --dry-run --json - { - "success": true, - "status": "success", - "executed": 2, - "skipped": 0, - "failed": 0, - "durationMs": 12, - "changes": [], - "dryRun": true - } +```mermaid +flowchart TD + N{"needs to run?<br/>status, checksums"} -->|no| S["skip"] + N -->|yes| L["take the config lock"] + L --> F["run files in order,<br/>skip passed files"] + F --> P{"every file passes?"} + P -->|yes| OK["record success"] + P -->|no| D{"PostgreSQL?"} + D -->|yes| RB["roll back the DDL<br/>and the records"] + D -->|no| FL["record failed,<br/>skip unreached files"] +``` -This is essential for production deployments. Always preview before applying. +On PostgreSQL the whole change is one transaction, so a failure leaves no record and no applied file. MySQL, SQL Server, and SQLite have no wrapping transaction, so files that passed stay applied. -## Handling Failures +## How do I recover from a failed change? -When a change fails: -1. **Execution stops immediately** - No further files in the change execute. The ones never reached are recorded as `skipped`, with the failing file named as the reason. +Fix the file, then retry: -2. **Status is recorded as failed** - The change shows `failed` in the list. +| | PostgreSQL | MySQL, SQL Server, SQLite | +|---|---|---| +| Status after a failure | none, so `pending` | `failed`, with the error | +| Retry with | `noorm change ff` | `noorm change run <name>` | -3. **Error details are captured** - The failing file name and the database's error message are stored on both the change record and that file's record. +The retry skips files that already passed. If a half-applied run needs manual cleanup first, clean up, then add `--force` to run every file. [History](/guide/changes/history#which-file-broke-the-change) shows which file failed. -4. **Next fast-forward will retry** - Failed changes are outstanding work, so the next `ff` attempt includes them. -On PostgreSQL, steps 2 and 3 do not survive the failure. The change runs inside one transaction, DDL and history rows alike, so a failure rolls all of it back and leaves no record behind. You still see the error in the command's output and exit code, and the next `ff` reruns the change because the tracking tables show it as never applied. MySQL, SQL Server, and SQLite have no wrapping transaction and keep the `failed` record. +## What happens when a change folder cannot be read? -### Investigating Failures +A manifest that names a file no longer in `db/sql/` makes its change unreadable. `ff` and `next` do not step over it: -**TUI:** From Changes, press `h` for history, then select the failed operation to see file-level details. +| The unreadable change | `ff` and `next` | +|---|---| +| never ran, or is `reverted` or `stale` | fail at its position, so later changes do not run out of order. Exit `1`, or `3` when earlier changes applied | +| already applied | keep its status and add a `warnings` entry | -**Headless:** -```bash -noorm change history +``` +✗ log Fast-forward partial executed=1 skipped=1 failed=1 +• log 2026-10-20-order-status (success) +✗ log 2026-10-22-open-orders (failed) +✗ log error: Manifest '/home/dana/orders/db/changes/2026-10-22-open-orders/change/001_rebuild-views.txt' references missing file: 02_views/02_unpaid_invoices.sql ``` -The history shows which specific file failed and includes the error message from the database. On PostgreSQL there is nothing to show, since a failed change rolls its history rows back with the DDL; read the error off the failing command instead. See [History](/guide/changes/history) for detailed debugging workflows. - - -### Recovering from Failures - -Option 1: **Fix and retry** -- Edit the SQL file to fix the issue -- Run `change ff` again (the change will be picked up automatically) - -Option 2: **Manual cleanup** -- If partial SQL executed, you may need to manually clean up -- Then run with `--force` to skip the checksum check: +`skipped=1` is `2026-10-23-payment`, which never ran. - noorm change run 2024-02-01-add-notifications --force +## How do I roll back? -## Common Workflows +| Command | Reverts | TUI | +|---|---|---| +| `noorm change revert <name>` | one change | `v` | +| `noorm change rewind <n>` | the last `n` applied changes, newest first | `w` | +| `noorm change rewind <name>` | that change and every change applied after it | `w` | -### Deploying to Production +`revert` runs the `revert/` files in filename order, writes a history record with direction `revert`, then sets the original record to `reverted`. `rewind` considers only `success` changes and orders them by apply time, breaking same-second ties by history id. - # Preview what will run - noorm change ff --dry-run - # Review the rendered SQL in tmp/ - # Then apply for real - noorm change ff +## How do I preview a run? -### Rolling Back a Bad Deploy +`--dry-run` renders every file, templates and manifests included, to `tmp/` and runs nothing: ```bash -# See current state -noorm change list - -# Revert the problematic change -noorm change revert 2024-02-15-broken-change +noorm change ff --dry-run ``` - -### Starting Fresh After Teardown - -After running `db teardown`, all changes are marked as `stale`. The next fast-forward re-applies everything: - -```bash -noorm db teardown -y -noorm run build # Rebuild base schema -noorm change ff # Re-apply all changes +``` +• log Dry run: rendering changes to tmp/ (no DB writes) +• log Fast-forward (dry-run) success executed=2 skipped=0 failed=0 dryRun=true +• log 2026-10-22-open-orders (success, dry-run) +• log 2026-10-23-payment (success, dry-run) ``` +A dry run takes no lock and writes no status or history. `ff`, `next`, and `rewind` still read the tracking tables to choose what to render. In the TUI, `Shift+D` toggles it. -### Testing Revert Scripts +::: warning Rendered secrets +Dry-run files hold every secret a template resolved, in plaintext. noorm writes them owner-only, and `noorm init` does not add `tmp/` to `.gitignore`. +::: -During development, verify your revert logic works: -```bash -# Apply the change -noorm change run 2024-02-01-add-notifications +## What decides whether a change runs? -# Revert it -noorm change revert 2024-02-01-add-notifications -# Apply again to confirm round-trip works -noorm change run 2024-02-01-add-notifications -``` +noorm compares checksums of every file in `change/`, plus every file a manifest lists. -This apply-revert-apply cycle catches revert scripts that are missing steps or have incorrect SQL. +| The change is | `ff` and `next` | `change run <name>` | +|---|---|---| +| new, `reverted`, or `stale` | runs | runs | +| `failed` | skips | runs the files that have not passed | +| `success`, files unchanged | skips | skips | +| `success`, a file edited | skips | runs the edited files | +`--force` (TUI: `Shift+F`) runs every file and ignores file history. On `revert` it skips the status check. [`change ff`](/headless#change-ff) lists the other flags. -### CI/CD Pipeline -A typical deployment script: +## What bites -```bash -#!/bin/bash -set -e -# Build base schema (idempotent) -noorm run build - -# Apply pending changes -noorm change ff - -# Verify the result -noorm db explore --json -``` +| Symptom | Cause | Fix | +|---|---|---| +| `ff` reports `executed=0` while a change shows `failed` | `ff` skips a recorded `failed` change | Fix the file, then `noorm change run <name>` | +| `ff` fails after `run build`: already exists | A build records no changes, and `db teardown` marks applied ones `stale`, so `ff` replays them over the built schema | Write each change to be safe to run twice ([Add a column](/guide/changes/add-a-column)) | +| `ff` warns `Changes directory not found` and runs nothing | `settings.yml` has no `paths`, which now resolves to `./db/changes` | Add `paths: { sql: ./sql, changes: ./changes }` | +| Exit `1`: `requires confirmation` or `(role: viewer)` | Your role on that config is `operator` or `viewer` | `NOORM_YES=1` for `operator`. `viewer` is denied: [Access roles](/headless#access-roles) | +| Exit `1`: `Lock for '<config>' held by ...` | Another operation holds the config lock | Wait, or check `noorm lock status` | -The build step handles the initial schema. The fast-forward applies any changes that have been merged since the last deploy. +## Next -## What's Next? -- [History](/guide/changes/history) - Track and debug execution history -- [Changes](/guide/changes/overview) - Change structure and file types -- [Teardown](/guide/database/teardown) - How teardown affects change status +- [History](/guide/changes/history): what each run recorded +- [Changes](/guide/changes/overview): folder layout and manifests +- [Add a column](/guide/changes/add-a-column): a change with a backfill +- [Teardown](/guide/database/teardown): how teardown makes changes `stale` diff --git a/docs/guide/changes/history.md b/docs/guide/changes/history.md index e4f6b6b9..5e79e5df 100644 --- a/docs/guide/changes/history.md +++ b/docs/guide/changes/history.md @@ -1,134 +1,105 @@ -# Execution History +# Execution history -Every database operation noorm performs is recorded. When a change runs, a build executes, or files get deployed, that execution goes into the history log with who ran it, when, how long it took, and whether it succeeded. +Find out who ran a change, when, and which file broke it. -A build, dir, files or file run where every file is unchanged writes no operation, so it appears neither in history nor in Recent Activity. - -This gives you an audit trail for compliance, a debugging tool for failures, and a reference for understanding what happened to your database. - - -## What Gets Tracked - -noorm records three types of operations: - -| Type | Description | Trigger | -|------|-------------|---------| -| **change** | Change execution | `change run`, `change ff`, `change next`, `change revert`, `change rewind` | -| **build** | Full schema rebuild | `run build` | -| **run** | Individual file execution | `run file`, `run directory` | +```bash +noorm change history +``` -Each record captures: +``` +Execution History: 4 records + 2026-10-20-order-status - success (10/21/2026, 1:40:29 AM) + 2026-10-21-invoice - failed (10/21/2026, 1:40:03 AM) + 2026-10-20-order-status - reverted (10/21/2026, 1:40:03 AM) + 2026-10-19-sales-order - success (10/21/2026, 1:40:03 AM) +``` -- **Name** - Which change, build, or file was executed -- **Direction** - Forward (`change`) or rollback (`revert`) -- **Status** - `pending` while the operation is in flight, then `success`, `failed`, `reverted`, or `stale` -- **Who** - The identity that ran it (name and email) -- **When** - Timestamp of execution -- **Duration** - How long it took in milliseconds -- **Checksum** - Hash of the executed files -- **Error** - If failed, what went wrong +Newest first, one line per operation. The first line is the revert of `2026-10-20-order-status`. The third is its original apply, now flagged `reverted`. The text omits direction, and `--json` carries it. -File-level details are also preserved. For a change with three SQL files, you can see exactly which file succeeded, which failed, and which were skipped. +## What does noorm record? -## Viewing History in the TUI -Navigate to history from the changes menu. Press `g` from home, then `h` for history. +Each operation is one `change` row, with one `executions` row per file that was not already applied: -``` -+- Execution History --------------------------------------------+ -| | -| Total: 47 Changes: 32 Builds: 10 Runs: 5 | -| Success: 45 Failed: 2 | -| | -| > [OK] [CHANGESET] 2025-01-15-add-verification 2 hours ago | -| [OK] [BUILD] schema-build 3 hours ago | -| [OK] [CHANGESET] 2025-01-14-add-user-roles yesterday | -| [ERR] [RUN] seed-data.sql yesterday | -| [OK] [CHANGESET] 2025-01-10-init-schema last week | -| | -| +------------------------------------------------------------+| -| | 2025-01-15-add-verification || -| | By: Alice <alice@example.com> || -| | Duration: 1.24s || -| | Press Enter to view file details || -| +------------------------------------------------------------+| -| | -| [Enter] View Files [Esc] Back | -+----------------------------------------------------------------+ +```mermaid +erDiagram + change ||--o{ executions : "is broken into" + change { + int id PK + text name + text change_type + text direction + text status + } + executions { + int id PK + int change_id FK + text filepath + text status + text skip_reason + } ``` -The summary bar shows totals by type. Color-coded status indicators make it easy to spot failures: +Rows also carry who ran it (`Name <email>`), when, how long, a checksum, and the error text. MySQL and SQLite name the tables `__noorm_change__` and `__noorm_executions__`. [Execution](/guide/sql-files/execution#what-gets-tracked) lists every column. -- Green `[OK]` - Success -- Red `[ERR]` - Anything else +| `change_type` | Written by | In `change history` | +|---|---|---| +| `change` | `change run`, `ff`, `next`, `revert`, `rewind` | yes | +| `build` | `run build` | no, TUI only | +| `run` | `run file`, `run dir`, `run files` | no, TUI only | -The list only separates success from not-success. The per-file detail view adds a yellow `[-]` for skipped files. +A build or run where every file is unchanged writes no operation. -Arrow keys move through the list. The detail panel at the bottom shows who ran the selected operation and how long it took. +## Which file broke the change? -## History Detail View -Press Enter on any record to see file-level execution details: +```bash +noorm change history-detail 2026-10-24-order-line +``` ``` -+- File Executions (2025-01-15-add-verification) ----------------+ -| | -| Total: 3 Success: 2 Failed: 1 | -| | -| > [OK] 001_create-tokens-table.sql (0.4s) | -| [OK] 002_add-user-column.sql (0.3s) | -| [ERR] 003_add-index.sql (0.5s) | -| | -| +------------------------------------------------------------+| -| | File Details || -| | Path: changes/2025-01-15-.../change/001_create-tokens... || -| | Checksum: 8f4a2b3c9d1e... || -| +------------------------------------------------------------+| -| | -| [Esc] Back | -+----------------------------------------------------------------+ +• log History for: 2026-10-24-order-line (1 operation(s)) +• log + [15] change — failed at 10/21/2026, 1:46:16 AM by Dana Reyes <dana@example.com> +• log duration: 27ms +• log error: 002_order-line-index.sql: [Err 1] no such column: nope +• log • 001_order-line.sql — success | 2ms +• log • 002_order-line-index.sql — failed | 1ms | error: [Err 1] no such column: nope +• log • 003_line-index.sql — skipped | skipped: 002_order-line-index.sql failed: [Err 1] no such column: nope ``` -For failed files, the detail panel shows the error message: +`[15]` is the operation id. The failing file carries the database's error, and every file after it is `skipped` with the failure as the reason. After you fix `002` and run `noorm change run 2026-10-24-order-line`, a second operation lists only `002` and `003`. File `001` already passed, so it leaves no row. -``` -| +------------------------------------------------------------+| -| | File Details || -| | Path: changes/2025-01-15-.../change/003_add-index.sql || -| | Checksum: 3b7c8d2a4f5e... || -| | Error: relation "users" does not exist || -| +------------------------------------------------------------+| -``` +| File status | TUI | Meaning | +|---|---|---| +| `success` | `[OK]` | ran without error | +| `failed` | `[ERR]` | threw an error | +| `skipped` | `[-]` | an earlier file failed, so this one was never reached | -This is invaluable for debugging. You can see exactly which file failed, what the error was, and what files succeeded before it. +`--count` sets how many operations to show, default 10. How to retry is in [Forward and revert](/guide/changes/forward-revert#how-do-i-recover-from-a-failed-change). -## Headless Mode +## How do I read history in the TUI? -For CI/CD pipelines and scripts, use the headless command: -```bash -noorm change history -``` +Press `g` from home, then `h`. [TUI](/tui#changes-list) shows the screens. -Unlike the TUI screen, this command returns change operations only. Builds and file runs live in the same tracking tables but do not appear here. +| Screen | Shows | Keys | +|---|---|---| +| History | the latest 200 operations of every type, with totals by type | `Enter` opens files, `e` shows the full error, `Esc` goes back | +| File executions | one operation's files, with path, checksum, and error or skip reason | `Enter` shows the full error, `Esc` goes back | -Output: +Rows carry `[OK]` for `success` and `[ERR]` for any other status, and a type tag: `[CHANGESET]`, `[BUILD]`, or `[RUN]`. Times show as `2 hours ago`. -``` -Execution History: 20 records - 2025-01-15-add-verification - success (1/15/2025, 2:30:00 PM) - 2025-01-14-add-user-roles - reverted (1/14/2025, 4:15:00 PM) - 2025-01-10-init-schema - success (1/10/2025, 9:05:00 AM) -``` -With JSON output for parsing: +## How do I script against history? + ```bash -noorm change history --json +noorm change history --json --count 100 ``` ```json @@ -136,201 +107,39 @@ noorm change history --json "success": true, "history": [ { - "id": 42, - "name": "2025-01-15-add-verification", + "id": 13, + "name": "2026-10-23-payment", "direction": "change", "status": "success", - "executedAt": "2025-01-15T14:30:00.000Z", - "executedBy": "Alice <alice@example.com>", - "durationMs": 1240, + "executedAt": "2026-10-21T05:41:50.000Z", + "executedBy": "Dana Reyes <dana@example.com>", + "durationMs": 19, "errorMessage": null, - "checksum": "8f4a2b3c9d1e..." - }, - { - "id": 41, - "name": "2025-01-14-add-user-roles", - "direction": "revert", - "status": "success", - "executedAt": "2025-01-14T16:15:00.000Z", - "executedBy": "CI Pipeline <ci@company.com>", - "durationMs": 320, - "errorMessage": null, - "checksum": "3b7c8d2a4f5e..." + "checksum": "73fbc0894afc..." } ] } ``` -The `id` is what `noorm change history-detail` and the TUI drill-down use to fetch the per-file records for an operation. - -Limit the number of records (default 20): - -```bash -noorm change history --count 50 -``` - - -## Filtering by Type - -The TUI history screen shows all three operation types by default. The summary bar helps you understand the distribution: - -``` -Total: 47 Changes: 32 Builds: 10 Runs: 5 -``` - -Type indicators make scanning easy: - -| Indicator | Color | Meaning | -|-----------|-------|---------| -| `[CHANGESET]` | Cyan | Change execution | -| `[BUILD]` | Blue | Full schema rebuild | -| `[RUN]` | Magenta | Individual file execution | - -In the list, you can quickly spot operation types by color before reading the name. - - -## Understanding History Entries - -Each history entry tells a complete story. Here's how to read them: - - -### Who Ran It - -The identity shows who executed the operation: - -``` -By: Alice <alice@example.com> -``` - -In CI/CD, this might be: - -``` -By: GitHub Actions <actions@github.com> -``` - -Identity is captured from git config or explicitly set via `noorm identity`. This creates an audit trail for compliance requirements. You can also set a per-config identity override in [Configs](/guide/environments/configs). - - -### When It Ran - -Timestamps appear in relative format in the TUI, rendered by day.js `fromNow()`: - -- `a few seconds ago` - Just happened -- `2 hours ago` - Recent operations -- `a day ago` - Previous day -- `7 days ago` - Within the past week -- `a month ago` - Older records - -Relative time is all the TUI shows, however old the record. The JSON output provides ISO timestamps for precise filtering and analysis. - - -### How Long It Took - -Duration is shown in seconds: - -``` -Duration: 1.24s -``` - -For file-level details, you can see individual file timings: - -``` -[OK] 001_create-tokens-table.sql (0.4s) -[OK] 002_add-user-column.sql (0.3s) -``` - -This helps identify slow changes or unexpected performance issues. - - -### What Files Executed - -The detail view breaks down each file: - -| Status | Icon | Meaning | -|--------|------|---------| -| Success | `[OK]` | File executed without errors | -| Failed | `[ERR]` | Execution threw an error | -| Skipped | `[-]` | An earlier file in the change failed and this one was never reached | - -A file a prior run already applied leaves no row, so it does not appear here. For skipped files, the skip reason names the failure: - -``` -[-] setup.sql - 001_tables.sql failed: table users already exists -``` - - -## Using History for Debugging - -When something goes wrong, history is your first stop, with one dialect-specific exception. On PostgreSQL a change runs inside a transaction that covers its history rows as well as its DDL, so a failed change rolls back completely and writes nothing to history. Read the error off the failing command there. MySQL, SQL Server, and SQLite keep the failed record and everything below applies. - - -### Finding the Failure - -Look for red `[ERR]` indicators in the list. The most recent failure appears near the top. - -``` - [ERR] [CHANGESET] 2025-01-16-add-payments 10 minutes ago -``` - - -### Drilling into Details - -Press Enter to see which file failed: - -``` - [OK] 001_create-table.sql (0.3s) - [ERR] 002_add-constraint.sql (0.1s) - [-] 003_add-index.sql - 002_add-constraint.sql failed: ... -``` - -The third file was never executed because the second one failed, and its skip reason names the culprit. This tells you exactly where to start looking. - - -### Reading the Error - -The error message often points directly to the problem: - -``` -Error: duplicate key value violates unique constraint "users_email_key" -``` - -Now you know the issue---there's duplicate data that needs cleaning before the constraint can be added. - - -### Checking Previous Runs - -Compare with successful runs of the same change on other environments. Did something change between runs? Did the data differ? - -```bash -noorm change history --count 100 --json | jq '.history[] | select(.name == "2025-01-16-add-payments")' -``` - - -### Recovery Workflow - -After fixing the issue: - -1. Fix the underlying problem (clean data, modify SQL, etc.) -2. Re-run the change with `--force` if needed -3. Check history to confirm success -4. Continue with `change ff` for remaining changes - +`--count` defaults to 20 here. `executedAt` is ISO UTC. The text form prints local time. `change history-detail <name> --json` returns each operation with its file records. -## History Retention -History is stored in noorm’s change tracking tables, described in [Execution](/guide/sql-files/execution#what-gets-tracked). These tables grow with usage. +## What bites -For long-running projects, consider: -- Periodic exports for archival -- Database maintenance to manage table size -- Separate tracking databases for high-volume environments +| Symptom | Cause | Fix | +|---|---|---| +| A failed change leaves no history on PostgreSQL | One transaction rolls back the DDL and the history rows together | Read the error from the failing command ([Forward and revert](/guide/changes/forward-revert#what-happens-when-a-change-runs)) | +| An entry named `__reset__` | `db teardown` records itself, with the drop summary in `errorMessage` | None. It marks where changes went `stale` | +| `[ERR]` on a `reverted` or `stale` row in the TUI | `[OK]` means `success` only | Check `change history --json` or `change list` for the exact status | +| `change history-detail` prints nothing into a pipe | The text goes through the logger to stderr | Use `--json`, or redirect with `2>&1` | +| History from another environment is missing | Each config keeps its own history | Pass `-c <config>` | -The history is per-database, per-config. Each configuration maintains its own execution history. +## Next -## What's Next? -- [Forward & Revert](/guide/changes/forward-revert) - Applying and rolling back changes -- [Changes](/guide/changes/overview) - Change directory structure and file types -- [Teardown](/guide/database/teardown) - What happens to history after teardown +- [Forward and revert](/guide/changes/forward-revert): apply, roll back, and recover +- [Changes](/guide/changes/overview): folder layout and manifests +- [Teardown](/guide/database/teardown): why changes go `stale` +- [`change history`](/headless#change-history): flags and JSON diff --git a/docs/guide/changes/overview.md b/docs/guide/changes/overview.md index 1da4fbd2..492def28 100644 --- a/docs/guide/changes/overview.md +++ b/docs/guide/changes/overview.md @@ -1,409 +1,150 @@ # Changes -Changes are versioned operations that evolve an existing database to match your ideal schema. When you need to add a column, transform data, or run SQL that should execute exactly once with rollback support, changes are the answer. +A change takes a database that already exists to the schema in `db/sql/`. noorm runs each change once per database and records the result. - -## Why "Changes", Not "Migrations" - -Most tools call these migrations. We don't, and here's why. - -**Your SQL files are the source of truth.** They define what your schema *is*—the ideal, current state. You handcraft them, review them, keep them up to date. When you need a fresh database, you run your SQL files. No archaeology through years of migration history. - -**Changes describe what you *do*, not what you *move*.** To migrate implies moving something from one place to another. Data migrations traditionally mean moving data between servers, warehouses, or systems. But when you add a column to an existing table, you're not migrating anything—you're changing the schema. - -Some argue that you *are* moving data—adding a column means moving data into it, splitting `full_name` into `first_name` and `last_name` is moving data between columns, or you're "moving the schema from one version to the next." This is mental gymnastics. It's stretching a word to fit a process it was never meant to describe. - -The term "migration" originated in enterprise contexts where it actually made sense—incrementally moving schemas and data from Oracle to SQL Server, or consolidating legacy systems into a new platform. Rails borrowed this terminology in 2005 and popularized it for web applications. Everyone followed suit without questioning whether the word fit. We're not moving databases between systems. We're changing them in place. The language matters. - -**Kysely already has migrations.** If you want a traditional migration system, Kysely has one built in. But it lacks what noorm provides: - -- **Identity tracking** - Who ran what, and when? -- **Flexible workflows** - Build from scratch *or* apply changes, your choice -- **Templates and secrets** - Dynamic SQL with encrypted credentials -- **Execution history** - Full audit trail with timing and checksums -- **Revert scripts** - Rollback support when things go wrong - -Changes are your changelog. They document *how* to get an existing database to match your SQL files. Your SQL files remain the source of truth. - - -## When to Use Changes - -The distinction is simple: **can you rebuild from scratch, or do you need to evolve what's there?** - -**Fresh database?** Run `noorm run build`. This executes your `sql/` folder—your source of truth. Tables, views, functions, seed data. Everything needed to create a working schema from nothing. - -**Existing database with data you can't lose?** Run `noorm change ff`. This applies pending changes from your `changes/` folder. Each change runs once, gets recorded in the database, and won't run again unless reverted. - -Changes exist because production databases can't be rebuilt. You can't drop a table with millions of rows and recreate it. You need to `ALTER TABLE` in place, transform data carefully, add indexes without blocking writes. Changes give you: - -- **Execution tracking** - Know which changes have been applied -- **Revert scripts** - Roll back when something goes wrong -- **Ordered execution** - Date prefixes ensure consistent ordering across teams - - -## Directory Structure - -Each change lives in its own folder with a specific structure: - -``` -changes/ -└── 2025-01-15-add-email-verification/ - ├── change/ # Forward scripts (required) - │ ├── 001_create-tokens-table.sql - │ └── 002_add-user-column.sql - ├── revert/ # Rollback scripts (optional) - │ ├── 001_drop-user-column.sql - │ └── 002_drop-tokens-table.sql - └── changelog.md # Human-readable description -``` - -**change/** contains your forward scripts. These run in sequence order when you apply the change. - -**revert/** contains rollback scripts. These run in forward sequence order (001, then 002), just like change scripts. You design them to undo your changes in reverse—notice how `001_drop-user-column` undoes the last change and `002_drop-tokens-table` undoes the first. - -**changelog.md** is optional but recommended. Document what the change does and why. Your future self will thank you. - - -## Naming Conventions - -Change folders follow the pattern `YYYY-MM-DD-description`: - -``` -2025-01-15-add-email-verification -2025-02-01-migrate-user-roles -2025-02-14-add-search-indexes +```bash +noorm change add order-status ``` -The date prefix ensures changes apply in chronological order across your team. When Alice creates `2025-01-15-add-users` and Bob creates `2025-01-16-add-posts`, they'll apply in the right order regardless of when they merge. - -Files within `change/` and `revert/` use sequence prefixes: - -``` -001_create-table.sql -002_add-column.sql -003_update-data.sql ``` - -The three-digit prefix guarantees execution order. Names after the underscore are for humans. - - -## Creating a Change - -Scaffold a change headlessly with `noorm change add`: - -```bash -noorm change add add-email-verification -noorm change add # prompts for a description (TTY only) +Created: 2026-10-20-order-status + /home/dana/orders/db/changes/2026-10-20-order-status ``` -Or from the TUI: run `noorm ui`, press `g` from home to enter the changes screen, then `a` -and enter a name like `add-email-verification`. Either path scaffolds the same layout: - ``` -changes/2025-01-15-add-email-verification/ +db/changes/2026-10-20-order-status/ ├── change/ -│ └── 001_add-email-verification.sql +│ └── 001_order-status.sql ├── revert/ -│ └── 001_add-email-verification.sql +│ └── 001_order-status.sql └── changelog.md ``` -Unlike an empty folder pair, `001_add-email-verification.sql` is a real, runnable stub — one -comment line naming what belongs there ("Add the SQL statements that apply this change" in -`change/`, "...undo this change" in `revert/`). Replace the comment with your SQL. A comment -alone isn't enough to run: `change run` rejects a changeset whose files contain only -comments or template placeholders, with a message that tells you to edit the files — not the -misleading "change not found" a genuinely empty folder pair used to produce. - -For a multi-statement change, add more files by hand following the same `NNN_description` -convention (see [Naming Conventions](#naming-conventions) above) — `002_`, `003_`, and so on. -Files run in filename order (a plain string sort, so keep sequence numbers zero-padded to -three digits or the sort stops being numeric past `009_`). +Each `.sql` file starts as one comment line. Replace it with SQL, as [Add a column](/guide/changes/add-a-column) does for this change. In the TUI, press `g` from home, then `a`. -Once the directory exists you can apply, revert, and inspect the change from the headless -CLI (`noorm change ff`, `noorm change run`, `noorm change revert`, `noorm change history`). +## What is in a change folder? -## Write Idempotent DDL -Always check for existence before making changes. Use `IF NOT EXISTS` when creating, `IF EXISTS` when dropping. This protects you when: +| Path | Holds | Read by | +|---|---|---| +| `change/` | forward scripts | `change run`, `ff`, `next` | +| `revert/` | rollback scripts | `change revert`, `rewind` | +| `changelog.md` | why the change exists | people. noorm records only that it exists | -- A change accidentally runs twice -- Someone executes the file directly outside of noorm -- You need to re-run after a partial failure +Files in `change/` and `revert/` run by extension: -```sql --- change/001_add-verification-column.sql (forward) -ALTER TABLE users ADD COLUMN IF NOT EXISTS email_verified BOOLEAN DEFAULT FALSE; +| Extension | Runs | +|---|---| +| `.sql` | as written | +| `.sql.tmpl` | after Eta renders it ([Templates](/guide/sql-files/templates)) | +| `.txt` | a manifest: files from `db/sql/`, in place | --- revert/001_drop-verification-column.sql (rollback) -ALTER TABLE users DROP COLUMN IF EXISTS email_verified; -``` - -```sql --- change/001_create-tokens-table.sql (forward) -CREATE TABLE IF NOT EXISTS verification_tokens ( - id SERIAL PRIMARY KEY, - user_id INTEGER REFERENCES users(id), - token TEXT NOT NULL, - expires_at TIMESTAMP NOT NULL -); - --- revert/001_drop-tokens-table.sql (rollback) -DROP TABLE IF EXISTS verification_tokens; -``` - -This applies to both forward and revert scripts. A revert that fails because "column doesn't exist" is just as broken as a change that fails because "column already exists." - - -## File Types - -Changes support three file types: +Other files are not run, though a template still reads the data files beside it. A folder with neither `change/` nor `revert/` is not a change, so shared fragments and archives can sit beside the real ones. -### SQL Files (.sql) -Raw SQL executed directly. The most common type. +## How are changes and files ordered? -```sql --- 001_add-verification-column.sql -ALTER TABLE users ADD COLUMN IF NOT EXISTS email_verified BOOLEAN DEFAULT FALSE; -``` - -### Templates (.sql.tmpl) -Dynamic SQL processed through the Eta template engine before execution. Useful when you need conditional logic or variable substitution. See [Templates](/guide/sql-files/templates) for full syntax reference. - -```sql --- 001_add-audit-table.sql.tmpl -CREATE TABLE {%~ $.config.schema %}.audit_log ( - id SERIAL PRIMARY KEY, - action TEXT NOT NULL, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP -); -``` +noorm sorts by name at both levels, so the name is the order. -### Manifests (.txt) +| Level | Pattern | Example | +|---|---|---| +| Change folder | `YYYY-MM-DD-slug` | `2026-10-20-order-status` | +| File | `NNN_slug.ext` | `001_order-status.sql` | -Reference existing schema files instead of duplicating SQL. The manifest lists paths relative to your schema directory. +- `change add` builds the folder name from today's date and a slug of your description. +- Pad file numbers to three digits. Names sort as text, so `10_` comes before `9_`. +- `revert/` runs in the same filename order as `change/`. Number its files in the order the undo has to happen. -Many database objects use `CREATE OR REPLACE`—views, functions, procedures, triggers. They're already idempotent. When you change a table, you often need to recreate dependent views or functions. Instead of copying that SQL into your change folder, reference the source of truth: +`invoice_line` references `invoice`, so the revert drops the child first: -```txt -# change/001_rebuild-views.txt -views/user_summary.sql -views/active_users.sql -functions/get_user_stats.sql ``` - -```sql --- change/002_add-status-column.sql -ALTER TABLE users ADD COLUMN IF NOT EXISTS status VARCHAR(20) DEFAULT 'active'; +change/ revert/ +├── 001_invoice.sql ├── 001_invoice-line.sql +└── 002_invoice-line.sql └── 002_invoice.sql ``` -Manifest files participate in ordering. Here, `001_rebuild-views.txt` runs first (recreating views that might depend on the old schema), then `002_add-status-column.sql` adds the column. You could also flip this—add the column first with `001_`, then rebuild views with `002_` so they pick up the new column. - -This keeps your `sql/` folder as the single source of truth. Views and functions are defined once, referenced wherever needed. +## How does a change reuse a view or function from db/sql/? -## Checking Change Status -From the TUI, the changes screen shows status for each change: +A `.txt` manifest lists files in `db/sql/`. noorm runs them in place, so the view is defined once and the change never copies its SQL: ``` -┌─ Changes ─────────────────────────────────────────────┐ -│ • 2025-01-15-add-email-verification success │ -│ 2025-02-01-migrate-user-roles pending │ -│ 2025-02-14-add-search-indexes pending │ -└───────────────────────────────────────────────────────┘ +# db/changes/2026-10-22-open-orders/change/001_rebuild-views.txt +02_views/01_open_orders.sql +02_views/02_unpaid_invoices.sql ``` -Each change has a status: - -| Status | Meaning | -|--------|---------| -| `pending` | Not yet applied | -| `success` | Applied successfully | -| `failed` | Last execution failed | -| `reverted` | Was applied, then rolled back | -| `stale` | Was applied, then `db teardown` dropped the objects it created | +A manifest sorts among the other files of its folder. Use that to drop a dependent view, alter the table, then rebuild the view: -`pending`, `reverted`, and `stale` all count as outstanding work, so `change ff` and -`change next` pick up all three. - -From the headless CLI, `noorm change list` prints every known change with its status. Bare `noorm change` renders help -- it does not connect to the database. - -```bash -noorm change list # Human-friendly table -noorm change list --json # Same data as JSON ``` - -```json -{ - "success": true, - "changes": [ - { - "name": "2025-01-15-add-email-verification", - "status": "success", - "appliedAt": "2025-01-15T14:30:00.000Z", - "appliedBy": "Alice <alice@example.com>", - "revertedAt": null, - "errorMessage": null, - "isNew": false, - "orphaned": false - } - ], - "pending": 1 -} -``` - -Each entry merges the database record above with the on-disk one, so it also carries -`path`, `date`, `description`, `changeFiles`, `revertFiles`, and `hasChangelog`. An entry -flagged `orphaned` exists in the change tracking tables but no longer on disk, so those -disk fields are absent. - -When you omit the change name on a TTY, several subcommands open an interactive picker. -`run`, `revert`, `rewind`, and `history-detail` query the database and filter to the -relevant subset: outstanding changes for `run`, successfully applied ones for `revert` and -`rewind`, anything with a history record for `history-detail`. `edit` and `rm` work offline -and list every folder in `changes/`. Non-TTY callers must supply the name. - - -## Common Workflows - - -### Apply a Single Change - -Run a specific change by name: - -```bash -noorm change run 2025-02-01-migrate-user-roles +change/ +├── 001_drop-unpaid-invoices.sql -- DROP VIEW IF EXISTS unpaid_invoices; +├── 002_widen-amount.sql -- ALTER TABLE invoice_line ALTER COLUMN amount TYPE numeric(14,2); +└── 003_rebuild-views.txt 02_views/02_unpaid_invoices.sql ``` -From the TUI, select the change and press enter. +| Rule | Detail | +|---|---| +| Paths | one per line, relative to `paths.sql`: `02_views/01_open_orders.sql`, not `db/sql/02_views/...` | +| Ignored lines | blank lines and lines starting with `#` | +| Order | lines run in file order. A path listed twice in one change runs once, at its first position | +| Content | the file as it is at run time, not as it was when you wrote the change | +| Revert | a manifest in `revert/` rebuilds the current definition. To restore an older one, write its `CREATE OR REPLACE` inline | +A manifest is resolved every time noorm reads the change, so one missing path makes the whole change unreadable: -### Apply All Pending Changes - -Fast-forward applies all pending changes in order: - -```bash -noorm change ff +```mermaid +flowchart TD + M["001_rebuild-views.txt"] --> E{"every listed path<br/>exists in db/sql/?"} + E -->|yes| R["runs each file's<br/>current content"] + E -->|no| U["change is unreadable"] + U --> L["change list shows<br/>failed with the path"] + U --> F["ff and next stop here<br/>unless it already ran"] ``` -This is what you typically run in CI/CD after deployment. +[Forward and revert](/guide/changes/forward-revert#what-happens-when-a-change-folder-cannot-be-read) covers what `ff` does with it. An empty manifest, or one with only comments, is unreadable too. -### Apply the Next Pending Change +## How do I see what is pending? -Run just the next change in sequence: ```bash -noorm change next +noorm change list ``` -Useful when you want to apply changes one at a time and verify each. - - -### Revert a Change - -Roll back a previously applied change: - -```bash -noorm change revert 2025-02-01-migrate-user-roles ``` - -This runs the scripts in `revert/` in sequence order. The change status becomes `reverted`, which means it will run again on the next fast-forward. - - -### Rewind Multiple Changes - -Revert the last N applied changes: - -```bash -noorm change rewind 3 +2026-10-19-sales-order (success) +2026-10-20-order-status (reverted) +2026-10-21-invoice (failed): 002_invoice-line.sql: [Err 1] near "(": syntax error +2026-10-23-payment (pending) +1 pending change(s) ``` -This reverts the three most recently applied changes, starting with the newest. Pass a -change name instead of a count to rewind back to and including that change: - -```bash -noorm change rewind 2025-01-15-add-email-verification -``` - - -### Preview Before Running - -`--dry-run` is the preview: it renders every file, templates and manifests included, and -writes the result under `tmp/` instead of executing it. See [Dry Run](#dry-run) below. - -To render a single template file to stdout without involving a change at all, use -`noorm run preview`: - -```bash -noorm run preview sql/views/user_summary.sql.tmpl -``` - - -### Dry Run - -Render files to a temporary directory without executing. Available on every apply/revert command: `change run`, `change revert`, `change ff`, `change next`, and `change rewind`: - - noorm change run 2025-02-01-migrate-user-roles --dry-run - noorm change ff --dry-run - -Files are written to `tmp/` so you can inspect them. A dry run never writes to the change -tracking tables: it skips the "does this need to run?" lookup, takes no lock, and creates -no history record, which makes it safe to run repeatedly in production. `ff`, `next`, and -`rewind` still read the tables to work out which changes to render. The CLI tags human -output with `(dry-run)` and adds `dryRun: true` to the JSON payload of `run`, `revert`, -`ff`, and `next` so CI pipelines can verify the result didn't touch the database. -`rewind --json` does not add the flag. - - -## When Changes Re-Run - -noorm tracks whether each change needs to run based on several conditions: - -| Condition | What Happens | -|-----------|--------------| -| New change folder | Runs | -| Files modified since last run | Runs | -| Previous execution failed | Runs | -| Change was reverted | Runs | -| Database was torn down | Runs (marked stale) | -| `--force` flag provided | Runs | -| Successfully applied, unchanged | Skipped | - -After a successful run, the change won't run again unless something changes. This is tracked via a combined checksum of every file in `change/` or `revert/` plus the schema files any `.txt` manifest references, so editing a referenced view or function re-triggers the change. - -::: tip Force Re-Run -Need to re-apply a change? Use `--force`: -```bash -noorm change run my-change --force -``` -::: - - -## Best Practices - -**One logical change per folder.** Don't bundle unrelated changes. If you're adding users and adding products, make two separate changes. - -**Always include revert scripts.** Even if you think you'll never need them. Recovery is worth the extra files. +Each line is `name (status)`. A change recorded in the database whose folder is gone shows `(status, orphaned)`. `--json` adds `appliedAt`, `appliedBy`, `revertedAt`, `errorMessage`, `isNew`, and `orphaned` to each entry. [Forward and revert](/guide/changes/forward-revert#which-statuses-can-a-change-have) defines the five statuses. -**Test reverts in development.** Run `revert` then `change` to verify rollback works before deploying. -**Verify with explore and terminal.** noorm includes a database explorer (`d` → `e` from home) and SQL terminal (`d` → `t`) so you can inspect your schema and query data without leaving the tool. After applying a change, explore the table to confirm the column exists. After reverting, run a quick SELECT to verify it's gone. Don't trust—verify. +## What bites -**Use manifests for shared SQL.** Reference existing schema files with `.txt` manifests instead of duplicating SQL. -**Preview before production.** Use `--dry-run` to render everything to `tmp/` and inspect exactly what will execute. +| Symptom | Cause | Fix | +|---|---|---| +| `change run` fails: files are empty or contain only template placeholders | The scaffolded stub is a comment, and a change needs real SQL | Replace the comment with SQL | +| A folder in `db/changes/` never appears in `change list` | It has neither `change/` nor `revert/` | Add `change/` with at least one file | +| `change list` shows `failed` with `references missing file` | A manifest names a file that `db/sql/` no longer has. Deleting or renaming a `db/sql/` file breaks every unapplied change that lists it | Restore the file, or fix the manifest line | +| You edited an applied change and `change ff` does nothing | `ff` skips applied changes, so databases that ran it keep the old body | Add a new change. `change run <name>` re-executes the edit on the one database you point it at | +| `change ff` fails on a database built from `db/sql/` | A build records no changes, so `ff` replays all of them | Write each statement to be safe to run twice ([Add a column](/guide/changes/add-a-column)) | +| A manifest-listed table file fails on replay: already exists | A manifest runs the file's current content, and `CREATE TABLE` fails on an existing table | List only `CREATE OR REPLACE` objects: views, functions, procedures, triggers | +| `1 pending change(s)` is fewer than `ff` will run | The footer counts `pending` only. `ff` also runs `reverted` and `stale` | Read each status in the list | -**Write meaningful changelog.md files.** Explain *why* the change exists, not just what it does. Future debugging sessions will benefit. +## Next -## What's Next? -- [Forward & Revert](/guide/changes/forward-revert) - Applying and rolling back changes -- [History](/guide/changes/history) - Tracking execution history -- [Templates](/guide/sql-files/templates) - Using `.sql.tmpl` files in changes +- [Forward and revert](/guide/changes/forward-revert): apply and roll back +- [History](/guide/changes/history): what each run recorded +- [Add a column](/guide/changes/add-a-column): one edit, both folders +- [Templates](/guide/sql-files/templates): `.sql.tmpl` files in changes diff --git a/docs/guide/database/create.md b/docs/guide/database/create.md index 3b693d6f..7f455bfb 100644 --- a/docs/guide/database/create.md +++ b/docs/guide/database/create.md @@ -1,169 +1,108 @@ # Creating a database -`noorm db create` provisions the database named in a stored config. It -has no `--name`, `--dialect`, or `--host` flag — by design. +Create the database a config points at, and set it up for noorm, before the first build. -The reasoning is short: a noorm config is the single source of truth -for *which* database, *where*, *as whom*, and *with what guardrails*. -Inline flags would duplicate that record and create two ways to -describe the same target. The CLI sidesteps that by requiring you to -declare the config first, then create against it. +`db create` takes no `--name`, `--dialect`, or `--host`. The config holds the connection, so write it first. This `dev-config.json` targets local PostgreSQL: -This page covers the intended order of operations and the cross-tool -discovery hints that get users back on track when their muscle memory -expects a `--name` flag. +```json +{ + "name": "dev", + "connection": { + "dialect": "postgres", + "host": "localhost", + "database": "order_to_cash_dev", + "user": "postgres", + "password": "postgres" + } +} +``` +```bash +noorm config import dev-config.json +noorm db create -c dev +noorm config use dev +``` -## The configs-first workflow +``` +Imported config 'dev'. +Database "order_to_cash_dev" created and initialized. +Active config set to: dev +``` -End-to-end, from a clean repo to a usable database: +Run `noorm init` before these and `noorm run build` after. In the TUI, press `d` on Home, then `c`. - # 1. Scaffold the project (settings.yml, .noorm/, sql/, changes/). - noorm init --yes - # 2. Add one config per environment, non-interactively. - noorm config import dev-config.json - noorm config import test-config.json +## What does it do? - # 3. Create the underlying database for each config. - noorm db create -c dev - noorm db create -c test - # 4. Pick the active config for ad-hoc commands. - noorm config use dev +`db create` checks the role first, then creates only what is missing: -After step 4, every `noorm` command that doesn't pass `-c <name>` -inherits the active config. `noorm run build`, `noorm change ff`, and -`noorm sql query "…"` all target the `dev` database until you switch. +```mermaid +flowchart TD + G{"db:create allowed<br/>for this role?"} -->|no| D["exit 1<br/>nothing touched"] + G -->|yes| S{"database exists?"} + S -->|no| C["CREATE DATABASE"] + S -->|yes| T{"tracking tables<br/>exist?"} + C --> B["create tracking tables"] + T -->|no| B + T -->|yes| N["alreadyExists: true<br/>exit 0"] + B --> R["Database created<br/>and initialized"] +``` +`CREATE DATABASE` runs over a connection that does not need the target to exist: -## What `noorm db create` does (and what it needs) +| Dialect | Connects to | +|---|---| +| PostgreSQL | the `postgres` database | +| MSSQL | the `master` database | +| MySQL | no database selected | +| SQLite | nothing; it checks that the folder is writable | -The command connects to the dialect's *system database* -(`postgres` for PostgreSQL, `master` for MSSQL; MySQL connects with no -database named at all) with the credentials in the named config, then -runs the dialect's `CREATE DATABASE` statement targeted at -`config.connection.database`. SQLite has no system database to borrow, -so noorm checks that the directory holding the file is writable -instead. That probe leaves nothing behind. +The tracking tables are `version`, `change`, `executions`, `lock`, `identities`, and `vault`. PostgreSQL and MSSQL keep them in a `noorm` schema. MySQL and SQLite name them `__noorm_<table>__`. -Creating the database is only half the job. `db create` then connects -to the new database and bootstraps noorm's own tracking tables -(`version`, `change`, `executions`, `lock`, `identities`, `vault`), so -the config is ready for `run build` and `change ff` when the command -returns. Running it against a database that already exists and is -already initialized is a no-op that reports `alreadyExists: true`. +`viewer` is refused, `operator` needs `--yes`, and `admin` runs. A new config gets `user: admin` and `agent: viewer`. [Configs](/guide/environments/configs#access-roles) has the full matrix. -It needs: -- An active config (or an explicit `-c <name>`). -- Credentials in the config that can reach the server. -- A server-level permission to create databases (e.g. `CREATEDB` on - PostgreSQL, `dbcreator` role on MSSQL). -- An access role that permits `db:create`. A `viewer` config is denied - outright; an `operator` config needs `--yes` (or `NOORM_YES=1`) to - satisfy the confirmation; an `admin` config runs unconfirmed. +## How do I bootstrap CI? -It does not need: -- The target database to already exist — that's the whole point. -- A `.noorm/` schema to be built — `noorm run build` is a separate - step. +A CI checkout already has the committed `.noorm/`, so skip `init`. With no stored config, the `NOORM_CONNECTION_*` variables define the connection: -The access check runs before any probe touches the server. That -ordering matters on SQLite, where merely opening the target creates the -file: checking afterwards would hand a denied role a database anyway. +```bash +# ci/bootstrap.sh +set -euo pipefail +export NOORM_CONNECTION_DIALECT=postgres +export NOORM_CONNECTION_HOST=localhost +export NOORM_CONNECTION_DATABASE=order_to_cash_ci +export NOORM_CONNECTION_USER=ci +export NOORM_CONNECTION_PASSWORD="$DB_PASSWORD" -## Why no `--name <database>` flag? +noorm db create +noorm run build +``` -Three reasons, in order of importance: +[CI](/guide/automation/ci) covers runners that need an identity or the vault. -1. **Configs already hold the database name.** The connection block - in a noorm config carries `dialect`, `host`, `port`, `database`, - `user`, `password`, and a handful of safety flags (`isTest`, - `access`). Passing `--name foo` would let you bypass the config's - `access` role and create a database the active config doesn't - know about — a foot-gun. -2. **The config is also where stage defaults, secrets, and identity - bindings live.** A flag-only path would need its own copy of every - one of those, or it would inherit silently from the active config - and surprise the user. +## What bites -3. **It pushes you to write the config down.** `noorm config import` - is non-interactive and idempotent; the JSON file becomes the - reviewable artifact in your repo or CI runner. -If you genuinely need ad-hoc database creation without a stored -config, fall back to a one-shot SQL statement issued through an -existing one: +| Symptom | Cause | Fix | +|---|---|---| +| `db create --name foo` creates nothing named `foo` | Unknown flags are ignored, so it used the config's database | Set `connection.database` in the config | +| `"db:create" is not allowed on config "dev" (role: viewer).` | The role is `viewer`, or the process is on the `agent` channel, which defaults to `viewer` | Raise the role, or see [channels](/guide/environments/configs#which-channel-is-a-process-on) | +| `This is a destructive operation requiring confirmation (yes-dev). Pass --yes to confirm.` | The role is `operator` | Pass `--yes` or set `NOORM_YES=1` | +| `Failed to create database: permission denied to create database` | The login lacks server-level rights | Grant `CREATEDB` on PostgreSQL or `dbcreator` on SQL Server | +| `Cannot connect to server: …` | Wrong host, port, or credentials | Fix the JSON, then `config import --force --yes` | +| MySQL: `Cannot connect to server: Database 'order_to_cash_dev' does not exist on …` | MySQL has no system database for the existence check to fall back to | Create the empty database with another client, then run `db create` | - noorm sql query -c dev "CREATE DATABASE foo" -That forces you to pick a config, and therefore a connection identity, -for the operation. Two caveats: the statement runs against the `dev` -database rather than the system database, so `dev` itself has to exist -and be reachable; and `CREATE DATABASE` classifies as `sql:ddl`, which -only an `admin`-role config may run. The SQL is passed through -verbatim, so it is on you to write it for the right dialect. +## Next -## Companion commands - -| Command | Purpose | -|---------|---------| -| `noorm config import <path>` | Add a config from a JSON file (non-interactive). | -| `noorm config list` | See which configs exist and which is active. | -| `noorm config use <name>` | Switch the active config. | -| `noorm db create [-c <name>]` | Create the database described by the named config. | -| `noorm db drop [-c <name>]` | Drop the database described by the named config. | - -`db drop` is gated harder than `db create`. Its `db:destroy` permission -denies both `viewer` and `operator`, and an `admin` config still has to -confirm, so dropping a database always takes `--yes` (or `NOORM_YES=1`). -A config using the default access role is `user: admin`, which is why -`db create` needs no flag on a fresh project. - -For interactive setup (when you're at a terminal), `noorm ui` launches -a wizard that walks through identity → config → DB create in order. -Scripted setups should stick with the `import` + `create` flow above. - - -## CI / non-interactive bootstrap - -The same flow inside a CI runner — every step has a non-interactive -path: - - #!/usr/bin/env bash - set -euo pipefail - - # 1. Identity (once per runner). - noorm identity init --name "$CI_USER" --email "$CI_EMAIL" - - # 2. Project scaffold. - noorm init --yes - - # 3. Configs from version-controlled JSON. - noorm config import ./ci/dev.config.json - noorm config import ./ci/test.config.json - - # 4. Databases. - noorm db create -c dev - noorm db create -c test - - # 5. Schema + changes. - noorm config use dev - noorm run build - noorm change ff - -See [Non-interactive operation](../automation/non-interactive.md) for -the full matrix of `--yes` / `NOORM_YES` behavior. - - -## Related - -- [`noorm init`](../../cli/init.md) — project scaffold and identity bootstrap. -- [Non-interactive operation](../automation/non-interactive.md) — `--yes` opt-in and CI flow. -- [Troubleshooting](../troubleshooting.md) — what to do when `noorm db create --name foo` doesn't behave as expected. +- [First build](/getting-started/first-build): build `db/sql/` into it +- [Configs](/guide/environments/configs): roles and channels +- [Non-interactive operation](/guide/automation/non-interactive): `--yes` per command +- [`db create` reference](/headless#db-create): flags and `--json` output diff --git a/docs/guide/database/explore.md b/docs/guide/database/explore.md index 797639f2..6ddfe48d 100644 --- a/docs/guide/database/explore.md +++ b/docs/guide/database/explore.md @@ -1,521 +1,129 @@ # Schema Explorer -The Schema Explorer gives you read-only access to your database structure directly from the terminal. No need to switch to pgAdmin, MySQL Workbench, or Azure Data Studio. Browse tables, views, indexes, foreign keys, functions, procedures, types, and triggers without leaving your workflow. +Read your database's structure, from tables to triggers, without opening a second client. - -## What It Shows - -The explorer queries your database's system catalogs to reveal its structure: - -| Category | Information | -|----------|-------------| -| Tables | Columns, data types, row estimates, indexes, foreign keys | -| Views | Columns, definition SQL, whether it is updatable | -| Indexes | Table, columns, unique/primary status | -| Foreign Keys | Source and target tables, column mappings, cascade rules | -| Functions | Parameters, return type, definition | -| Procedures | Parameters, definition | -| Types | Enum values, composite attributes, domain base types | -| Triggers | Table, timing (BEFORE/AFTER/INSTEAD OF), events, definition | - -Two more categories, **locks** and **connections**, describe runtime state rather than schema. They are counted in the `--json` overview and reachable from the SDK, but they have no CLI listing and no TUI screen. - -Which catalogs each dialect reads: - -| Dialect | Catalogs | -|---------|----------| -| PostgreSQL | `information_schema` for tables, columns, views, FKs, and parameters; `pg_catalog` (`pg_proc`, `pg_type`, `pg_enum`, `pg_indexes`, `pg_locks`, `pg_stat_activity`) for everything it cannot express | -| MySQL | `information_schema` throughout, plus `performance_schema.metadata_locks` for locks | -| MSSQL | `sys.*` catalog views (`sys.tables`, `sys.views`, `sys.procedures`, `sys.types`, `sys.indexes`, `sys.foreign_keys`, `sys.triggers`) | -| SQLite | `sqlite_master` for object names and DDL, `PRAGMA table_info` / `index_info` / `foreign_key_list` for the rest | - -noorm's own tracking objects are filtered out: PostgreSQL and MSSQL exclude the `noorm` schema in the catalog query, MySQL and SQLite drop names starting with `__noorm_`. Set the log level to `verbose` to include them. - - -## Accessing the Explorer - - -### Terminal UI - -From the home screen, press `d` to enter the Database menu, then `x` to explore: - -``` -Home → [d] Database → [x] Explore -``` - -The Database menu: - -``` -┌─ Database ──────────────────────────────────────────────────┐ -│ │ -│ [c] create Create the database for this config │ -│ [d] destroy Drop the database for this config │ -│ [x] explore Browse tables, views, indexes │ -│ [w] wipe Truncate all data (keep schema) │ -│ [t] teardown Drop all objects │ -│ [r] transfer Move data between configs │ -│ │ -└─────────────────────────────────────────────────────────────┘ -``` - -The SQL terminal is not on this menu. Press `Shift+Q` from anywhere, or run `noorm sql repl`. See [SQL Terminal](/guide/database/terminal). - - -### Scripted Usage - -The same subcommands run non-interactively — add `--json` to get machine-readable output: +Counts first, then one table: ```bash -noorm db explore # Overview -noorm db explore tables # List tables -noorm db explore tables detail users # Table detail -noorm db explore fks # List foreign keys -noorm db explore triggers # List triggers -noorm db explore tables --schema app # Restrict to one schema -noorm db explore --json > schema.json # JSON output -``` - -The subcommands are `tables`, `views`, `procedures`, `functions`, `types`, `indexes`, `fks`, and `triggers`. Every one accepts `--schema` except on SQLite, which has no schemas and rejects the flag rather than returning an empty list. `tables`, `views`, `procedures`, `functions`, `types`, and `triggers` also take a `detail <name>` subcommand. - - -## Overview Screen - -The overview shows counts of all database objects at a glance: - -``` -┌─ DB Explore ────────────────────────────────────────────────┐ -│ │ -│ Config: dev (postgres) │ -│ Database: myapp_development │ -│ Total Objects: 47 │ -│ │ -│ [1] Tables 12 │ -│ [2] Views 3 │ -│ [3] Procedures 0 │ -│ [4] Functions 5 │ -│ [5] Types 2 │ -│ [6] Indexes 18 │ -│ [7] Foreign Keys 7 │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [1-7] Navigate [Esc] Back -``` - -Press the number key or hotkey to drill into any category. Categories with zero items appear dimmed and cannot be selected. - -| Key | Hotkey | Category | -|-----|--------|----------| -| `1` | `t` | Tables | -| `2` | `v` | Views | -| `3` | `p` | Procedures | -| `4` | `f` | Functions | -| `5` | `y` | Types | -| `6` | `i` | Indexes | -| `7` | `k` | Foreign Keys | - -The TUI overview lists these seven. `noorm db explore` on the command line prints a Triggers count as well, and `--json` carries all ten counters including locks and connections. The Total Objects figure sums every counter the overview computed, so it can exceed the seven rows on screen when triggers, locks, or connections are present. - - -## Browsing Tables - -Press `1` or `t` from the overview to see all tables: - -``` -┌─ Tables (12) ───────────────────────────────────────────────┐ -│ │ -│ / Filter tables... │ -│ │ -│ 1. > public.users 8 columns, ~1.5K rows │ -│ 2. public.posts 5 columns, ~42K rows │ -│ 3. public.comments 4 columns, ~128K rows │ -│ 4. public.notifications 6 columns, ~8.9K rows │ -│ 5. public.sessions 3 columns, ~2.1K rows │ -│ │ -│ ... 7 more │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Enter] View detail [Esc] Back -``` - -The list is searchable. Press `/` to focus the filter field and type to narrow results. Press `1-9` for quick selection. - - -### Table Detail - -Select a table to see its full schema: - -``` -┌─ Table ─────────────────────────────────────────────────────┐ -│ │ -│ public.users ~1,500 rows │ -│ │ -│ Columns (8) │ -│ ───────────────────────────────────────────────────────── │ -│ * id uuid NOT NULL │ -│ email varchar(255) NOT NULL │ -│ name varchar(100) NULL │ -│ password_hash text NOT NULL │ -│ role user_role NOT NULL DEFAULT 'user' │ -│ active boolean NOT NULL DEFAULT true │ -│ created_at timestamptz NOT NULL DEFAULT now() │ -│ updated_at timestamptz NULL │ -│ │ -│ Indexes (2) │ -│ ───────────────────────────────────────────────────────── │ -│ * users_pkey (id) │ -│ users_email_idx (email) UNIQUE │ -│ │ -│ Foreign Keys (0) │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back -``` - -The `*` marker indicates primary key columns and primary key indexes. - - -## Browsing Views - -Press `2` or `v` from the overview: - -``` -┌─ Views (3) ─────────────────────────────────────────────────┐ -│ │ -│ 1. > public.active_users 4 columns, READ-ONLY │ -│ 2. public.post_stats 6 columns, READ-ONLY │ -│ 3. public.user_activity 8 columns, UPDATABLE │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Enter] View detail [Esc] Back -``` - - -### View Detail - -``` -┌─ View ──────────────────────────────────────────────────────┐ -│ │ -│ public.active_users READ-ONLY │ -│ │ -│ Columns (4) │ -│ ───────────────────────────────────────────────────────── │ -│ id uuid NOT NULL │ -│ email varchar(255) NOT NULL │ -│ name varchar(100) NULL │ -│ created_at timestamptz NOT NULL │ -│ │ -│ Definition │ -│ ───────────────────────────────────────────────────────── │ -│ SELECT id, email, name, created_at │ -│ FROM users │ -│ WHERE active = true │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back -``` - - -## Browsing Indexes and Foreign Keys - -Indexes and foreign keys are list-only categories. They show all the information inline without a separate detail screen. - - -### Indexes - -``` -┌─ Indexes (18) ──────────────────────────────────────────────┐ -│ │ -│ 1. users_pkey on users, PRIMARY │ -│ 2. users_email_idx on users, UNIQUE │ -│ 3. posts_pkey on posts, PRIMARY │ -│ 4. posts_user_id_idx on posts │ -│ 5. posts_created_at_idx on posts │ -│ 6. comments_pkey on comments, PRIMARY │ -│ │ -│ ... 12 more │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back -``` - - -### Foreign Keys - +noorm db explore +noorm db explore tables detail sales_order ``` -┌─ Foreign Keys (7) ──────────────────────────────────────────┐ -│ │ -│ 1. posts_user_id_fkey │ -│ posts(user_id) → users(id) ON DELETE CASCADE │ -│ │ -│ 2. comments_post_id_fkey │ -│ comments(post_id) → posts(id) ON DELETE CASCADE │ -│ │ -│ 3. comments_user_id_fkey │ -│ comments(user_id) → users(id) ON DELETE SET NULL │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back -``` - - -## Browsing Functions and Procedures - -Functions and procedures show parameters, return types, and definitions. - - -### Functions ``` -┌─ Functions (5) ─────────────────────────────────────────────┐ -│ │ -│ 1. > public.calculate_total 2 params → numeric │ -│ 2. public.get_user_posts 1 param → SETOF posts │ -│ 3. public.update_timestamp 0 params → trigger │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Enter] View detail [Esc] Back +Database Overview + Tables: 11 + Views: 1 + Functions: 1 + Procedures: 0 + Types: 0 + Indexes: 14 + Foreign Keys: 12 + Triggers: 0 ``` - -### Function Detail - ``` -┌─ Function ──────────────────────────────────────────────────┐ -│ │ -│ public.calculate_total → numeric │ -│ │ -│ Parameters (2) │ -│ ───────────────────────────────────────────────────────── │ -│ price numeric IN │ -│ quantity integer IN │ -│ │ -│ Definition │ -│ ───────────────────────────────────────────────────────── │ -│ BEGIN │ -│ RETURN price * quantity; │ -│ END; │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back +• log Table: sales_order +• log party_id: integer (not null) [PK] +• log order_no: integer (not null) [PK] +• log placed_at: timestamp with time zone (not null) +• log status: text (not null) ``` +The explorer is read-only and every role may use it. The TUI path is Home, `d` for Database, `x` for Explore. -### Procedures - -Procedures work the same way as functions, but without a return type. +## What can I look at? -## Browsing Types -Custom types are available in PostgreSQL and MSSQL. The explorer shows enum values, composite type attributes, and domain/alias base types. +| Category | List | One object | TUI keys | Empty on | +|---|---|---|---|---| +| Tables | `tables` | `tables detail <name>` | `1` `t` | | +| Views | `views` | `views <name>` | `2` `v` | | +| Procedures | `procedures` | `procedures <name>` | `3` `p` | SQLite | +| Functions | `functions` | `functions <name>` | `4` `f` | SQLite | +| Types | `types` | `types <name>` | `5` `y` | MySQL, SQLite | +| Indexes | `indexes` | | `6` `i` | | +| Foreign keys | `fks` | | `7` `k` | | +| Triggers | `triggers` | `triggers <name>` | | | -``` -┌─ Types (2) ─────────────────────────────────────────────────┐ -│ │ -│ 1. > public.user_role ENUM, 3 values │ -│ 2. public.address COMPOSITE, 5 attributes │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Enter] View detail [Esc] Back -``` +Every list and detail command takes `--schema <name>`. SQLite has no schemas and rejects the flag. +Types are enums, composites, and domains on PostgreSQL, and alias and table types on MSSQL. Triggers have no TUI screen. Locks and connections are runtime state, so they appear only as counters in the `--json` overview. -### Enum Type Detail -``` -┌─ Type ──────────────────────────────────────────────────────┐ -│ │ -│ public.user_role ENUM │ -│ │ -│ Values (3) │ -│ ───────────────────────────────────────────────────────── │ -│ user │ -│ moderator │ -│ admin │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back -``` +## How do I move around the TUI? -### Composite Type Detail +Indexes and foreign keys end at the list. The other categories continue to a detail screen: -``` -┌─ Type ──────────────────────────────────────────────────────┐ -│ │ -│ public.address COMPOSITE │ -│ │ -│ Attributes (5) │ -│ ───────────────────────────────────────────────────────── │ -│ street varchar(200) NOT NULL │ -│ city varchar(100) NOT NULL │ -│ state varchar(50) NULL │ -│ postal_code varchar(20) NOT NULL │ -│ country varchar(100) NOT NULL │ -│ │ -└─────────────────────────────────────────────────────────────┘ - [Esc] Back +```mermaid +flowchart TD + DB["Database screen<br/>x"] --> OV["Overview<br/>counts per category"] + OV -->|"1-7 or t v p f y i k"| LS["List<br/>filter with /"] + LS -->|Enter| DT["Detail<br/>columns, parameters,<br/>definition"] + DT -->|v| FT["Full text<br/>nothing truncated"] + DT -->|r| RW["Rows<br/>first and last"] ``` +| Screen | Keys | +|---|---| +| Overview | A category with a zero count is dimmed and cannot be opened. `Esc` goes back | +| List | `/` filters, `1`-`9` picks a row, `Enter` opens it. The filter text is kept until you leave the explorer | +| Detail | `↑` `↓` scroll, `Ctrl+U` `Ctrl+D` half a page, `PageUp` `PageDown` a page, `Home` `End` the ends | +| Rows | `Tab` switches between the first and last set, `Enter` opens a row, `/` filters, `s` sorts, `Esc` closes | -## Headless JSON Output +`r` works on tables only. It reads rows ordered by primary key: up to 10 from each end, fewer on a short terminal. A table with no primary key shows only its first rows, in storage order. -Use `--json` flag for machine-readable output. Every noorm command wraps its `--json` payload in an object carrying a top-level `success` boolean, so `jq -e '.success'` works the same way against all of them. A list is never emitted as a bare array; it is always named, as in `{ "tables": [...] }`. - - -### Overview - -```bash -noorm db explore --json -``` - -```json -{ - "success": true, - "tables": 12, - "views": 3, - "procedures": 0, - "functions": 5, - "types": 2, - "indexes": 18, - "foreignKeys": 7, - "triggers": 4, - "locks": 0, - "connections": 2 -} -``` - - -### Table List - -```bash -noorm db explore tables --json -``` - -```json -{ - "success": true, - "tables": [ - { "name": "users", "schema": "public", "columnCount": 8, "rowCountEstimate": 1500 }, - { "name": "posts", "schema": "public", "columnCount": 5, "rowCountEstimate": 42000 }, - { "name": "comments", "schema": "public", "columnCount": 4, "rowCountEstimate": 128000 } - ] -} -``` -The other list subcommands follow the same shape under their own key: `views`, `procedures`, `functions`, `types`, `indexes`, `foreignKeys` (from `noorm db explore fks`), and `triggers`. +## How do I use it in scripts? -### Table Detail +Add `--json`. Every payload carries a top-level `success`, and a list is always named, never a bare array: ```bash -noorm db explore tables detail users --json +noorm db explore tables detail sales_order --json ``` ```json { "success": true, - "name": "users", + "name": "sales_order", "schema": "public", "columns": [ - { "name": "id", "dataType": "uuid", "isNullable": false, "isPrimaryKey": true, "ordinalPosition": 1 }, - { "name": "email", "dataType": "varchar(255)", "isNullable": false, "isPrimaryKey": false, "ordinalPosition": 2 }, - { "name": "name", "dataType": "varchar(100)", "isNullable": true, "isPrimaryKey": false, "ordinalPosition": 3 }, - { "name": "created_at", "dataType": "timestamptz", "isNullable": false, "defaultValue": "now()", "isPrimaryKey": false, "ordinalPosition": 7 } + { "name": "party_id", "dataType": "integer", "isNullable": false, "isPrimaryKey": true, "ordinalPosition": 1 }, + { "name": "status", "dataType": "text", "isNullable": false, "defaultValue": "'open'::text", "isPrimaryKey": false, "ordinalPosition": 4 } ], "indexes": [ - { "name": "users_pkey", "columns": ["id"], "isUnique": true, "isPrimary": true }, - { "name": "users_email_idx", "columns": ["email"], "isUnique": true, "isPrimary": false } + { "name": "sales_order_pk", "columns": ["party_id", "order_no"], "isUnique": true, "isPrimary": true } ], - "foreignKeys": [], - "rowCountEstimate": 1500 -} -``` - - -### View Detail - -```bash -noorm db explore views detail active_users --json -``` - -```json -{ - "success": true, - "name": "active_users", - "schema": "public", - "columns": [ - { "name": "id", "dataType": "uuid", "isNullable": false, "isPrimaryKey": false, "ordinalPosition": 1 }, - { "name": "email", "dataType": "varchar(255)", "isNullable": false, "isPrimaryKey": false, "ordinalPosition": 2 } + "foreignKeys": [ + { "name": "sales_order_isplacedby_party", "columns": ["party_id"], "referencedTable": "party", "referencedColumns": ["party_id"], "onUpdate": "NO ACTION", "onDelete": "NO ACTION" } ], - "definition": "SELECT id, email FROM users WHERE active = true", - "isUpdatable": false + "rowCountEstimate": 3 } ``` +This output is trimmed to two columns, and the repeated `schema` keys inside indexes and foreign keys are dropped. The list commands return `{ "tables": [...] }`, `{ "views": [...] }`, and so on, with `foreignKeys` for `fks`. The overview adds `locks` and `connections` to the eight counters. Full flags: [`db explore`](/headless#db-explore). -### Function Detail - -```bash -noorm db explore functions detail calculate_total --json -``` - -```json -{ - "success": true, - "name": "calculate_total", - "schema": "public", - "parameters": [ - { "name": "price", "dataType": "numeric", "mode": "IN", "ordinalPosition": 1 }, - { "name": "quantity", "dataType": "integer", "mode": "IN", "ordinalPosition": 2 } - ], - "returnType": "numeric", - "definition": "BEGIN RETURN price * quantity; END;" -} -``` - - -## Dialect Support - -The explorer works across all supported databases with appropriate translations: - -| Category | PostgreSQL | MySQL | MSSQL | SQLite | -|----------|:----------:|:-----:|:-----:|:------:| -| Tables | Yes | Yes | Yes | Yes | -| Views | Yes | Yes | Yes | Yes | -| Indexes | Yes | Yes | Yes | Yes | -| Foreign Keys | Yes | Yes | Yes | Yes | -| Triggers | Yes | Yes | Yes | Yes | -| Functions | Yes | Yes | Yes | -- | -| Procedures | Yes | Yes | Yes | -- | -| Types | Yes | -- | Yes | -- | -| Locks | Yes | Yes | Yes | -- | -| Connections | Yes | Yes | Yes | -- | - -SQLite does not support stored procedures, functions, or user-defined types, and has no server to report locks or connections for. MySQL does not support user-defined types. These categories return empty results for unsupported dialects. - -**Type support by dialect:** - -- **PostgreSQL** - Enums, composite types, domains -- **MSSQL** - Alias types (e.g., `CREATE TYPE EmailAddress FROM VARCHAR(255)`), table types - - -## Common Use Cases - -**Verify builds.** After running [schema builds](/guide/sql-files/execution), use explore to confirm tables, views, and functions were created correctly. -**Check constraints.** Browse foreign keys to verify referential integrity is set up as expected before deploying. +## What bites -**Inspect indexes.** Review index coverage before and after adding new queries to ensure performance. -**Compare environments.** Run overview on dev vs prod to spot schema drift quickly. Use different [configs](/guide/environments/configs) with the `-c` flag. +| Symptom | Cause | Fix | +|---|---|---| +| `noorm db explore tables \| grep party` prints nothing | Listings and detail text are log lines on stderr. Only the overview text and `--json` use stdout | Use `--json` | +| `noorm db explore -c prod tables` explores the active config | A flag before the subcommand is read by `explore`, not by `tables` | `noorm db explore tables -c prod` | +| `View not found: detail` | Only `tables` has a `detail` subcommand | `noorm db explore views open_orders` | +| `SQLite has no schemas; drop the schema filter to explore this database` | `--schema` on SQLite | Drop the flag | +| noorm's tracking tables are missing from the lists | Explore hides the `noorm` schema on PostgreSQL and MSSQL, and `__noorm_*` tables on MySQL and SQLite | The TUI shows them when `logging.level` is `verbose` in `.noorm/settings.yml`. The CLI never does | +| A composite foreign key lists its columns several times, mispaired (PostgreSQL) | The query joins `key_column_usage` to `constraint_column_usage` on the constraint name alone | Read the key from its `CREATE TABLE` in `db/sql/` | -**Debug changes.** When [changes](/guide/changes/overview) fail, explore the current schema state to understand what exists. +## Next -## What's Next? -- [SQL Terminal](/guide/database/terminal) - Run queries against your database -- [Teardown](/guide/database/teardown) - Reset database objects -- [Configs](/guide/environments/configs) - Connect to different databases +- [SQL terminal](/guide/database/terminal): query the data itself +- [Teardown](/guide/database/teardown): drop what you found +- [Configs](/guide/environments/configs): point the explorer at another database +- [`db explore` reference](/headless#db-explore): every subcommand and flag diff --git a/docs/guide/database/teardown.md b/docs/guide/database/teardown.md index c3da576d..fff4893d 100644 --- a/docs/guide/database/teardown.md +++ b/docs/guide/database/teardown.md @@ -1,397 +1,141 @@ # Teardown -Database cleanup that handles the tedious parts. Foreign key constraints, drop order, identity resets. Two operations, one goal: a clean slate. +Reset a database to a known state: delete its rows, drop its objects, rebuild it from `db/sql/`, or drop the database. - -## When You Need This - -Development databases accumulate debris. Half-applied changes, test data from last week, orphaned tables from abandoned features. Dropping and recreating the entire database works, but it is slow. Filesystem operations, permission grants, connection pool resets. What should take milliseconds becomes 5-10 seconds. - -noorm provides controlled reset operations that complete in milliseconds while handling the complexity for you. - - -## Two Operations - -| Operation | What happens | When to use | -|-----------|--------------|-------------| -| `truncate` | Delete all rows, keep tables | Test resets, re-seeding data | -| `teardown` | Drop all database objects | Full rebuild, change testing | - -::: danger Destructive Operations -Both operations permanently destroy data. `truncate` wipes all rows. `teardown` drops tables, views, functions, procedures, types, and foreign key constraints. There is no undo. These commands require explicit confirmation. -::: - - -## Truncate: Wipe Data, Keep Schema - -Truncate removes all data while preserving table structure. Fast reset for test cycles. +Preview what a teardown would remove: ```bash -noorm db truncate -y -``` - -### What Gets Truncated - -- All user tables (rows deleted) -- Identity/auto-increment counters restarted, on every dialect except SQLite -- Foreign key constraints temporarily disabled during operation - -Foreign key enforcement is re-enabled even when the truncate itself fails partway through, so a failed run never leaves the database with constraints switched off. - -### What Stays - -- Table definitions and indexes -- Views, functions, stored procedures -- noorm internal tracking tables (the `noorm` schema, or `__noorm_*__` tables) - - -## Teardown: Drop Everything - -Teardown removes all database objects. Clean slate for full rebuilds. - -```bash -noorm db teardown -y -``` - -::: warning Access Roles -`db teardown` is gated by the config's `db:teardown` permission: `viewer` and `operator` are both denied outright, and even `admin` must confirm (`yes-<config>`, or `NOORM_YES=1` in CI). Only an `admin`-role config can tear down at all. See [Access Roles](#access-roles) below. -::: - -### Drop Order - -Objects are removed in dependency-safe order: - -1. **Foreign key constraints** - Must go first to allow table drops -2. **CHECK constraints** - MSSQL only, and only when functions are being dropped: a scalar UDF referenced by a CHECK constraint cannot be dropped while its table stands -3. **Procedures** - Dropped early so they hold no references to the functions, views, and tables below -4. **Functions** - Must precede table drops so schema-bound UDFs release their dependency locks -5. **Views** - Schema-bound views also lock the tables they reference -6. **Tables** - Safe once every schema-bound dependent is gone -7. **Types** - Table types first, then the rest - -Procedures, functions, and views all go before tables, not after. A `WITH SCHEMABINDING` object on MSSQL holds a dependency lock on the table it references, and dropping the table first fails with "Cannot DROP TABLE ... because it is being referenced by object ...". - -You do not need to manage this ordering. noorm handles it automatically. - -Pass `--preserve-schemas <a,b>` to leave whole schemas untouched. Teardown reaches every non-system schema, so this is the way to protect one it did not create. - -### What Stays - -noorm internal tables are always preserved: - -- `change` - Change execution history -- `executions` - File execution records -- `lock` - Active operation locks - -On PostgreSQL and SQL Server these live in a dedicated `noorm` schema, so they read as `noorm.change`, `noorm.executions`, and `noorm.lock`. MySQL and SQLite have no schemas, so they keep the prefixed forms in the default schema: `__noorm_change__`, `__noorm_executions__`, `__noorm_lock__`. - -After teardown, noorm can still track what was applied previously. Changes are marked as `stale`, meaning they'll re-run on the next [fast-forward](/guide/changes/forward-revert). See [History](/guide/changes/history) for how this affects your execution log. - - -## Access Roles - -[Configs](/guide/environments/configs) declare an `access.user` role — `viewer`, `operator`, or `admin` — and each destructive operation resolves against its own permission, not one shared one. This prevents accidentally wiping production data. You can also cap it at the [stage level](/guide/environments/stages): a stage with `protected: true` clamps every linked config's resolved access to at most `operator`/`viewer`. - -```yaml -# .noorm/settings.yml -stages: - prod: - defaults: - protected: true # access ceiling: at most operator/viewer +noorm db teardown --dry-run ``` -The three permissions in this family resolve differently: - -| Operation | Permission | `viewer` | `operator` | `admin` | -|-----------|------------|----------|------------|---------| -| `db teardown` | `db:teardown` | Denied | Denied | Confirm | -| `db truncate` | `db:truncate` | Denied | Confirm | Confirm | -| `db reset` / `ctx.noorm.db.reset()` | `db:reset` | Denied | Confirm | Allowed | - -Teardown is the strictest of the three: it is the only one an `operator` cannot reach at all, and the only one that still asks an `admin` to confirm. `reset` is the loosest, because it rebuilds the schema it just dropped. - -When you attempt teardown on a `viewer`- or `operator`-role config: - ``` -Cannot teardown on config "prod": "db:teardown" is not allowed on config "prod" (role: operator) +• log Would drop 5 objects +• log Tables: 4 +• log Views: 1 +• log Functions: 0 +• log Types: 0 ``` -No flag, environment variable, or SDK option overrides a denial. Give the config `admin` access if it genuinely needs to be torn down. +The counts come from a project with `party`, `product`, `sales_order`, `order_line`, and the view `open_orders`. Replace `--dry-run` with `--yes` to run it. -### Confirming on `admin` -**CLI/TUI**: Type the confirmation phrase, or pass `--yes` (`NOORM_YES=1` in CI) to skip the prompt: +## Which command removes what? -```bash -NOORM_YES=1 noorm db teardown --config prod -``` -**SDK**: There is no prompt to answer. `ctx.noorm.db.teardown()` throws `ProtectedConfigError` on any config whose confirmation the caller has not pre-supplied. Pass `yes: true` to `createContext`, or set `NOORM_YES=1` in the environment, the same variable the CLI honors: +| Command | Removes | Survives | `viewer` | `operator` | `admin` | +|---|---|---|---|---|---| +| `db truncate` | every row | tables, indexes, views, routines, types, noorm's tracking tables | deny | confirm | confirm | +| `db teardown` | foreign keys, procedures, functions, views, tables, types | noorm's tracking tables, `preserveTables`, `--preserve-schemas` | deny | deny | confirm | +| `db reset` | what teardown removes, then rebuilds from `db/sql/` | noorm's tracking tables | deny | confirm | allow | +| `db drop` | the database | nothing in it; the config stays | deny | deny | confirm | -```typescript -const ctx = await createContext({ config: 'prod', yes: true }) - -await ctx.noorm.db.teardown() // Throws ProtectedConfigError without `yes` (or NOORM_YES=1) -``` +`allow` runs. `confirm` needs `--yes` or `NOORM_YES=1`, and the TUI asks for the phrase `yes-<config>`. `deny` has no override. `db reset` asks for `--yes` even where its cell is `allow`. -Neither `yes: true` nor `NOORM_YES=1` unblocks a denial. On a `viewer`- or `operator`-role config, `teardown()` throws no matter what you pass. +On the `agent` channel every `confirm` is a `deny`. A stage with `protected: true` caps a config at `operator`, so teardown and drop are refused there. [Concepts](/getting-started/concepts#configs-stages-and-roles) defines the roles. -A dry run is checked against the role but never against the confirmation: any role allowed to tear down can preview one without `--yes`. Previewing is the safety mechanism, so it stays reachable. +noorm's tracking tables live in the `noorm` schema on PostgreSQL and MSSQL, and as `__noorm_*__` tables on MySQL and SQLite. -Contexts created on the `agent` channel (MCP, or an AI agent driving the CLI) never satisfy a confirmation at all. `confirm` collapses to `deny` before `yes` is consulted, so an agent cannot walk through a gate a human was meant to answer. -::: danger -Running teardown against production should be exceedingly rare. If you find yourself doing this regularly, reconsider your deployment workflow. -::: +## In what order does teardown drop things? -## Preserving Tables +Procedures, functions, and views drop before tables, because a schema-bound object on MSSQL locks the table it references: -Some tables should survive resets. Lookup tables, configuration tables, audit logs. Configure these in settings: - -```yaml -# .noorm/settings.yml -teardown: - preserveTables: - - countries - - currencies - - app_settings - - audit_log +```mermaid +flowchart TD + FK["foreign keys"] --> D{"MSSQL?"} + D -->|yes| CK["all CHECK constraints"] + D -->|no| PR[procedures] + CK --> PR + PR --> FN[functions] + FN --> VW[views] + VW --> TB[tables] + TB --> TY[types] ``` -These tables will be skipped during both truncate and teardown operations. Their data and structure remain intact. +`--preserve-schemas a,b` leaves whole schemas alone. Teardown otherwise reaches every non-system schema, including ones noorm did not create. -`ctx.noorm.db.reset()` is the exception: it ignores `preserveTables` on purpose. Reset rebuilds the whole schema from `sql/`, so any table left standing would collide with the build's `CREATE TABLE` and abort the rebuild. +## How do I keep tables through a wipe? -## Post-Teardown Scripts -Re-seed essential data after teardown completes: +List them in settings. `postScript` is relative to the project root: ```yaml # .noorm/settings.yml teardown: - postScript: sql/teardown/seed.sql + preserveTables: + - party_type + postScript: scripts/post-teardown.sql ``` -The `postScript` runs immediately after teardown finishes. Use it to: +| Setting | `db truncate` | `db teardown` | `db reset` | +|---|---|---|---| +| `preserveTables` | kept | kept | ignored, because a surviving table would collide with the rebuild's `CREATE TABLE` | +| `postScript` | not run | runs after the drop | runs after the drop, before the rebuild | -- Insert required lookup data -- Create default admin accounts -- Set up test fixtures +`db truncate --preserve a,b` replaces the settings list for that run. `--only a,b` wipes those tables alone. -```sql --- sql/teardown/seed.sql -INSERT INTO countries (code, name) VALUES - ('US', 'United States'), - ('CA', 'Canada'), - ('MX', 'Mexico'); -INSERT INTO app_settings (key, value) VALUES - ('version', '1.0.0'), - ('maintenance_mode', 'false'); -``` +## How do I reset between tests? -## Using in Tests - -The SDK provides teardown methods with test-oriented safety guards. - -### Basic Test Setup +Truncate between tests while the schema is stable: ```typescript -import { createContext, RequireTestError } from '@noormdev/sdk' - -describe('user service', () => { +// tests/setup.ts +import { createContext } from '@noormdev/sdk' - let ctx +const ctx = await createContext({ config: 'test', requireTest: true, yes: true }) - beforeAll(async () => { - // requireTest: true prevents accidents with real databases - ctx = await createContext({ - config: 'test', - requireTest: true, - }) - await ctx.connect() - }) +await ctx.connect() - beforeEach(async () => { - // Fast reset between tests - await ctx.noorm.db.truncate() - await ctx.noorm.run.file('./seeds/test-data.sql') - }) - - afterAll(async () => { - await ctx.disconnect() - }) - - it('creates a user', async () => { - // Clean database, ready for test - }) - -}) +beforeEach(() => ctx.noorm.db.truncate()) ``` -### The `requireTest` Guard - -When `requireTest: true`, the SDK throws if the config does not have `isTest: true`: - -```typescript -// If 'staging' does not have isTest: true, this throws -const ctx = await createContext({ - config: 'staging', - requireTest: true, // RequireTestError: Config "staging" does not have isTest: true -}) -``` - -This catches configuration mistakes before tests run against the wrong database. - -### Full Reset for Integration Tests - -When schema changes are involved: - -```typescript -beforeAll(async () => { - ctx = await createContext({ config: 'test', requireTest: true }) - await ctx.connect() - - // Full teardown + rebuild - await ctx.noorm.db.reset() -}) -``` - -The `reset()` method combines a full teardown with `run.build({ force: true })` for complete schema reconstruction. It is gated by `db:reset`, which an `admin`-role config satisfies without confirmation. - - -## Scripted Usage - -For CI/CD pipelines, drive the same teardown commands non-interactively. Always pair them with `-y` to skip the confirmation prompt — the CLI refuses to wipe data otherwise. - -### Basic Commands +`requireTest: true` throws `RequireTestError` unless the config has `isTest: true`. `yes: true` answers the confirmation the SDK cannot prompt for. When the schema changed, call `ctx.noorm.db.reset()`, which runs a teardown and a forced build. In CI: ```bash -# Truncate all data -noorm db truncate -y - -# Full teardown -noorm db teardown -y - -# Preview what would be dropped -noorm db teardown --dry-run -``` - -### JSON Output - -```bash -noorm db teardown --json --yes -``` - -```json -{ - "success": true, - "dropped": { - "tables": ["users", "posts", "comments"], - "views": ["active_users"], - "functions": [], - "procedures": [], - "types": [], - "foreignKeys": ["posts_user_id_fkey"] - }, - "count": 4 -} -``` - -`count` sums tables, views, functions, and types, but not procedures or foreign keys. A dry run adds `"dryRun": true`, and a configured post-script adds `postScriptResult`. When a post-script fails, the objects are still gone but the command exits non-zero, because a teardown whose seeding step never ran is half-finished. - -`noorm db truncate --json` reports `truncated`, `preserved`, and `count`, plus `statements` on a dry run. - -### CI/CD Pattern - -```yaml -# GitHub Actions example -test: - runs-on: ubuntu-latest - services: - postgres: - image: postgres:16 - env: - POSTGRES_PASSWORD: test - POSTGRES_DB: test_db - ports: - - 5432:5432 - - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - - - name: Setup test database - env: - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: localhost - NOORM_CONNECTION_DATABASE: test_db - NOORM_CONNECTION_USER: postgres - NOORM_CONNECTION_PASSWORD: test - run: | - npx noorm run build - npx noorm change ff - - - name: Run tests - run: npm test -``` - -### Test Database Reset Script - -```bash -#!/bin/bash -set -e - -# Reset to clean state before test run -noorm db teardown --config test -y -noorm run build --config test -noorm change ff --config test - -echo "Test database ready" +noorm db reset --config test --yes ``` -## Dialect Differences +## How does each dialect empty a table? -noorm generates appropriate SQL for each database: -| Feature | PostgreSQL | MySQL | MSSQL | SQLite | -|---------|------------|-------|-------|--------| -| Truncate | `TRUNCATE TABLE ... CASCADE` | `TRUNCATE TABLE` | `DELETE FROM` | `DELETE FROM` | -| Reset identity | `RESTART IDENTITY` | Automatic with `TRUNCATE` | `DBCC CHECKIDENT ... RESEED` | N/A | -| Disable FKs | `SET session_replication_role` | `SET FOREIGN_KEY_CHECKS=0` | Per-table `NOCHECK CONSTRAINT` | `PRAGMA foreign_keys=OFF` | -| Drop cascade | `CASCADE` | Manual order | Manual order | Manual order | +| Dialect | Empties with | Disables foreign keys with | Restarts identity | +|---|---|---|---| +| PostgreSQL | `TRUNCATE ... CASCADE` | `session_replication_role` | yes | +| MySQL | `TRUNCATE` | `FOREIGN_KEY_CHECKS` | yes | +| MSSQL | `DELETE FROM` | `NOCHECK CONSTRAINT` per table | yes, reseed to 0 | +| SQLite | `DELETE FROM` | `PRAGMA foreign_keys` | no | -Two dialects use DELETE rather than TRUNCATE, for different reasons. SQLite has no TRUNCATE statement. MSSQL has one, but it refuses to run against a table referenced by a foreign key even with constraints set to NOCHECK, so noorm issues a DELETE and reseeds the identity separately. +Foreign keys come back on even when the truncate fails partway. [Teardown internals](/dev/teardown#mssql-truncate-strategy) covers why MSSQL cannot use `TRUNCATE`. -MSSQL also has no session-level foreign key switch, so noorm emits one `ALTER TABLE ... NOCHECK CONSTRAINT ALL` per table rather than using `sp_MSforeachtable`, whose parallel workers deadlock on schema locks. +## What bites -## Best Practices -**Preview first.** Run with `--dry-run` before executing, especially in shared environments. +| Mistake or symptom | Cause | Fix | +|---|---|---| +| PostgreSQL truncate fails: `permission denied to set parameter "session_replication_role"` | Truncate turns off foreign-key triggers with `session_replication_role`, which only superusers, or roles granted `SET` on it, may change | Use a superuser for the test database, or on PostgreSQL 15+ run `GRANT SET ON PARAMETER session_replication_role TO <role>` | +| `"db:teardown" is not allowed on config "prod" (role: operator)` | The matrix denies `db:teardown` to `viewer` and `operator`. No flag overrides a denial | Give the config the `admin` role, or tear down another config | +| `requires confirmation` | The role's cell is `confirm` | Pass `--yes`, or set `NOORM_YES=1` | +| `requires confirmation from a human` | The `agent` channel never confirms | Run it as `user` | +| `db.truncate()` throws `ProtectedConfigError` on an `admin` config | `db:truncate` is `confirm` even for `admin`, and the SDK has no prompt | `createContext({ yes: true })` or `NOORM_YES=1` | +| `db teardown --dry-run` is refused on `operator` | A dry run skips the confirmation, not the role check | Look with `noorm db explore` | +| `Post-teardown script failed: Script failed: no such table` | The script runs after the tables drop. It splits on `;` and skips every statement that starts with `--`, comment line included | Put no comment above a statement, and do not insert into dropped tables | +| `postScript` seed rows are missing after `db reset`, which exits 0 | Reset runs the script before the rebuild recreates the tables | Seed from `db/sql/`, where the build inserts them | +| A preserved table is empty after `db truncate` on PostgreSQL | `TRUNCATE ... CASCADE` also empties every table that references a truncated one, and `--only` does the same | Preserve the tables it references too | +| `--preserve seeds` wipes a table listed in `preserveTables` | The flag replaces the settings list | Repeat every table in the flag | +| `change ff` re-runs every change after a teardown or reset | Teardown marks applied changes `stale`, and the rebuilt schema already holds their effect | Write changes that are safe to run twice: [Add a column](/guide/changes/add-a-column) | -**Preserve audit tables.** If you have logging or audit tables, add them to `preserveTables`. - -**Use truncate for test cycles.** Faster than teardown + rebuild when schema has not changed. - -**Use postScript for seeds.** Re-insert required data automatically after teardown. - -**Check the config's access role.** A `viewer`/`operator` role exists for a reason. Switching a config to `admin` is what makes teardown possible at all, so think twice before doing it to get past a denial. - -```bash -# Safe teardown pattern -noorm db teardown --dry-run # Preview first -noorm db teardown -y # Execute after review -``` +## Next -## What's Next? -- [Execution](/guide/sql-files/execution) - Rebuild schema after teardown -- [Changes](/guide/changes/overview) - How changes work with teardown -- [Schema Explorer](/guide/database/explore) - Verify what was dropped +- [Execution](/guide/sql-files/execution): rebuild the schema after a teardown +- [Forward and revert](/guide/changes/forward-revert): what `stale` changes do on the next `change ff` +- [Stages](/guide/environments/stages): cap a config's role with `protected: true` +- [CLI reference](/headless#db-truncate): every flag for `db truncate`, `db teardown`, `db reset`, and `db drop` diff --git a/docs/guide/database/terminal.md b/docs/guide/database/terminal.md index e9e3aae8..196444d0 100644 --- a/docs/guide/database/terminal.md +++ b/docs/guide/database/terminal.md @@ -1,269 +1,127 @@ # SQL Terminal -## What It Is - -The SQL terminal is an interactive REPL built into noorm. Run queries against your database without leaving the tool or switching to another client. - -Debugging a change? Check if that INSERT worked. Exploring a table structure? Run a quick SELECT. The terminal keeps you in flow while you work. - - -## Launching the Terminal - -Press `Shift+Q` from anywhere in the TUI. The terminal is a global shortcut, not an entry on the Database menu, so there is no `[d]` then `[t]` path to it. On the Database screen, `t` is teardown. - -From a shell, launch straight into it: +Run SQL against a config's database from the shell or the TUI, with every statement checked against the config's role first. ```bash -noorm sql repl # active config -noorm sql repl --config dev # a specific config +noorm sql query "SELECT party_id, order_no, status FROM sales_order ORDER BY 1, 2" ``` -`noorm sql repl` opens the same TUI screen, so everything on this page applies to both. It needs an interactive terminal and refuses to run under `--yes` or `NOORM_YES`; for scripted SQL use `noorm sql query "SELECT 1"` or `noorm sql query -f query.sql` instead. - -The terminal connects to your [active configuration](/guide/environments/configs) automatically. - - -## The Interface - ``` -┌─ SQL Terminal ──────────────────────────────────────────────┐ -│ │ -│ > SELECT * FROM users LIMIT 3; │ -│ │ -│ ┌────┬─────────┬─────────────────┬─────────────────────┐ │ -│ │ id │ name │ email │ created_at │ │ -│ ├────┼─────────┼─────────────────┼─────────────────────┤ │ -│ │ 1 │ Alice │ alice@email.com │ 2024-01-15 09:30:00 │ │ -│ │ 2 │ Bob │ bob@email.com │ 2024-01-16 14:22:00 │ │ -│ │ 3 │ Charlie │ charlie@co.com │ 2024-01-17 11:45:00 │ │ -│ └────┴─────────┴─────────────────┴─────────────────────┘ │ -│ │ -│ 3 rows (12ms) │ -│ │ -│ > _ │ -│ │ -│ [Shift+Tab] Edit mode [Enter] Execute [Shift+Enter] New │ -│ [h] History [Esc] Clear │ -└─────────────────────────────────────────────────────────────┘ +Columns: party_id, order_no, status +{"party_id":1,"order_no":1,"status":"paid"} +{"party_id":2,"order_no":1,"status":"invoiced"} +{"party_id":2,"order_no":2,"status":"open"} +3 rows returned (2ms) ``` -Type your SQL at the `>` prompt and press Enter to execute. The prompt changes to `[EDIT]>` in edit mode, where Enter inserts a newline instead of running the query. - - -## What You Are Allowed to Run - -Before anything reaches the database, noorm classifies the statement and checks that class against the [config's access role](/guide/environments/configs): - -| Class | Permission | `viewer` | `operator` | `admin` | -|-------|------------|----------|------------|---------| -| Reads (`SELECT`, `SHOW`, `DESCRIBE`) | `sql:read` | Yes | Yes | Yes | -| Writes (`INSERT`, `UPDATE`, `DELETE`, `MERGE`) | `sql:write` | No | Yes | Yes | -| Everything else | `sql:ddl` | No | No | Yes | - -`sql:ddl` is the catch-all, not a DDL-keyword list. `CREATE`/`ALTER`/`DROP`/`TRUNCATE`/`GRANT`/`REVOKE`/`SET`, `EXEC`/`CALL`, and anything the classifier cannot make sense of all land there. Unrecognized input fails closed, on the reasoning that a statement nobody can classify could do anything. +`Shift+Q` on any TUI screen opens the active config in an interactive terminal with a results grid and saved history. From a shell, run `noorm sql repl --config dev`, which needs a TTY and exits with a usage error under `--yes` or `NOORM_YES`. [`noorm sql`](/cli/sql) covers `query`, including `-f file.sql` and `--json`. -Classification looks past the outer keyword, which matters more than it sounds: -- A statement is parsed, not pattern-matched, and the **highest** class anywhere in the tree wins. `WITH t AS (DELETE FROM logs ...) SELECT * FROM t` is a write, not a read. -- `EXPLAIN` takes the class of the statement it wraps, because `EXPLAIN ANALYZE` executes that statement. `EXPLAIN (ANALYZE) DELETE FROM t` is a write. -- `SELECT ... INTO` creates a table or writes a file, so it classifies as `sql:ddl` rather than a read. -- A `SELECT` calling a side-effecting or filesystem builtin (`pg_terminate_backend`, `setval`, `pg_read_file`, and similar) is raised to `sql:write`, which is the lowest class a `viewer` cannot reach. -- Multi-statement input takes the highest class present, so one `DROP` after a semicolon governs the whole submission. +## What am I allowed to run? -The denylist behind that last rule covers known-dangerous builtins, not every function that could have a side effect. It stops casual and accidental writes on a `viewer` config; it is not a sandbox for a determined adversary. +The text is parsed and classified before anything reaches the database. The highest class wins: -## Running Queries - -The terminal runs any SQL your database understands, subject to the access role above: - -**SELECT queries** return formatted tables with timing: - -``` -> SELECT id, email FROM users WHERE active = true LIMIT 5; - -┌────┬────────────────────┐ -│ id │ email │ -├────┼────────────────────┤ -│ 1 │ alice@example.com │ -│ 2 │ bob@example.com │ -└────┴────────────────────┘ - -2 rows (8ms) -``` - -**INSERT, UPDATE, DELETE** report affected rows: - +```mermaid +flowchart TD + Q["SQL text"] --> P{"parses?"} + P -->|yes| T["class of every<br/>statement in the tree"] + P -->|no| K["class of each statement<br/>by its first keyword"] + T --> H["highest class wins"] + K --> H + H --> G{"role allows it?"} + G -->|no| X["refused, nothing sent"] + G -->|yes| DB["runs"] ``` -> UPDATE users SET last_login = NOW() WHERE id = 1; -1 row affected (5ms) -``` +| Class | Statements | `viewer` | `operator` | `admin` | +|---|---|---|---|---| +| `sql:read` | `SELECT`, `VALUES`, `TABLE`, an `EXPLAIN` of a read, `SHOW` and `DESCRIBE` on MySQL | yes | yes | yes | +| `sql:write` | `INSERT`, `UPDATE`, `DELETE`, `MERGE` | no | yes | yes | +| `sql:ddl` | everything else: `CREATE`, `ALTER`, `DROP`, `TRUNCATE`, `GRANT`, `SET`, `CALL`, `EXEC`, and input no keyword matches | no | no | yes | -**DDL statements** (CREATE, ALTER, DROP) work too, on an `admin`-role config: +`src/core/policy/classify.ts` decides these cases: -``` -> CREATE INDEX users_email_idx ON users(email); +| Input (PostgreSQL) | Class | Why | +|---|---|---| +| `WITH gone AS (DELETE FROM order_line RETURNING *) SELECT * FROM gone` | `sql:write` | a `DELETE` inside the `WITH` | +| `EXPLAIN ANALYZE DELETE FROM order_line` | `sql:write` | `EXPLAIN` takes the class of what it runs | +| `SELECT * INTO order_copy FROM sales_order` | `sql:ddl` | it creates a table | +| `SELECT pg_terminate_backend(pid) FROM pg_stat_activity` | `sql:write` | a side-effecting builtin on the denylist | -OK (42ms) -``` +The denylist names known-dangerous builtins, such as `nextval` and `pg_read_file`. It stops accidental writes on a `viewer` config and is not a sandbox. -## Stopping a Query That Will Not Come Back +## How do I work in the terminal? -`Escape` while a query is running gives the terminal back. What that does to the -database depends on the dialect, and the terminal says which one happened: -| Dialect | On `Escape` | What you see | -|---------|-------------|--------------| -| PostgreSQL | `pg_cancel_backend` on a second connection | `Cancelled. The server was asked to stop the query.` | -| MySQL | `KILL QUERY` on a second connection | `Cancelled. The server was asked to stop the query.` | -| SQL Server | nothing reaches the server | `Stopped waiting. The query may still be running on the server.` | -| SQLite | nothing reaches the server | `Stopped waiting. The query may still be running on the server.` | +| Where | Keys | +|---|---| +| Query input | `Enter` runs, `Shift+Enter` adds a line. `Shift+Tab` toggles edit mode (`[EDIT]>`), where `Enter` adds a line. `Tab` inserts four spaces. `↑` `↓` recall earlier queries and `h` opens history, both on an empty input | +| Results grid | `↑` `↓` move. `Enter` opens the row as YAML or JSON (`←` `→` change row, `f` switches format). `/` filters, `s` sorts, `c` clears both | +| Between them | A query that returns rows opens the grid. `Tab` or `Esc` returns to the query | +| Any time | `Esc` stops a running query or connect, otherwise clears the input, and on an empty input leaves | -The difference is not cosmetic. On the bottom two rows the query keeps running -to completion, holding locks and burning CPU, and the only thing that changed is -that noorm stopped listening for the answer. Kill it from the server side if it -matters. +A statement without rows shows `✓ 1 row(s) affected (5ms)` or `✓ Query executed (42ms)`. A failure shows `✗` and the server's message. -SQL Server is a limitation of the driver layer rather than of the database: -tedious exposes a per-request cancel, but Kysely's MSSQL dialect owns the -request object and never hands it out. SQLite has no second connection to -interrupt the first from. -`Escape` also works while the terminal is still connecting, which is what a -network problem looks like before a query ever runs. +## How do I stop a query? -## Result Formatting +`Esc` returns the terminal to you. Whether the server stops working depends on the dialect: -Results display in a formatted table with column alignment: +| Dialect | On `Esc` | +|---|---| +| PostgreSQL | `pg_cancel_backend` on a second connection | +| MySQL | `KILL QUERY` on a second connection | +| SQL Server | nothing reaches the server | +| SQLite | nothing reaches the server | -``` -┌─────┬────────────────────┬────────┬───────────────────────┐ -│ id │ email │ name │ created_at │ -├─────┼────────────────────┼────────┼───────────────────────┤ -│ 1 │ alice@example.com │ Alice │ 2024-01-15T10:30:00Z │ -│ 2 │ bob@example.com │ Bob │ 2024-01-15T11:45:00Z │ -└─────┴────────────────────┴────────┴───────────────────────┘ - -2 rows (15ms) -``` +The first two show `Cancelled. The server was asked to stop the query.` The last two show `Stopped waiting. The query may still be running on the server.` There the query may keep running and holding its locks, so kill it server-side if that matters. -Long values are truncated with an ellipsis to fit the column. There is no per-cell detail view, so to read a value in full, re-run the query selecting fewer columns. -Once a query returns rows, press `Tab` to move into the results table. From there: +## Where does history go? -| Key | Action | -|-----|--------| -| `↑` / `↓` | Navigate rows | -| `/` | Filter rows | -| `s` | Sort by a column | -| `c` | Clear the filter and sort | -| `Tab` or `Esc` | Return to the query input | -Errors display inline with the original query: +Only the interactive terminal records history, one file set per config. `sql query` records nothing. A statement the role refused is recorded as a failure. +```bash +noorm sql history -n 2 ``` -> SELECT * FROM nonexistent; - -Error: relation "nonexistent" does not exist -``` - - -## Query History - -Every query you run is saved. Browse and re-execute past queries without retyping. - - -### Browsing History - -Press `h` to open the history viewer. The input must be empty, so clear the prompt first: ``` -┌─ Query History ─────────────────────────────────────────────┐ -│ │ -│ 1. SELECT * FROM users LIMIT 10 12ms ✓ 1h ago │ -│ 2. UPDATE users SET active = true... 5ms ✓ 2h ago │ -│ 3. SELECT * FROM nonexistent - ✗ 2h ago │ -│ 4. INSERT INTO logs (msg) VALUES... 8ms ✓ 3h ago │ -│ │ -│ [r] Re-run [Enter] View result [c] Clear [↑/↓] Navigate │ -└─────────────────────────────────────────────────────────────┘ -``` - -- `✓` indicates successful queries -- `✗` indicates failed queries -- `r` sends the selected query back to the terminal, pre-filled and ready to edit before you run it -- `Enter` opens the stored results for that entry - -The history viewer has no search. To find an old query, scroll with the arrow keys. - - -### Viewing Saved Results +SQL history for 'dev' (2 entries): -Select a history entry and press `Enter` to view the full results from when the query ran. Results are stored gzipped, so you can review query output even after the data has changed. - -Only queries that returned rows have stored results. Pressing `Enter` on a failed entry shows its error message instead, and on a row-less entry it tells you there is nothing stored. - - -## Writing Multi-Line Queries - -The terminal has no tab completion. `Tab` inserts four spaces, so you can indent a query by hand. - -There are two ways to write across several lines. `Shift+Enter` inserts a newline without running anything. For a longer statement, press `Shift+Tab` to enter edit mode, where plain `Enter` becomes a newline and the prompt changes to `[EDIT]>`. Press `Shift+Tab` again to leave edit mode and get Enter-to-execute back. - - -## Keyboard Shortcuts - -| Key | Action | -|-----|--------| -| `Enter` | Execute query (insert a newline in edit mode) | -| `Shift+Enter` | Insert a newline | -| `Shift+Tab` | Toggle edit mode | -| `Tab` | Insert four spaces, or move between the query and the results table once a query has returned rows | -| `Up` / `Down` | Navigate command history when the input is empty, otherwise move the cursor | -| `Left` / `Right` | Move the cursor | -| `Escape` | Stop a running query or a connect in progress; otherwise clear the input, and on an empty input leave the terminal | -| `h` | Open the history viewer (empty input only) | - - -## Clearing History - -Open the history viewer with `h`, then press `c` for the clear menu: + ✓ Oct 21 at 01:42:51 5ms · 1 row(s) + UPDATE sales_order SET status = 'paid' WHERE party_id = 2 AND order_no = 1 -``` -┌─ Clear History ─────────────────────────────────────────────┐ -│ │ -│ 1. Clear last 3 months │ -│ 2. Clear all history │ -│ │ -│ History: 47 entries, 2.3 MB stored │ -│ │ -└─────────────────────────────────────────────────────────────┘ + ✗ Oct 21 at 01:42:51 2ms · relation "sale_order" does not exist + SELECT * FROM sale_order ``` -"Clear last 3 months" removes entries older than three months and keeps the rest. Either option deletes the stored result files along with their history entries. `noorm sql clear` does the same job from a shell. +In the TUI, `h` opens the same list. `r` sends the selected query back to the input, `Enter` opens its stored rows or error, and `c` opens the clear menu. From a shell: `noorm sql clear --older-than 3 --yes`. +::: warning History is not encrypted +Query text sits in `.noorm/state/history/<config>.json` and returned rows sit gzipped in `.noorm/state/history/<config>/`. Both are owner-only (`0600`, in a `0700` directory), but anything a `SELECT` pulled from a credentials or PII table is readable by your user account. Clear the history after a sensitive query. +::: -## Tips -**Always use LIMIT** when exploring unfamiliar tables. A table with millions of rows will slow down your terminal. +## What bites -**Preview before modifying** - Run a SELECT with your WHERE clause before executing UPDATE or DELETE: -``` -> SELECT id, email FROM users WHERE last_login < '2023-01-01'; --- Review the results -> DELETE FROM users WHERE last_login < '2023-01-01'; -``` +| Symptom | Cause | Fix | +|---|---|---| +| `"sql:write" is not allowed on config "ro" (role: viewer).` | The class needs a higher role than your channel has. In an agent session the channel is `agent`, which defaults to `viewer` | Raise the role, or see [channels](/guide/environments/configs#which-channel-is-a-process-on) | +| `SHOW search_path` (PostgreSQL) or `PRAGMA …` (SQLite) is refused below `admin` | Both parse as statements the classifier does not list as reads, so they count as `sql:ddl` | `SELECT current_setting('search_path')`, or `SELECT * FROM pragma_table_info('sales_order')` | +| `noorm sql -f reports/monthly.sql` prints `Unknown command` | The bare form needs the SQL as the first token after `sql` | `noorm sql query -f reports/monthly.sql` | +| `noorm sql history` is empty after you ran queries | They ran through `sql query` | Run them in the terminal | -**History is not encrypted** - Query text sits in plain JSON at `.noorm/state/history/<config>.json`, and returned rows sit gzipped beside it in `.noorm/state/history/<config>/`. Neither is encrypted, unlike `state.enc`. Both are written owner-only (`0600`, in a `0700` directory), but anything a `SELECT` pulled out of a credentials or PII table is readable by your user account. Avoid running queries that contain passwords or API keys, and clear the history after one slips through. +## Next -## What's Next? -- [Schema Explorer](/guide/database/explore) - Browse tables, views, and other database objects -- [Configs](/guide/environments/configs) - Switch between different databases -- [Execution](/guide/sql-files/execution) - Run SQL files with change detection +- [Schema Explorer](/guide/database/explore): read the structure before you query it +- [Configs](/guide/environments/configs): roles and channels +- [`noorm sql`](/cli/sql): flags for `query`, `history`, and `clear` +- [Run files](/guide/sql-files/execution): run SQL files with change detection diff --git a/docs/guide/database/transfer.md b/docs/guide/database/transfer.md index 362b2520..bad06167 100644 --- a/docs/guide/database/transfer.md +++ b/docs/guide/database/transfer.md @@ -1,388 +1,151 @@ # Data Transfer -Move data between databases using your existing noorm configs. Tables are transferred in foreign key order so referential integrity is maintained. Cross-dialect transfers (e.g., PostgreSQL to MySQL) are supported with automatic type conversion. +Copy rows from one database to another, parents before children, using two configs you already have. - -## When You Need This - -You have a staging database full of test data that you want to push to a fresh QA environment. Or you need to seed a local dev database from a shared development server. Or you're migrating data between two production instances. - -The databases share the same schema—same tables, same columns. You need the data moved, not the structure. - - -## Supported Dialects - -| Dialect | Supported | Same-server optimization | Cross-dialect | -|---------|-----------|--------------------------|---------------| -| PostgreSQL | Yes | No | Yes | -| MySQL | Yes | Yes (cross-database on same host) | Yes | -| MSSQL | Yes | Yes (cross-database on same host) | Yes | -| SQLite | No | — | — | - -Same-server optimization uses direct `INSERT...SELECT` SQL instead of reading data into the application and writing it back. Faster for large datasets. - -PostgreSQL never takes that path. Without `dblink` or `postgres_fdw` it cannot reach another database, and when both configs name the same database the statement degenerates into copying a table into itself. PostgreSQL transfers always run through the batch path instead. - -Cross-dialect transfers convert data types automatically through a universal type system. Most common types map cleanly; some dialect-specific features (like PostgreSQL arrays) become JSON in dialects that don't support them natively. - - -## Interactive Mode - -From the home screen: - -1. Press `d` to enter the database menu -2. Select a config -3. Choose the transfer option -4. Walk through the wizard: - - Pick destination (another config, export to file, or import from file) - - Select tables (all or specific) - - Choose a conflict strategy - - Review the plan - - Execute - -The TUI shows live progress per table with row counts and batch completion. - -**Export/Import options** appear in the destination selection list: -- "Export to .dt file" — saves data to portable files -- "Import from .dt file" — loads data from previously exported files - - -## Headless Mode - -Exactly one of `--to`, `--export`, or `--import` is required; they are mutually exclusive. The source is the active config unless `-c` names another. +Plan a copy of the order tables from `staging` to `qa`: ```bash -# Transfer all tables from active config to backup -noorm db transfer --to backup - -# Specify source explicitly with -c -noorm db transfer -c staging --to production - -# Transfer specific tables only -noorm db transfer --to backup --tables users,posts,comments +noorm db transfer -c staging --to qa \ + --tables party,product,sales_order,order_line \ + --dry-run ``` -There is no positional source argument. `noorm db transfer staging --to production` does not name `staging` as the source; use `-c staging` (or `--config staging`). - -### Dry Run - -Preview what will happen without transferring anything: - -```bash -noorm db transfer --to backup --dry-run ``` +Dry run - transfer plan: + Same server: no + Tables: 4 + Estimated rows: 1250 -Output shows: -- Whether same-server optimization applies -- Table count and estimated row counts -- Transfer order (FK dependency sorted) -- Warnings (missing destination tables, etc.) - -### Conflict Strategies - -When destination tables already contain data with matching primary keys: - -```bash -# Abort on first conflict (default) -noorm db transfer --to backup --on-conflict fail - -# Skip rows that already exist -noorm db transfer --to backup --on-conflict skip - -# Update existing rows with source data -noorm db transfer --to backup --on-conflict update - -# Delete and re-insert conflicting rows -noorm db transfer --to backup --on-conflict replace + Transfer order: + party - 120 rows + product - 40 rows + sales_order - 310 rows (after: party) + order_line - 780 rows (after: sales_order, product) ``` -| Strategy | What happens | -|----------|-------------| -| `fail` | Stops transfer on first primary key conflict | -| `skip` | Leaves existing rows untouched, inserts new ones | -| `update` | Overwrites non-PK columns on existing rows | -| `replace` | Removes conflicting rows entirely, inserts fresh copies | +Remove `--dry-run` to run it: -### Options - -```bash -# Set batch size for cross-server transfers (default: 1000) -noorm db transfer --to backup --batch-size 5000 - -# Clear destination tables before transfer -noorm db transfer --to backup --truncate - -# Don't disable foreign key checks (risky for dependent tables) -noorm db transfer --to backup --no-fk - -# Don't preserve identity/auto-increment values -noorm db transfer --to backup --no-identity -``` - -### JSON Output - -```bash -noorm db transfer --to backup --json ``` - -Transfer result: - -```json -{ - "success": true, - "status": "success", - "tables": [ - { - "table": "users", - "status": "success", - "rowsTransferred": 1500, - "rowsSkipped": 0, - "durationMs": 234 - } - ], - "totalRows": 1500, - "durationMs": 1234, - "fkChecksRestored": true -} -``` - -Check `fkChecksRestored`. When it is `false`, the transfer finished but foreign key enforcement could not be turned back on at the destination, and referential integrity is still disabled there until you re-enable it by hand. The CLI also writes a warning to stderr in that case. - -Dry run result: - -```json -{ - "success": true, - "dryRun": true, - "sameServer": false, - "tableCount": 5, - "estimatedRows": 10000, - "tables": [ - { - "name": "users", - "rowCount": 1000, - "hasIdentity": true, - "dependsOn": [] - } - ], - "warnings": [] -} +Transfer complete: success + Total rows: 1250 + Tables: 4 success, 0 failed + Duration: 0.84s ``` -## What Happens During a Transfer +## In what order are tables copied? -1. **Planning** — noorm reads the source schema, builds a foreign key dependency graph, and sorts tables so parents are transferred before children. A circular foreign key relationship cannot be sorted, so noorm falls back to the source's own table order and records a warning in the plan. -2. **FK checks disabled** — Foreign key constraints are temporarily disabled on the destination to avoid ordering issues within batches. +noorm copies a table after every table it references. An arrow means "copied before": -3. **Truncate (optional)** — If `--truncate` is set, destination tables are cleared first. PostgreSQL uses `TRUNCATE ... CASCADE`, MSSQL uses `DELETE` (can't truncate with FKs), MySQL uses standard `TRUNCATE`. - -4. **Data transfer** — Each table is transferred in order. Same-server uses direct SQL. Cross-server reads batches from source and writes them to destination. - -5. **Identity handling** — Identity/auto-increment values from the source are preserved by default. Sequences are reset after transfer to continue from the max value. - -6. **FK checks re-enabled** — Constraints are turned back on. - - -## Requirements - -- Destination tables must **already exist** with compatible column structure -- The noorm project must have configs for both source and destination databases -- The destination config's access role must permit `db:reset`: `viewer` is denied, `operator` and `admin` may write. From the SDK, an `operator` destination needs `yes: true` on the context (or `NOORM_YES=1`), since there is no prompt to answer -- For cross-dialect transfers, column types must be convertible (most are; check the dry-run output for warnings) - -noorm's own tracking tables are never transferred. They are filtered out of the plan on both ends. - - -## Common Patterns - -### Seed a dev database from staging - -```bash -noorm db transfer -c staging --to local --truncate +```mermaid +flowchart TD + party --> sales_order + sales_order --> order_line + product --> order_line ``` -Clears the local database first, then copies everything from staging. +Only foreign keys between the selected tables count, so a parent outside `--tables` must already exist in the destination. Tables with no key between them can appear in either order. On a foreign key cycle, noorm copies in alphabetical order and warns `Circular FK dependency detected`. -### Incremental sync with skip -```bash -noorm db transfer --to backup --on-conflict skip -``` +## What does a run do? -Only inserts rows that don't already exist in backup. Existing rows are left untouched. -### Upsert from source of truth +Foreign keys stay off at the destination until every table is copied: -```bash -noorm db transfer --to target --on-conflict update +```mermaid +flowchart TD + P["plan: sort tables,<br/>parents first"] --> X["disable FK checks<br/>on destination"] + X --> T["next table in order"] + T --> C{"--truncate?"} + C -->|yes| TR["empty the table"] + C -->|no| CP["copy rows"] + TR --> CP + CP -->|more tables| T + CP -->|done| E["re-enable FK checks"] + E --> R["status and<br/>fkChecksRestored"] ``` -Updates all existing rows with the latest data from source, inserts new rows. +A failed table does not stop the run. The rest still copy, and the status is `partial`, or `failed` if none succeeded. A failed table can keep rows it inserted before the error. -### Transfer specific tables -```bash -noorm db transfer --to backup --tables users,user_preferences -``` +## What happens when a row already exists? -Only transfers the specified tables. FK dependencies between selected tables are still respected. -### CI/CD test data setup +A row exists when its primary key is already in the destination: -```bash -noorm db transfer -c staging --to ci-test --truncate --on-conflict fail --json -``` - -Clean transfer for test environments. JSON output for pipeline integration. Fails fast if anything goes wrong. +| `--on-conflict` | Existing row | +|---|---| +| `fail` (default) | The table fails at its first conflict | +| `skip` | Left as is | +| `update` | Non-key columns take the source values | +| `replace` | MySQL deletes and reinserts it. MSSQL overwrites every column. PostgreSQL leaves it as is | -### Cross-dialect migration +A cross-dialect transfer treats `update` and `replace` as `skip`. -```bash -noorm db transfer -c postgres-legacy --to mysql-new --dry-run -noorm db transfer -c postgres-legacy --to mysql-new -``` -Migrate from PostgreSQL to MySQL. Run `--dry-run` first to check for type conversion warnings. +## What do I need first? -## File Export/Import +| Needed | Detail | +|---|---| +| Destination tables | Present, with compatible columns. A missing one warns `Table "x" exists in source but not destination`, then fails | +| Saved configs | `--to` names the destination. The source is the active config, or `-c` | +| Destination role | `db:reset`: `viewer` denied, `operator` needs `--yes` or `NOORM_YES=1`, `admin` allowed. `--dry-run` needs `transfer:plan`, which `operator` has | +| Dialect | PostgreSQL, MySQL, or MSSQL, in any pairing. SQLite is refused | -Export data to portable `.dt` files for backup, sharing, or migration without a live destination database. +noorm's own tracking tables never transfer. Two MySQL or two MSSQL configs on one host and port copy each table with one `INSERT ... SELECT`, but only with `--on-conflict fail`. Everything else, PostgreSQL included, copies in batches of `--batch-size` rows. -### File Formats -| Extension | Description | -|-----------|-------------| -| `.dt` | Line-delimited JSON5, uncompressed and human-readable | -| `.dtz` | The same content, gzipped | -| `.dtzx` | Gzipped and AES-256-GCM encrypted (requires passphrase) | +## How do I move tables through a file? -`.dt` and `.dtz` stream row by row on import. `.dtzx` is decrypted and inflated whole before the first row is read, so it carries a size ceiling the other two do not. - -### Export to Files - -`--tables` defaults to every table in the source. Pass it to narrow the export. ```bash -# Export every table to a directory -noorm db transfer --export ./backup/ - -# Export single table -noorm db transfer --export ./backup/users.dt --tables users - -# Export multiple tables to a directory -noorm db transfer --export ./backup/ --tables users,posts,comments - -# Export compressed -noorm db transfer --export ./backup/ --tables users,posts --compress - -# Export encrypted (implies compression) -noorm db transfer --export ./backup/ --tables users --passphrase "my-secret" +noorm db transfer -c staging --export ./backup/ --tables party,product +noorm db transfer -c qa --import ./backup/party.dt ``` -An export that would write nothing is an error, not a silent success: both an empty `--tables` list and a source with no tables fail rather than leaving you with an empty backup you believe is good. +| Extension | Content | Made by | +|---|---|---| +| `.dt` | line-delimited JSON5 | the default | +| `.dtz` | the same, gzipped | `--compress` | +| `.dtzx` | gzipped and AES-256-GCM encrypted | `--passphrase`, 12 characters or more | -On an interactive terminal, omitting `--passphrase` for a `.dtzx` path prompts for it with masked input, which keeps the secret out of your shell history. Non-interactive runs need the flag. +One table writes to the path you give. Several write one file each into that directory. A path ending in `.dtzx` prompts for the passphrase on a terminal. The format is in [Data transfer internals](/dev/transfer#file-export-import). -**Path rules:** -- Single table → path is the output file, with the extension appended if missing -- Multiple tables → path is a directory, noorm creates `<table>.dt` per table +Import takes one file per call and leaves foreign keys on, so import parents first. -### Import from Files -```bash -# Import from .dt file -noorm db transfer --import ./backup/users.dt +## How do I script it? -# Import with upsert -noorm db transfer --import ./backup/users.dtz --on-conflict update -# Import encrypted file -noorm db transfer --import ./backup.dtzx --passphrase "my-secret" +| Exit | Meaning | +|---|---| +| `0` | Every table succeeded | +| `1` | No table succeeded, or the run failed before copying | +| `2` | A flag or config name was wrong. Nothing ran | +| `3` | Some tables succeeded and some failed: a mixed state | -# Validate schema compatibility only -noorm db transfer --import ./backup/users.dt --dry-run -``` +With `--json`, read `fkChecksRestored`. `false` means the data moved but the destination still has foreign keys off, and `status` does not change. -### Export/Import JSON Output - -Export result (single table): - -```json -{ - "success": true, - "mode": "export", - "filepath": "./backup/users.dt", - "tables": [ - { "table": "users", "filepath": "./backup/users.dt", "rowsExported": 1500, "bytesWritten": 45230 } - ], - "totalRows": 1500, - "totalBytes": 45230 -} -``` -Export result (multiple tables): - -```json -{ - "success": true, - "mode": "export", - "directory": "./backup/", - "tables": [ - { "table": "users", "filepath": "./backup/users.dt", "rowsExported": 1500, "bytesWritten": 45230 }, - { "table": "posts", "filepath": "./backup/posts.dt", "rowsExported": 800, "bytesWritten": 23100 } - ], - "totalRows": 2300, - "totalBytes": 68330 -} -``` +## What bites -Import result: -```json -{ - "success": true, - "mode": "import", - "filepath": "./backup/users.dt", - "rowsImported": 1500, - "rowsSkipped": 0 -} -``` +| Mistake or symptom | Cause | Fix | +|---|---|---| +| `noorm db transfer staging --to qa` copies from the active config | There is no positional source | `-c staging` | +| `Failed to disable FK checks` | The destination role cannot disable constraints. Nothing was copied | `--no-fk`, which relies on the parents-first order, or a more privileged role | +| Exit `3` | A table hit a conflict under `fail`, or had no destination | Read `tables[].error` in `--json`. For conflicts, rerun with `--on-conflict skip` | +| `--truncate` empties destination tables outside `--tables` (PostgreSQL) | `TRUNCATE ... CASCADE` also empties every table that references the truncated one | Put the child tables in `--tables` | +| `--import --dry-run` imports for real | `--dry-run` only plans database-to-database copies. `--export` ignores it too | Preview the schema in the TUI import wizard | -### Common Export/Import Patterns -**Backup specific tables before risky operation:** +## Next -```bash -noorm db transfer --export ./pre-migration-backup/ --tables users,orders --compress -# ... run migration ... -# If something goes wrong: -noorm db transfer --import ./pre-migration-backup/users.dt --truncate -noorm db transfer --import ./pre-migration-backup/orders.dt --truncate -``` - -**Share test data with team:** - -```bash -# Export encrypted for sharing -noorm db transfer --export ./fixtures.dtzx --passphrase "team-secret" --tables users,posts -# Teammate imports -noorm db transfer --import ./fixtures.dtzx --passphrase "team-secret" -``` - -**Cross-dialect migration via file:** - -```bash -# Export from PostgreSQL -noorm use postgres-source -noorm db transfer --export ./migration-data/ --tables users,posts - -# Import into MySQL -noorm use mysql-target -noorm db transfer --import ./migration-data/users.dt -noorm db transfer --import ./migration-data/posts.dt -``` +- [Teardown](/guide/database/teardown): empty or rebuild the destination first +- [Data transfer internals](/dev/transfer): planner, executors, file format +- [Execution](/guide/sql-files/execution): build the destination schema before copying +- [CLI reference](/headless#db-transfer): every `db transfer` flag diff --git a/docs/guide/deployment.md b/docs/guide/deployment.md index 7693f03d..d1e83b75 100644 --- a/docs/guide/deployment.md +++ b/docs/guide/deployment.md @@ -1,30 +1,10 @@ # Deploying an Application -On your laptop, `createContext({ config: 'dev' })` works because two things are sitting on disk: an encrypted `.noorm/state/state.enc` holding your configs, and an identity keypair in `~/.noorm/`. Neither belongs in a production image. You do not want your team's staging credentials baked into a container, and you certainly do not want a developer's private key there. - -The good news is that noorm already runs without either. The mode CI uses — configuration from environment variables alone — is the same mode your application should use in production. This guide covers what changes between the laptop and the cluster. - - -## Split the job in two - -Most deployment pain comes from treating "get the schema up to date" and "serve traffic" as one step. They have opposite requirements: - -| | Schema delivery | Application runtime | -|---|---|---| -| **Runs** | Once per release | In every replica, continuously | -| **Needs** | Write access, your `sql/` and `changes/` directories | A connection and nothing else | -| **Scales** | Exactly one at a time | Horizontally, to N replicas | -| **Surface** | `noorm run build`, `noorm change ff` | `createContext()` | - -Keep them apart and horizontal scaling is uneventful. Fold schema delivery into application boot and every new replica becomes a schema-mutating process racing its siblings. - - -## Runtime: connect from the environment - -When no stored config is in play and `NOORM_CONNECTION_DIALECT` and `NOORM_CONNECTION_DATABASE` are both set, `createContext()` builds a config from the environment alone. No project directory, no identity, no state file: a missing `state.enc` resolves to empty state rather than an error, and nothing asks for a private key. +Deliver the schema once per release, and give every application replica a connection and nothing else. ```typescript +// src/db.ts import { createContext } from '@noormdev/sdk'; import type { DB } from './schema'; @@ -33,158 +13,83 @@ export const ctx = await createContext<DB>(); await ctx.connect(); ``` -That is the whole runtime setup. The variables: +```bash +# environment of each replica +NOORM_CONNECTION_DIALECT=postgres +NOORM_CONNECTION_HOST=db.internal +NOORM_CONNECTION_DATABASE=order_to_cash +NOORM_CONNECTION_USER=app +NOORM_CONNECTION_PASSWORD=$DB_PASSWORD +``` -| Variable | Required | Notes | -|----------|----------|-------| -| `NOORM_CONNECTION_DIALECT` | Yes | `postgres`, `mysql`, `sqlite`, or `mssql` | -| `NOORM_CONNECTION_DATABASE` | Yes | Together with `DIALECT`, this is what triggers environment-only mode | -| `NOORM_CONNECTION_HOST` | No | Defaults to `localhost` | -| `NOORM_CONNECTION_PORT` | No | Defaults per dialect: `5432` / `3306` / `1433` | -| `NOORM_CONNECTION_USER` | Usually | | -| `NOORM_CONNECTION_PASSWORD` | Usually | Read from your secret store, never from the image | -| `NOORM_CONNECTION_SSL` | No | | -| `NOORM_CONNECTION_POOL_MIN` | No | Defaults to `0` | -| `NOORM_CONNECTION_POOL_MAX` | No | Defaults to `10` | +With no stored config, `createContext()` builds a config named `__env__` from these variables. The image needs no `.noorm/` directory, identity, or state file. Install the driver beside the SDK (`npm install @noormdev/sdk kysely pg`), because it is an optional peer dependency. -The resulting config is named `__env__`, which is what you will see in any error message it raises. +Unset variables take defaults: `_HOST` is `localhost`, `_PORT` follows the dialect (`5432`, `3306`, `1433`), and the pool runs from `_POOL_MIN=0` to `_POOL_MAX=10`. -Install the driver alongside the SDK — it is an optional peer dependency, so noorm does not pull it in for you: -```bash -npm install @noormdev/sdk kysely pg -``` +## Why split schema delivery from the application? -## One context per process +A replica that changes the schema on boot races its siblings. Run the schema step once, and let replicas start after it: -A context owns a connection pool. Create it once when the process starts, share it across requests, and close it on shutdown: +```mermaid +flowchart TD + MIG["migrate job<br/>noorm change ff"] -->|exits 0| R1["replica 1<br/>createContext()"] + MIG -->|exits 0| R2["replica 2<br/>createContext()"] + MIG -->|exits 0| RN["replica N"] + R1 --> DB[(database)] + R2 --> DB + RN --> DB +``` -```typescript -const ctx = await createContext<DB>(); +The migrate step is a release step, an init container, or a Kubernetes `Job`. It connects as a deploy user through its own `NOORM_CONNECTION_*` variables, and only it needs `db/sql/` and `db/changes/`. An empty database starts with one `noorm run build`. Every later release runs `change ff` only. If your changes render vault secrets, it also needs an enrolled identity: [CI/CD Automation](/guide/automation/ci#how-do-i-set-up-prod-ci). -await ctx.connect(); -process.on('SIGTERM', async () => { +## Why one context per process? - await server.close(); - await ctx.disconnect(); - process.exit(0); -}); -``` -Calling `createContext()` per request would open a fresh pool per request, which is the classic way to exhaust a database's connection limit under load. +`connect()` opens a pool of up to `NOORM_CONNECTION_POOL_MAX` connections, so a context per request opens a pool per request. Create one at startup, share it, and call `ctx.disconnect()` on `SIGTERM`. -That limit is the number worth doing arithmetic on before you scale out. `NOORM_CONNECTION_POOL_MAX` is a **per-replica** ceiling, so your real demand is replicas × pool max. Twenty replicas at the default of 10 will try to open 200 backends — comfortably past a stock PostgreSQL's `max_connections` of 100. Either lower the per-replica pool or put PgBouncer in front: +Demand is replicas times pool max. Twenty replicas at the default ask for 200 backends, past PostgreSQL's default `max_connections` (typically 100). Lower the pool: ```bash NOORM_CONNECTION_POOL_MAX=4 # 20 replicas x 4 = 80 backends ``` -## Give the runtime the access it actually needs +## How do I limit what the runtime can do? -Your application queries tables. It does not need to build schemas or apply changes, and a config that permits those operations is a wider blast radius than the job requires. Configs carry per-channel roles, and environment-only configs default to `user: admin` — full access. -Narrow it: +Environment-only configs default to `user: admin`. The application only queries, so narrow it. Set both channels, because the access object needs both: ```bash NOORM_ACCESS_USER=viewer NOORM_ACCESS_AGENT=false ``` -With those set, Kysely queries work exactly as before, but the management operations refuse: - -``` -ProtectedConfigError: Cannot run.build on config "__env__": -"run:build" is not allowed on config "__env__" (role: viewer). -``` - -Set both variables or neither. The access object requires both channels, so setting only `NOORM_ACCESS_USER` fails config validation at startup rather than falling back to a default. - -This is defense in depth, not a substitute for database-level permissions. Grant the runtime's database user only the privileges it needs; the noorm role stops the SDK's own management surface, not hand-written SQL. - -See [Access Roles](/guide/environments/configs#access-roles) for what each role permits. - - -## Schema delivery runs once, elsewhere - -Run `build` and `change ff` as a release step, an init container, or a Kubernetes `Job` — anywhere that runs exactly once per deploy, before the new replicas take traffic: +Kysely queries still work. The management surface refuses: -```yaml -- name: Apply schema - env: - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} - NOORM_CONNECTION_DATABASE: myapp - NOORM_CONNECTION_USER: ${{ secrets.DB_DEPLOY_USER }} - NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_DEPLOY_PASSWORD }} - run: | - noorm run build - noorm change ff ``` - -noorm takes a database lock around change execution, so a second run cannot interleave with the first. It does not queue: the lock is acquired with `wait: false`, so the loser fails immediately with a lock error naming the holder. Treat that as a safety net, not a design. Fold schema delivery into application boot and the second replica to start does not wait its turn, it crashes. - -If your changes render vault-backed secrets, the deploy step needs an enrolled identity too. That flow is covered in [CI/CD Automation](/guide/automation/ci#prod-ci) and applies unchanged here. - - -## Docker - -Nothing noorm-specific goes in the image. No `.noorm/`, no identity, no config: - -```dockerfile -FROM node:22-slim - -WORKDIR /app -COPY package*.json ./ -RUN npm ci --omit=dev -COPY dist ./dist - -CMD ["node", "dist/server.js"] -``` - -Configuration arrives as environment variables at run time: - -```yaml -services: - app: - image: myapp:latest - environment: - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: db - NOORM_CONNECTION_DATABASE: myapp - NOORM_CONNECTION_USER: app - NOORM_CONNECTION_PASSWORD: ${DB_PASSWORD} - NOORM_CONNECTION_POOL_MAX: 4 - NOORM_ACCESS_USER: viewer - NOORM_ACCESS_AGENT: 'false' +ProtectedConfigError: Cannot run.build on config "__env__": "run:build" is not allowed on config "__env__" (role: viewer). ``` -The `sql/` and `changes/` directories only need to be present wherever schema delivery runs — the CI checkout, or a separate migration image. Your application image does not need them. +This stops the SDK's own `ctx.noorm` operations, not hand-written SQL, so grant the runtime's database user only the privileges it needs. The migrate job needs its own environment without these variables. [Access roles](/guide/environments/configs#access-roles) lists what each role permits. -## One process, one database +## What bites -Connection details come from the process environment, and `createContext()` reads that environment when it resolves. There is no `createContext({ host, user, password })` overload today, which has a consequence worth stating plainly: **in environment-only mode, a process serves one database.** (A process that does have a `state.enc` can hold several contexts at once, one per stored config, by passing `createContext({ config: 'name' })`. That route is not available here, because environment-only mode is what you get when there is no state file.) -If you are tempted to reassign `process.env` between calls to reach a second database, don't. The environment is process-global while `createContext()` is asynchronous, so two concurrent calls interleave and the later assignment wins for both: - -```typescript -// Broken — both contexts connect to tenant_b. -await Promise.all([ - forTenant('tenant_a'), - forTenant('tenant_b'), -]); -``` - -For multi-tenant or shard-per-customer designs, run a process per database, or hold a long-lived context per tenant created serially at startup rather than per request. +| Symptom | Cause | Fix | +|---|---|---| +| `Private key required to decrypt state` at startup | A `.noorm/state/state.enc` is in the image | Keep `state.enc` out of the image | +| `Lock for 'prod' held by … since …, expires …` | Another `change ff` is running. The lock is per change and does not wait | Rerun after it finishes | +| Two deploys run `change ff` at once and neither fails | The lock holder is the identity's `name <email>`. A second job with the same name and email extends the lock | Serialize deploys in your CI, for example with a `concurrency` group | +| Two contexts for two databases both connect to the last one | `createContext()` reads `process.env` after awaiting state and settings, so concurrent calls see the last assignment | One process per database, or create each context in turn at startup | +| `noorm run build` in the deploy step errors or changes nothing | A changed `CREATE TABLE` file re-runs and cannot alter a table that holds data | Run `noorm change ff` only | -Single-database services — the overwhelming majority — are unaffected. +## Next -## What's Next? -- [CI/CD Automation](/guide/automation/ci) — pipelines, identity enrollment, vault-backed secrets -- [Configs](/guide/environments/configs) — access roles and the full environment variable table -- [SDK Reference](/reference/sdk) — the context API in full +- [CI/CD Automation](/guide/automation/ci): pipelines, identity enrollment, vault secrets +- [SDK reference](/reference/sdk): the context API diff --git a/docs/guide/environments/configs.md b/docs/guide/environments/configs.md index c0e05e7b..8ef629a8 100644 --- a/docs/guide/environments/configs.md +++ b/docs/guide/environments/configs.md @@ -1,383 +1,219 @@ # Configs -You need to connect to different databases. Your local dev database uses SQLite. Staging runs PostgreSQL on a remote server. Production is locked down tight. noorm handles all of this with configs. +Switch between your dev, staging, and prod databases, and decide what a person or an AI agent may do on each. -A **config** is a saved database connection with all its details: dialect, host, credentials, paths. You create configs once, then switch between them as needed. - - -## Why Multiple Configs? - -Real projects talk to multiple databases: - -- `dev` - Your local machine, fast iteration -- `test` - Gets wiped between test runs -- `staging` - Mirrors production, catches issues early -- `prod` - The real thing, locked down by access roles - -Without saved configs, you'd type connection details every time. With them, you switch environments in one keystroke. - - -## Creating a Config (TUI) - -Launch the TUI and navigate to configs: - -```bash -noorm -``` - -Press `c` to open the config menu, then `a` to add a new config. - -The setup wizard walks you through each field: - -1. **Config Name** - A short identifier like `dev`, `staging`, or `prod`. Use lowercase with hyphens. - -2. **Database Type** - Your database engine: PostgreSQL, MySQL, SQLite, or SQL Server. - -3. **Host** - The server address. For local databases, this is usually `localhost`. SQLite doesn't need a host. - -4. **Port** - The port number. noorm fills in the default for your dialect (PostgreSQL: 5432, MySQL: 3306, MSSQL: 1433). - -5. **Database** - The database name, or file path for SQLite. - -6. **Username** and **Password** - Your credentials. Passwords are stored encrypted. - -7. **User Role** and **Agent Role** - Who may do what through this config. See [Access Roles](#access-roles) below. - -8. **Test Database** - Marks the config as a test target, which build rules and the SDK's `requireTest` guard read. - -Before saving, noorm connects to the server to check the credentials. It uses the dialect's system database, so the target database does not have to exist yet. Schema and change directories are not part of a config: they come from `paths` in `settings.yml`. - - -## Creating a Config Non-Interactively - -`noorm config add` and `edit` do not run headlessly. They print the interactive-only message and exit non-zero, because credential entry and the connection test belong in the wizard. Two commands are the exception: `noorm config rm <name> --yes` deletes headlessly (see the [CLI Reference](/headless#config-rm-name)), and `noorm config import <file>` creates or replaces a config from JSON. For CI/CD pipelines and scripts that need to skip the config wizard entirely, build the config from **environment variables** instead: - -```bash -export NOORM_CONNECTION_DIALECT=postgres -export NOORM_CONNECTION_HOST=db.staging.example.com -export NOORM_CONNECTION_PORT=5432 -export NOORM_CONNECTION_DATABASE=myapp_staging -export NOORM_CONNECTION_USER=deploy -export NOORM_CONNECTION_PASSWORD="$DB_PASSWORD" - -noorm run build # Uses the env-only config -noorm change ff # Same — no stored config required -``` - -When at least `NOORM_CONNECTION_DIALECT` and `NOORM_CONNECTION_DATABASE` are set, noorm runs without ever touching `.noorm/state/state.enc`. See the [CLI Reference](/headless#environment-only-mode) for the full list of supported variables. - - -## Switching Configs - -Only one config is active at a time. All commands use the active config unless you specify otherwise. - -**In the TUI:** Press `c` for configs, highlight one in the list, then press `Enter` to activate it. - -**From the command line:** +A config is a saved database connection. This project has three: ```bash noorm config use staging +noorm config list ``` -**Per-command override:** - -```bash -noorm -c prod run build -``` - -`noorm config list` marks the active one and tags anything whose access is no longer the default: - ``` +Active config set to: staging Configurations: - dev (active) — sqlite/./data/dev.db - staging — postgres/staging_db - prod — postgres/prod_db [user:operator agent:off] + dev — postgres/order_to_cash_dev + staging (active) — postgres/order_to_cash_staging + prod — postgres/order_to_cash [user:operator agent:off] ``` +The bracket tag marks a test config, or roles that differ from the default (`user:admin agent:viewer`). `agent:off` hides the config from AI agents. -## Config Properties - -Every config has these fields: - -| Property | Required | Description | -|----------|----------|-------------| -| `name` | Yes | Unique identifier (alphanumeric, hyphens, underscores) | -| `type` | Yes | `local` or `remote` | -| `dialect` | Yes | `postgres`, `mysql`, `sqlite`, or `mssql` | -| `host` | No* | Server address (*required for non-SQLite) | -| `port` | No | Port number (defaults by dialect) | -| `database` | Yes | Database name or file path | -| `user` | No | Authentication username | -| `password` | No | Authentication password (stored encrypted) | -| `access.user` | No | CLI/TUI/SDK role: `viewer`, `operator`, or `admin` (default: `admin`) | -| `access.agent` | No | AI agent role, over MCP and the CLI alike: `viewer`, `operator`, `admin`, or `false` to hide the config from agents (default: `viewer`) | -| `isTest` | No | Marks as test database (default: false) | -| `ssl` | No | SSL/TLS configuration | -| `pool` | No | Connection pool settings | -| `identity` | No | Override the audit identity | - -Connection details nest under a `connection` object in the stored JSON, so `host` is `connection.host` and `ssl` is `connection.ssl`. `name`, `type`, `isTest`, `access`, and `identity` sit at the top level. - -**Default ports by dialect:** - -| Dialect | Default Port | -|---------|--------------| -| PostgreSQL | 5432 | -| MySQL | 3306 | -| MSSQL | 1433 | -| SQLite | N/A | - - -## Access Roles - -Production databases need safeguards. Every config carries an access role per **channel** — who's *driving*. `noorm ui` → Config → Edit sets `access.user` (a human) and `access.agent` (an AI agent) independently, so a config can be wide open to you at the terminal while showing an agent only read access. - -The channel is the caller, not the transport. An agent reaches noorm over MCP or by running `noorm` on the command line, and both get the `agent` role — the CLI recognizes the agent harness it was spawned from, so an agent that is refused over MCP cannot route around it by shelling out. -| Role | Behavior | -|------|----------| -| `viewer` | Read-only: explore schema, run `SELECT`/`EXPLAIN`. Cannot read secrets | -| `operator` | Reads plus writes; migrations, builds, and `db create`/`db reset` require typing a confirmation phrase; raw DDL, `db drop`, and `db teardown` are denied | -| `admin` | Everything is permitted, but the operations that cannot be walked back still prompt | +## How do I create a config? -| Operation | viewer | operator | admin | -|-----------|--------|----------|-------| -| `db explore`, `SELECT` / `EXPLAIN` | allowed | allowed | allowed | -| Raw `INSERT` / `UPDATE` / `DELETE` | denied | allowed | allowed | -| Raw DDL (`CREATE` / `ALTER` / `DROP`) | denied | denied | allowed | -| `change run` / `change ff` / `change revert` | denied | confirm | allowed | -| `run build` / `run file` / `run dir` | denied | confirm | allowed | -| `db create` / `db reset` | denied | confirm | allowed | -| `db truncate` | denied | confirm | confirm | -| `db drop` / `db teardown` | denied | denied | confirm | -| `change rm`, `config rm`, overwriting a config | denied | confirm | confirm | -| Reading vault or local secrets | denied | allowed | allowed | -| Writing vault or local secrets | denied | confirm | allowed | -| `vault propagate` | denied | confirm | confirm | -`admin` is not a bypass. Dropping a database, tearing down a schema, truncating tables, removing a change or a config, overwriting a config's access roles, and propagating the vault key all confirm at every role that is allowed to attempt them. The full matrix lives in `src/core/policy/matrix.ts`. +Use the TUI wizard: run `noorm ui`, press `c` for configs, then `a`. Before it saves, the wizard connects to the server's system database, so the target database does not have to exist yet. MySQL has no system database to fall back to, so on MySQL [create the database first](/guide/database/create#what-bites). -When an operation needs confirmation in the TUI, noorm asks you to type a phrase (`yes-<config-name>`). This catches the "oops, wrong database" moment before damage happens. +| Field | Notes | +|---|---| +| Config Name | Letters, numbers, hyphens, underscores. A name that matches a [stage](/guide/environments/stages) links the config to it | +| Database Type | `postgres`, `mysql`, `sqlite`, `mssql` | +| Host, Port | SQLite has no host. Port defaults to 5432, 3306, or 1433 | +| Database | Database name, or file path for SQLite | +| Username, Password | Stored encrypted in `.noorm/state/state.enc` | +| User Role, Agent Role | See [Access roles](#access-roles) | +| Test Database | Sets `isTest`, read by `rules` in `settings.yml` and the SDK's `requireTest` | -**Skipping confirmations in CI:** +Schema and change folders are not config fields. `paths` in `.noorm/settings.yml` sets them. -Automated pipelines can't type confirmations. Set `NOORM_YES=1` to skip them on the `user` channel: +`noorm config add` and `config edit` exit 1 and point you at the TUI. To create a config without it, write a JSON file and import it. This `staging.json` is hidden from agents: -```bash -export NOORM_YES=1 -noorm -c prod change run +```json +{ + "name": "staging", + "access": { "user": "operator", "agent": false }, + "connection": { + "dialect": "postgres", + "host": "staging.db.internal", + "database": "order_to_cash_staging", + "user": "deploy" + } +} ``` -There is no equivalent skip on the `agent` channel — an agent hitting a `confirm` cell is always denied, and `NOORM_YES` / `--yes` do not change that. Otherwise the confirmation would be one flag away from meaningless. See [MCP](/guide/automation/mcp#access-roles) for the agent-facing side of this. - -**Which channel am I on?** - -noorm resolves the channel from the environment it was started in, in this order: - -1. `NOORM_CHANNEL=user` or `NOORM_CHANNEL=agent`, if set. -2. The variables agent harnesses export for their child processes — Claude Code, OpenAI Codex, Cursor, Gemini CLI, or the generic `AI_AGENT` / `NOORM_AGENT`. Any of those means `agent`. -3. Otherwise `user`. - -`TERM_PROGRAM`, `CI`, and whether stdout is a TTY are deliberately ignored: they describe the terminal or the pipeline, not the caller, and keying on them would lock a human out of their own CLI. - -`noorm mcp serve` sits above this chain. It declares `agent` outright, so stdio traffic stays on the agent channel even under `NOORM_CHANNEL=user`. - -Set `NOORM_CHANNEL=user` when *you* are scripting from inside an agent session and want your own role: - ```bash -NOORM_CHANNEL=user noorm -c prod change run +noorm config import ./staging.json ``` -::: warning Access Roles Are Not Security -Roles prevent accidents, not attacks. They won't stop a determined user or malicious script — an agent can set `NOORM_CHANNEL` too. The channel defends against an agent routing around a refusal, which is the realistic case, not one setting out to evade the check. Use proper database permissions for real security. -::: +``` +Imported config 'staging'. +``` -## Environment Variable Overrides +## How do I switch configs? -Environment variables override stored config values. This is how you inject secrets in CI/CD without storing them. -**Connection variables:** +Commands use the first of these that is set: -| Variable | Config Path | -|----------|-------------| -| `NOORM_CONNECTION_DIALECT` | `connection.dialect` | -| `NOORM_CONNECTION_HOST` | `connection.host` | -| `NOORM_CONNECTION_PORT` | `connection.port` | -| `NOORM_CONNECTION_DATABASE` | `connection.database` | -| `NOORM_CONNECTION_USER` | `connection.user` | -| `NOORM_CONNECTION_PASSWORD` | `connection.password` | -| `NOORM_CONNECTION_SSL` | `connection.ssl` | +| Scope | How | +|---|---| +| One command | `noorm run build --config prod`, with the flag after the subcommand | +| One shell | `export NOORM_CONFIG=prod` | +| Every command | `noorm config use staging`, or highlight a config in the TUI and press Enter | -**Path variables.** These two override `settings.yml`, not the config, because schema and change directories are project-wide: -| Variable | Settings Path | -|----------|---------------| -| `NOORM_PATHS_SQL` | `paths.sql` | -| `NOORM_PATHS_CHANGES` | `paths.changes` | +## How do I override a config in CI? -The overlay stays out of the file itself. noorm never writes an ambient `NOORM_*` value back into `settings.yml`, which is version controlled. -**Behavior variables:** +Environment variables override the stored config. [Stages](/guide/environments/stages#which-value-wins) shows the full order. -| Variable | Purpose | -|----------|---------| -| `NOORM_CONFIG` | Which config to use | -| `NOORM_YES` | Skip confirmations (set to `1`) | -| `NOORM_CHANNEL` | Force the policy channel: `user` or `agent` (default: detected from the environment) | +| Variable | Sets | +|---|---| +| `NOORM_CONNECTION_DIALECT`, `_HOST`, `_PORT`, `_DATABASE`, `_USER`, `_PASSWORD`, `_SSL` | `connection.*` on the config | +| `NOORM_PATHS_SQL`, `NOORM_PATHS_CHANGES` | `paths.sql`, `paths.changes`. Never written back to `settings.yml` | +| `NOORM_CONFIG` | Which stored config to use | +| `NOORM_YES` | Any value except empty, `0`, or `false` skips `confirm` prompts on the `user` channel | +| `NOORM_CHANNEL` | `user` or `agent`. See [Which channel is a process on?](#which-channel-is-a-process-on) | -**Example: Override host for CI runner** - -```bash -# Use stored 'staging' config but connect to CI database server -export NOORM_CONFIG=staging -export NOORM_CONNECTION_HOST=db.ci-runner.internal -noorm run build -``` - -**Example: Build config entirely from environment** - -When you provide at least `NOORM_CONNECTION_DIALECT` and `NOORM_CONNECTION_DATABASE`, noorm creates a temporary config from environment variables alone: +When no config is selected by flag, `NOORM_CONFIG`, or `config use`, noorm builds one named `__env__` from the variables. It needs `NOORM_CONNECTION_DIALECT` and `NOORM_CONNECTION_DATABASE`: ```bash export NOORM_CONNECTION_DIALECT=postgres -export NOORM_CONNECTION_HOST=localhost -export NOORM_CONNECTION_DATABASE=ci_test -export NOORM_CONNECTION_USER=postgres +export NOORM_CONNECTION_HOST=db.ci.internal +export NOORM_CONNECTION_DATABASE=order_to_cash_ci export NOORM_CONNECTION_PASSWORD="$DB_PASSWORD" -noorm run build # Uses env-only config +noorm run build ``` +[CI/CD](/guide/automation/ci) covers the full pipeline. -## Validating Connections -Before running changes, verify your config connects successfully. In the TUI, highlight a config, press `+` for More, then press `v` to run a full validation: +## Access roles -1. Can noorm connect with the provided credentials? -2. Does the target database exist? -3. Are the sql/changes paths accessible? -The TUI also offers a **server-only** mode that skips the database existence check by connecting to the dialect's system database (PostgreSQL uses `postgres`, MSSQL uses `master`). Use it during initial setup, before the target database is created. +Each config has two roles, one per channel. The `user` channel is you, in the CLI, TUI, or SDK. The `agent` channel is an AI agent, over MCP or the CLI. Change them in the TUI (config list, `e`). -For headless smoke checks, run any command that opens a connection — e.g. `noorm info` or `noorm db explore` — and inspect the exit code or `--json` output. +| Role | Meaning | +|---|---| +| `viewer` | Reads only, no secrets. Default for `agent` | +| `operator` | Writes data. Builds, changes, and `db create` ask first | +| `admin` | Everything allowed, but irreversible operations still ask. Default for `user` | +| `false` | `agent` only. Hides the config: `config list` omits it and MCP answers as if it does not exist | +Every permission has one cell per role, fixed in `src/core/policy/matrix.ts`: -## Exporting and Importing Configs +| Permission | viewer | operator | admin | +|---|---|---|---| +| `explore`, `sql:read` (`SELECT`, `EXPLAIN`) | allow | allow | allow | +| `sql:write` (`INSERT`, `UPDATE`, `DELETE`) | deny | allow | allow | +| `sql:ddl` (`CREATE`, `ALTER`, `DROP`) | deny | deny | allow | +| `run:build`, `run:file`, `run:dir`, `change:run`, `change:ff`, `change:revert`, `db:create`, `db:reset` | deny | confirm | allow | +| `db:truncate`, `change:rm`, `config:rm`, `config:write`, `vault:propagate`, `lock:force` | deny | confirm | confirm | +| `db:destroy` (`db drop`), `db:teardown` | deny | deny | confirm | +| `vault:read`, `secret:read`, `transfer:plan` | deny | allow | allow | +| `vault:write`, `secret:write` | deny | confirm | allow | +| `debug:read` | allow | allow | allow | +| `debug:write` | deny | deny | confirm | -There are two export paths, and they produce different files. +A `confirm` cell never opens for an agent: -**TUI, for sharing with a teammate:** `noorm ui` → Config → `+` for More → `x` exports the highlighted config, `i` imports one. This path encrypts the config and its local secrets for a single recipient, using an ephemeral X25519 key exchange and AES-256-GCM, and writes `<name>.noorm.enc`. It leaves `user` and `password` out on purpose, so the recipient connects with their own credentials. You need the recipient's identity locally before you can export to them, and only their private key can decrypt the file. On import, noorm decrypts with your private key and prompts for your database credentials. +```mermaid +flowchart TD + A["noorm run build"] --> B{"role on this<br/>channel"} + B -->|false| H["config hidden"] + B -->|a role| C{"matrix cell"} + C -->|allow| R["runs"] + C -->|deny| D["refused"] + C -->|confirm| E{"channel"} + E -->|user| F["type yes-prod<br/>or pass --yes"] + E -->|agent| D +``` -**CLI, for backup and cross-machine transfer:** `noorm config export` writes plain JSON, and that JSON carries the connection password. Treat the file as a credential. noorm writes it with `0600` permissions and requires a role that can read secrets, so a `viewer` is refused. +`--yes` and `NOORM_YES=1` satisfy a `confirm` cell. They never open a `deny` cell. -```bash -# Export to stdout or file -noorm config export dev -noorm config export dev --output ./dev-config.json -# Import from file -noorm config import ./dev-config.json -noorm config import ./staging-config.json --force --yes # overwrite existing -``` +### Which channel is a process on? -The exported JSON looks like this: -```json -{ - "name": "dev", - "type": "local", - "isTest": false, - "access": { - "user": "admin", - "agent": "viewer" - }, - "connection": { - "dialect": "sqlite", - "database": "./data/dev.db" - } -} -``` +A process takes the channel from its environment, not from which binary it called: -Overwriting an existing config rewrites its access roles, so `--force` on its own is not enough: `config import` also wants `--yes` (or `NOORM_YES`) to confirm. When you import a file that has no credentials in it, set the password afterward through the TUI's edit screen or with `NOORM_CONNECTION_PASSWORD` at command time. +```mermaid +flowchart TD + S["noorm starts"] --> M{"noorm mcp serve?"} + M -->|yes| AG["agent"] + M -->|no| E{"NOORM_CHANNEL is<br/>user or agent?"} + E -->|yes| SET["that value"] + E -->|no| H{"harness variable<br/>set and non-empty?"} + H -->|yes| AG + H -->|no| US["user"] +``` +The harness variables are `CLAUDECODE`, `CLAUDE_CODE`, `CLAUDE_CODE_ENTRYPOINT`, `CODEX_SANDBOX`, `CURSOR_AGENT`, `GEMINI_CLI`, `AI_AGENT`, and `NOORM_AGENT`. `TERM_PROGRAM`, `CI`, and whether stdout is a terminal are ignored, because they describe the terminal, not the caller. -## Managing Configs from the CLI +::: warning Roles prevent accidents +Roles stop a wrong-database teardown and an agent that shells out after an MCP refusal. They do not stop someone with the credentials, and an agent can set `NOORM_CHANNEL` too. Use database permissions for security. +::: -noorm also has headless commands for scripting and CI: -```bash -# List all configs -noorm config list -noorm config list --json +## How do I check a config? -# Copy a config -noorm config cp dev staging -# Validate connection +```bash noorm config validate prod - -# Export and import (see above) -noorm config export dev --output ./backup.json -noorm config import ./backup.json ``` -`config validate` connects to the database and exits 0 on success, 1 on failure. Use it as a CI health check before running migrations. - - -## Common Workflows - - -### Setting Up a New Developer +``` +prod: VALID + [+] Connection: Connection successful + [+] Name: prod + [+] Database: order_to_cash + [+] Host: db.example.com +``` -1. Clone the repository -2. Run `noorm ui` and import the shared config under Config → `+` → `i` -3. Edit the imported config and set the password (Config → highlight → `e`) -4. Back on the config list, press `+` then `v` to validate the connection +Exit 0 when valid, 1 when not. Every check runs, so one report lists all failures. In the TUI, highlight a config, press `+`, then `v`. -### CI/CD Pipeline +## How do I share a config with a teammate? -```yaml -# GitHub Actions example -env: - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} - NOORM_CONNECTION_DATABASE: ${{ secrets.DB_NAME }} - NOORM_CONNECTION_USER: ${{ secrets.DB_USER }} - NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_PASSWORD }} - NOORM_YES: 1 -steps: - - uses: actions/checkout@v4 - - run: npm ci - - run: noorm run build - - run: noorm change run -``` +| | TUI export (`+`, then `x`) | `noorm config export` | +|---|---|---| +| File | `<name>.noorm.enc`, encrypted for one recipient | JSON to stdout, or `--output <file>` | +| Contents | The config and its local secrets, without `user` and `password` | The config, with `password` | +| Needs | The recipient's identity known locally | A role that can read secrets | +| Import | TUI: `+`, then `i` | `noorm config import <file>` | +`config export --output` writes the file with `0600` permissions. Treat it as a credential. Importing over an existing name needs `--force --yes`, because an overwrite rewrites the config's roles. -### Switching Between Dev and Staging -```bash -# Morning: work on local -noorm config use dev -noorm run build +## What bites -# Afternoon: deploy to staging -noorm config use staging -noorm change run -``` -Or use per-command overrides without switching: +| Symptom | Cause | Fix | +|---|---|---| +| `noorm -c prod run build` fails: `--cwd path is not a directory` | Before the subcommand, `-c` is `--cwd` | `noorm run build --config prod` | +| `"db:teardown" is not allowed on config "prod" (role: operator).` | The role's cell is `deny`, and no flag opens it | Change the role in the TUI, or use a config with `admin` | +| `"run:build" on config "prod" requires confirmation from a human.` | A `confirm` cell is `deny` on the `agent` channel | Run it in your own terminal, or with `NOORM_CHANNEL=user` when you script inside an agent session | +| An imported config is open to `admin` | The file has no `access`, so it gets `user: admin`, `agent: viewer` | Put `access` in the file | -```bash -noorm -c staging change run -``` +## Next -## What's Next? -- [Stages](/guide/environments/stages) - Config templates for teams -- [Secrets](/guide/environments/secrets) - Managing sensitive values -- [CLI Reference](/headless) - Environment variables and command surface for CI/CD +- [Stages](/guide/environments/stages): defaults and required secrets shared by a team +- [Secrets](/guide/environments/secrets): keep sensitive values out of SQL +- [Headless reference](/headless): every `config` flag and exit code +- [MCP](/guide/automation/mcp): the agent side of access roles diff --git a/docs/guide/environments/secrets.md b/docs/guide/environments/secrets.md index a5293548..b057ea24 100644 --- a/docs/guide/environments/secrets.md +++ b/docs/guide/environments/secrets.md @@ -1,202 +1,93 @@ # Secrets -Sensitive values like database passwords and API keys have no place in config files. noorm provides encrypted secret storage that travels with your project, unlocks with your identity, and injects into templates at runtime. +Keep API keys and other sensitive values out of committed SQL, and still use them when a template runs. - -## What Secrets Are - -Secrets are encrypted key-value pairs stored in your local state file (`.noorm/state/state.enc`). They never touch disk in plain text. - -Secrets are for sensitive values that get **inserted into your database** via SQL templates—not for database connection credentials. Common uses: - -- API keys stored in configuration tables -- Encryption keys for sensitive data columns -- Service credentials (AWS, Stripe, SendGrid) -- OAuth client secrets - -Secrets integrate directly with SQL templates, so you can reference them without exposing values in your source code or version control. - - -## Setting a Secret - -You can manage local secrets through the TUI (masked input, nothing in shell history) or the CLI. - -**TUI:** Launch `noorm ui`, press `c` for configs, highlight a config, and press `k` for its secrets. The list groups required secrets (declared in `settings.yml`) above optional ones and shows which are set, which are missing, and their declared types. Values stay hidden. Press `a` to add one; the form masks `password` and `api_key` input, validates `connection_string` values as URIs, and writes the encrypted result to `.noorm/state/state.enc`. - -Settings → Secrets is a different screen. It edits the secret *declarations* in `settings.yml` (key, type, description, required), not the encrypted values. - -**CLI:** The `noorm secret` namespace manages config-scoped local secrets headlessly: - -```bash -# Set a secret -noorm secret set API_KEY "sk-live-abc123" -noorm secret set DB_PASSWORD "secret" --config prod - -# List secret keys (values never shown) -noorm secret list -noorm secret list --config staging --json - -# Remove a secret -noorm secret rm OLD_KEY --yes -``` - -::: tip Vault vs Local Secrets -**Local secrets** (`noorm secret`) are per-user, per-machine. They override vault values and stay on your disk. - -**Vault secrets** (`noorm vault`) are team-shared and stored encrypted in the database. Anyone with vault access can read them. - -In CI/CD pipelines, push secrets with the vault (`noorm vault set KEY "value"`) or batch-load via `noorm ci secrets --file`. -::: - - -## Config-Scoped vs Global Secrets - -Secrets come in two scopes: - -| Scope | Use Case | Example | -|-------|----------|---------| -| Config-scoped | Per-environment credentials | Database password for `prod` | -| Global | Shared across all configs | Third-party API key | - -**Config-scoped secrets** belong to a single config. When you delete the config, its secrets go with it. - -**Global secrets** persist independently and are available everywhere. - -In templates, access them differently: +The template names the secret. The value never appears in the file: ```sql --- Config-scoped secret -CREATE USER readonly WITH PASSWORD '{%~ $.secrets.DB_PASSWORD %}'; - --- Global secret --- Using API key: {%~ $.globalSecrets.SHARED_API_KEY %} +-- db/sql/09_seeds/02_gateway.sql.tmpl +INSERT INTO app_setting (name, value) +VALUES ('payment_gateway_key', {%~ $.quote($.secrets.PAYMENT_GATEWAY_KEY) %}); ``` -See [Templates](/guide/sql-files/templates) for full syntax reference. +Set the value once per config. `--stdin` keeps it out of shell history and the process table: - -## Using Secrets in Templates - -Secrets inject into SQL templates via the `$` context object: - -```sql --- sql/users/create-readonly.sql.tmpl -CREATE USER {%~ $.secrets.READONLY_USER %} -WITH PASSWORD '{%~ $.secrets.READONLY_PASSWORD %}'; - -GRANT SELECT ON ALL TABLES TO {%~ $.secrets.READONLY_USER %}; +```bash +printf %s "$KEY" | noorm secret set PAYMENT_GATEWAY_KEY --stdin --config prod +noorm secret list --config prod ``` -If a template references a missing secret, it fails at runtime. Define your required secrets upfront to catch this early. - - -## Secret Requirements in Settings - -You can require secrets at two levels: **universally** (all configs) or **per-stage** (specific environments). - - -### Universal Secrets - -Define in the root `secrets` section of `settings.yml`: - -```yaml -# .noorm/settings.yml -secrets: - - key: ENCRYPTION_KEY - type: password - description: App-wide encryption key ``` - -Every config requires these secrets, regardless of stage. - - -### Stage Secrets - -Define within a stage to require secrets only for matching configs: - -```yaml -# .noorm/settings.yml -stages: - prod: - description: Production database - secrets: - - key: DB_PASSWORD - type: password - description: Main database password - required: true - - staging: - secrets: - - key: DB_PASSWORD - type: password - - key: DEBUG_KEY - type: string +Secret "PAYMENT_GATEWAY_KEY" set for config "prod". +Secrets for "prod" (1): + PAYMENT_GATEWAY_KEY ``` -A config named `prod` requires its stage secrets plus universal secrets. A config named `dev` requires only universal secrets (plus any dev-stage secrets). - - -### Secret Types +Secrets are for values a template writes into your database. Database connection credentials belong to the [config](/guide/environments/configs). -The `type` field controls input behavior: -| Type | Input Behavior | -|------|----------------| -| `string` | Plain text | -| `password` | Masked, no echo | -| `api_key` | Masked, no echo | -| `connection_string` | Plain text, URI validation | +## Which value does a template get? -Types are hints for the CLI. All secrets are stored identically (encrypted strings). +The first tier that holds the key wins, so a local value overrides the team's: -## Secrets in CI/CD - -The local secret store exists for per-developer overrides — `noorm secret set` is headless, but it writes to `.noorm/state/state.enc` on disk, which a fresh CI runner doesn't have. For non-interactive environments, push values to the [vault](/guide/environments/vault) instead, which is scriptable end-to-end and reachable from a clean checkout: - -```bash -# Write a secret to the vault -noorm vault set DB_PASSWORD "$DB_PASSWORD" +```mermaid +flowchart TD + T["$.secrets.<br/>PAYMENT_GATEWAY_KEY"] --> A{"set on this config?"} + A -->|yes| V["use it"] + A -->|no| B{"global secret?"} + B -->|yes| V + B -->|no| C{"in the vault?"} + C -->|yes| V + C -->|no| X["throw: not found"] +``` -# Pipe a secret to keep it out of the process table and shell history -echo "$DB_PASSWORD" | noorm vault set DB_PASSWORD --stdin +| Tier | Stored in | Set with | Seen by | +|---|---|---|---| +| Config-local | `.noorm/state/state.enc`, per config | `noorm secret set`, or the TUI | You | +| Global-local | `state.enc`, all your configs | No CLI or TUI command writes one | You | +| Vault | The database, encrypted | `noorm vault set` | The team | -# List vault secrets as JSON -noorm vault list --json +`$.secrets` merges all three tiers. `$.globalSecrets` holds only the global tier. A missing key throws `Secret "PAYMENT_GATEWAY_KEY" not found (searched: config-local, global-local, vault)` instead of rendering `undefined`. -# Remove a vault secret -noorm vault rm OLD_API_KEY -``` -Vault secrets sit below local secrets in the [resolution hierarchy](/guide/environments/vault#secret-resolution-hierarchy), so a developer can still override a vault value locally through `noorm ui` without affecting anyone else. +## How do I set a secret? -## Security Notes +| | CLI | TUI | +|---|---|---| +| Set | `noorm secret set <KEY> [value] [--stdin] [--config <name>]` | `noorm ui`, `c`, highlight a config, `k`, then `a` | +| List keys | `noorm secret list`, which never prints values | The same screen. Values show masked, with their first characters visible | +| Remove | `noorm secret rm <KEY> --yes` | The same screen, `d` | -noorm takes several measures to protect your secrets: +Keys start with a letter and contain only letters, numbers, and underscores. The TUI masks `password` and `api_key` input and checks that a `connection_string` parses as a URI. A `viewer` role cannot read or write secrets, and an `operator` needs `--yes` to write one. -1. **Encryption at rest** - Secrets are stored in `.noorm/state/state.enc` using AES-256-GCM encryption -2. **Key derivation** - The encryption key derives from your private identity key via HKDF -3. **Values never displayed** - The CLI shows secret keys only, never values -4. **Masked input** - Password-type secrets use non-echoing input -5. **No logging** - Secret values are never emitted to observer events -6. **Automatic redaction** - The logger masks secret fields if they appear in event data +[Stages](/guide/environments/stages#how-do-i-require-a-secret) declares which secrets a config must have. The TUI lists those first, marked set or missing. -::: warning Don't Commit state.enc -Add `.noorm/state/state.enc` to your `.gitignore`. Its encryption key is derived from your private identity key in `~/.noorm/identity.key`, so nobody else can read the file, and neither can you on a machine that does not have that key. +::: warning The rendered SQL holds the plain value +The database receives the secret as text, and so does anything that logs the rendered SQL. Render a secret only into the column that needs it. ::: -## Deleting Secrets +## What bites + -Remove a local secret through the TUI: Config → highlight the config → `k` → highlight the entry and press `d`. Headlessly, `noorm secret rm <KEY> --yes`. For team-shared values stored in the vault, use `noorm vault rm <KEY>`. +| Symptom | Cause | Fix | +|---|---|---| +| `Secret "PAYMENT_GATEWAY_KEY" not found (searched: ...)` | No tier has the key. The vault tier is skipped without vault access | Set it locally, or ask a teammate to run `noorm vault propagate` | +| `Failed to set secret: Secret key "payment-gateway-key" is invalid` | A hyphen or leading digit in the key | Use `PAYMENT_GATEWAY_KEY` | +| A secret set locally is missing in CI | A CI runner has no `state.enc` | Put it in the [vault](/guide/environments/vault), or run `noorm ci secrets --file <env file>` after `noorm ci init` | +| `secret set` says it requires confirmation | The config's role for your channel is `operator` | Pass `--yes` | +| The TUI refuses to delete a secret | The stage requires it, and no vault copy exists | Remove the requirement from `settings.yml`. `noorm secret rm` does not check | +| A teammate's build fails where yours passes | Your local value hides a missing vault value | Run `noorm vault list`, then `noorm vault set` | +| Someone committed `state.enc` | It holds your credentials and secrets. `noorm init` ignores `.noorm/state/`, so the ignore line was removed or init never ran | Restore `.noorm/state/` in `.gitignore` and remove the file from the repo. Your identity key encrypts it, so another machine cannot read it | -A required secret (one declared in `settings.yml`) cannot be deleted outright, only updated. The exception is a required secret that also exists in the vault: deleting the local copy is allowed there, because the vault value takes over. To retire the requirement itself, edit `settings.yml`. +## Next -## What's Next? -- [Stages](/guide/environments/stages) - Environment templates with secret requirements -- [Templates](/guide/sql-files/templates) - Using secrets in dynamic SQL -- [Configs](/guide/environments/configs) - Config-scoped vs global secrets +- [Vault](/guide/environments/vault): share a secret with the team +- [Stages](/guide/environments/stages): require a secret per environment +- [Templates](/guide/sql-files/templates): the full `$` context +- [Headless reference](/headless): every `secret` and `vault` flag diff --git a/docs/guide/environments/stages.md b/docs/guide/environments/stages.md index f2d4b618..f8f1c69f 100644 --- a/docs/guide/environments/stages.md +++ b/docs/guide/environments/stages.md @@ -1,252 +1,127 @@ # Stages -## The Problem +Give every developer the same starting configs, the same access ceiling, and the same list of required secrets for dev, staging, and prod. -Your team has three environments: development, staging, and production. Each has different requirements: - -- **Production** should be protected from accidental teardowns -- **Staging** might require specific API keys for integration testing -- **Development** should default to localhost - -Without a shared definition, every developer configures these differently. Someone creates an unprotected production config. Someone else forgets the staging API key. The inconsistency causes problems down the line. - -Stages solve this. They're templates defined in your project's `settings.yml` that every team member shares. - - -## What Stages Are - -A **stage** is a named template for database configs. A config whose name matches a stage inherits that stage's defaults and requirements. - -Think of stages as blueprints: - -- `dev` stage: localhost defaults, no protection needed -- `staging` stage: requires integration API keys -- `prod` stage: always protected, cannot be deleted - -Your project defines these once in `settings.yml`. Every developer on the team gets the same blueprints. - - -## Defining Stages - -Stages live in `.noorm/settings.yml` under the `stages` key: +A stage is a template in `.noorm/settings.yml`. It is committed, so the team shares it: ```yaml # .noorm/settings.yml stages: dev: - description: Local development database + description: Local order-to-cash database defaults: dialect: postgres host: localhost port: 5432 - staging: - description: Staging environment - defaults: - dialect: postgres - protected: false - secrets: - - key: STRIPE_TEST_KEY - type: api_key - description: Stripe test API key for payment testing - required: true - prod: - description: Production database + description: Production locked: true defaults: dialect: postgres protected: true + secrets: + - key: PAYMENT_GATEWAY_KEY + type: api_key + description: Key the payment sync sends to the gateway + required: true ``` -A config named `staging` then automatically: -- Defaults to the postgres dialect -- Keeps its own access roles, with no ceiling applied (staging data is expendable) -- Declares `STRIPE_TEST_KEY` as required, so the TUI lists it as missing until it is set and any template that reads it fails until then +A [config](/guide/environments/configs) uses the stage with its name. The stage is not stored on the config, so it cannot differ between developers. A config named `prod`: +- is created by the TUI from `defaults` if it does not exist yet +- cannot be deleted, because the stage is `locked` +- resolves to at most `operator` for you and `viewer` for agents, because the stage is `protected` +- shows `PAYMENT_GATEWAY_KEY` as missing in the TUI until someone sets it -## Stage Properties +From the SDK, `createContext({ stage: 'prod' })` picks a stage for a config with another name. -Each stage can define these properties: -| Property | Type | Description | -|----------|------|-------------| -| `description` | string | Human-readable description shown in the TUI | -| `locked` | boolean | If true, configs cannot be deleted (default: false) | -| `defaults` | object | Values applied when creating a config from this stage | -| `secrets` | array | Secrets that must be configured | +## Where do `defaults` apply? -### Defaults +`defaults` is a template for the first config, not a layer under every config. When `noorm ui` opens, it creates a config for each stage that sets `defaults.dialect` and has no config of its name yet, provided this machine has an identity. `host` falls back to `localhost`, `database` to the stage name, and a `protected` stage starts the config at `operator` for you and `viewer` for agents. -Defaults provide initial values when creating a config. Most can be overridden by the developer in the TUI: +From then on, the stored config holds the values. Editing `defaults` later does not change it. A stage selected for an [environment-only](/headless#environment-only-mode) run also passes its `isTest`. -```yaml -stages: - dev: - defaults: - dialect: postgres - host: localhost - port: 5432 - database: myapp_dev -``` -Some defaults are **enforced** and cannot be overridden in the TUI: +## Which value wins? -| Default | Behavior | -|---------|----------| -| `protected: true` | Acts as an access ceiling: whatever `access` the config or the developer sets, the *resolved* access is clamped to at most `{ user: 'operator', agent: 'viewer' }` | -| `dialect` | Cannot be changed after config creation. The TUI edit screen shows it read-only | -This means if your `prod` stage sets `protected: true`, developers cannot create a fully-open production config through the normal workflow. Even picking `admin` in the TUI's access editor gets clamped back down at resolution time. +Each source overrides the one above it, then a protected stage caps `access`: -`isTest` is a seed value, not a ceiling. A stage that sets `isTest: true` supplies the default, but a stored config's own `isTest` wins at resolution, and the TUI edit screen lets a developer uncheck it. - -::: tip Manual Override -The ceiling is enforced every time the config is resolved, so it survives even a manually edited state file — there's no stored value to hand-edit around. The protection is against accidents, not an attempt at real security. -::: - - -### Locked Stages - -Setting `locked: true` prevents configs from being deleted: - -```yaml -stages: - prod: - locked: true - defaults: - protected: true +```mermaid +flowchart TD + D["built-in defaults"] --> C["stored config<br/>state.enc"] + C --> E["NOORM_* variables"] + E --> P{"stage has<br/>protected: true?"} + P -->|yes| K["cap access at<br/>user operator<br/>agent viewer"] + P -->|no| R["effective config"] + K --> R ``` -A locked production config cannot be accidentally removed. Deletion stays blocked until someone edits `settings.yml` or the config stops matching the stage name. - -## Assigning a Config to a Stage +## What can a stage set? -There is no stage picker. A config links to a stage when their names match: a config named `prod` picks up the `prod` stage's defaults, secrets, and lock. Name your configs after your stages and the link happens on its own. -`settings.yml` is the only place that link is defined, so a config's stage is not stored on the config and cannot drift per developer. The SDK can override the lookup with `createContext({ stage: 'prod' })` when a config's name does not match the stage you want. +| Key | Effect | +|---|---| +| `description` | Shown in the TUI | +| `locked: true` | Configs on this stage cannot be deleted | +| `defaults` | `dialect`, `host`, `port`, `database`, `user`, `password`, `ssl`, `isTest`: the template the TUI creates the config from. `protected: true` also caps access on every resolve, below | +| `secrets` | Secrets this stage requires | -## Stage-Specific Secrets - -Stages can require secrets that get inserted into your database via SQL templates. These are for sensitive values like API keys and encryption keys—not database connection credentials. - -```yaml -stages: - staging: - secrets: - - key: STRIPE_TEST_KEY - type: api_key - description: Stripe test API key for payment testing - required: true +### What does `protected: true` cap? - - key: SENDGRID_KEY - type: api_key - description: SendGrid API key for email testing - required: false -``` -Secret types control how the TUI handles input: +It lowers each channel to the ceiling and leaves stricter values alone. The cap applies on every resolve, so editing stored state does not raise it. -| Type | Behavior | -|------|----------| -| `string` | Plain text input | -| `password` | Masked input, no echo | -| `api_key` | Masked input, no echo | -| `connection_string` | Plain text, URI validation | +| Config sets | Resolves to | +|---|---| +| `user: admin` | `operator` | +| `user: viewer` | `viewer` | +| `agent: admin` or `agent: operator` | `viewer` | +| `agent: false` | `false` (hidden) | -**Required secrets** show as missing in the TUI secret list until you set them, and the enforcement point is the template render. Reading `$.secrets.STRIPE_TEST_KEY` when nothing resolved it throws `Secret "STRIPE_TEST_KEY" not found (searched: config-local, global-local, vault)` and the build stops. It never renders as the literal text `undefined`. -**Optional secrets** work the same way at render time, so a template that may run without one should probe first with `{% if ('DEBUG_KEY' in $.secrets) { %}`. The `in` check does not throw. +## How do I require a secret? -### Universal Secrets +List it under a stage, or at the root of `settings.yml` to require it on every stage. A stage entry replaces a root entry with the same key. -Some secrets apply to all configs regardless of stage. Define these at the settings root level: +| Field | Meaning | +|---|---| +| `key` | The name templates read: `$.secrets.PAYMENT_GATEWAY_KEY` | +| `type` | `string` is plain text. `password` and `api_key` are masked. `connection_string` is plain text and must parse as a URI | +| `description` | Hint shown in the value prompt | +| `required` | Defaults to `true` | -```yaml -# Required by ALL configs -secrets: - - key: ENCRYPTION_KEY - type: password - description: Application encryption key for sensitive data +A required secret does not block a command by itself. The TUI lists it as missing, and the first template that reads it fails: -stages: - prod: - # Additional secrets for prod only - secrets: - - key: AWS_SECRET_KEY - type: api_key - description: AWS credentials for S3 uploads ``` - -A production config would need both `ENCRYPTION_KEY` (universal) and `AWS_SECRET_KEY` (stage-specific). - - -## Common Stage Patterns - -Here are patterns that work well for most teams: - - -### Three-Environment Setup - -```yaml -stages: - dev: - description: Local development - defaults: - dialect: postgres - host: localhost - port: 5432 - - staging: - description: Staging server - defaults: - dialect: postgres - protected: false - secrets: - - key: STRIPE_TEST_KEY - type: api_key - required: true - - prod: - description: Production - locked: true - defaults: - dialect: postgres - protected: true - secrets: - - key: STRIPE_LIVE_KEY - type: api_key - required: true +Secret "PAYMENT_GATEWAY_KEY" not found (searched: config-local, global-local, vault) ``` +For an optional secret, probe first with `{% if ('PAYMENT_GATEWAY_KEY' in $.secrets) { %}`. The `in` check does not throw. [Secrets](/guide/environments/secrets) covers where values come from. -### Test Database Stage -```yaml -stages: - test: - description: Ephemeral test database - defaults: - dialect: postgres - host: localhost - port: 5432 - isTest: true -``` +## What bites + -The `isTest: true` flag marks this as a test database. noorm uses this for: +| Symptom | Cause | Fix | +|---|---|---| +| Editing a stage's `host` or `port` does not change the config | `defaults` only seeds a config the TUI creates; the stored config wins after that | Edit the config, or set a `NOORM_CONNECTION_*` variable | +| No config appears for a stage | The stage sets no `defaults.dialect`, or this machine had no identity when `noorm ui` opened | Add `dialect` under `defaults`, or set up an identity, then reopen `noorm ui` | +| A database password is in git history | `defaults.password` sits in the committed `settings.yml` | Leave `password` out of `defaults`. Set it on each config in `noorm ui` | +| `Config "prod" is linked to locked stage "prod" and cannot be deleted` | The stage has `locked: true` | Remove `locked` from the stage in `settings.yml` | +| A config set to `admin` behaves as `operator` | The stage is `protected` | Intended. Put the config on an unprotected stage | -- Conditional build rules (include seed data only in test environments) -- SDK's `requireTest` option (refuses to connect if `isTest` is not true) -- Preventing accidental test operations against production +## Next -## What's Next? -- [Secrets](/guide/environments/secrets) - Managing sensitive values per stage -- [Configs](/guide/environments/configs) - Creating configs from stages -- [Organization](/guide/sql-files/organization) - Using build rules with settings.yml +- [Configs](/guide/environments/configs): roles, channels, and the permission matrix +- [Secrets](/guide/environments/secrets): set values and resolve them in templates +- [Organization](/guide/sql-files/organization): build folders and rules in `settings.yml` diff --git a/docs/guide/environments/vault.md b/docs/guide/environments/vault.md index 36c1d728..5a5bb295 100644 --- a/docs/guide/environments/vault.md +++ b/docs/guide/environments/vault.md @@ -1,258 +1,124 @@ # Vault -Team-shared secrets need to live somewhere everyone can access them. Local secrets stay on your machine, but what about API keys that the whole team needs? The vault stores encrypted secrets in the database itself, distributing them to team members automatically. +Share a secret with your whole team without chat messages or committed files. - -## Secret Resolution Hierarchy - -When a template references `secrets.API_KEY`, noorm checks three sources in order. The first match wins: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ 1. Config-specific local secret (highest priority) │ -│ Your override for this specific config │ -│ Location: .noorm/state/state.enc │ -├─────────────────────────────────────────────────────────────┤ -│ 2. Global local secret │ -│ Your shared secret across all configs │ -│ Location: .noorm/state/state.enc │ -├─────────────────────────────────────────────────────────────┤ -│ 3. Vault secret (lowest priority) │ -│ Team-shared value from the database │ -│ Location: the vault table │ -└─────────────────────────────────────────────────────────────┘ -``` - -**Local always wins.** This lets you override team secrets for testing without affecting others. - - -### Example: Overriding a Team Secret - -Your team stores the production Stripe key in the vault: - -``` -Vault: STRIPE_KEY = "sk_live_abc123..." -``` - -For local development, you want to use Stripe's test mode. Set a local override through the TUI: `noorm ui` → Config → highlight the config → `k` → `a`, with key `STRIPE_KEY` and value `sk_test_xyz789...`. Headlessly, `noorm secret set STRIPE_KEY "sk_test_xyz789..."`. Local secrets are stored encrypted in your `.noorm/state/state.enc` and never leave your machine. - -Now when templates reference `secrets.STRIPE_KEY`: - -| Context | Resolved Value | Source | -|---------|----------------|--------| -| Your machine | `sk_test_xyz789...` | Local (your override) | -| Teammate's machine | `sk_live_abc123...` | Vault (no local override) | -| CI/CD pipeline | `sk_live_abc123...` | Vault (no local override) | - -Your local override doesn't affect anyone else. - - -### When to Use Each Layer - -| Layer | Use When | -|-------|----------| -| **Config-specific local** | Testing with different credentials per environment | -| **Global local** | Personal API keys you use across all configs | -| **Vault** | Team-shared secrets everyone needs | - - -## How Encryption Works - -The vault uses dual-layer encryption to share secrets securely: - -**Layer 1: Vault Key Distribution** - -A single 256-bit vault key encrypts all secrets. Each team member receives their own encrypted copy of this key: - -1. Generate ephemeral X25519 keypair -2. Perform ECDH with recipient's public key -3. Derive encryption key via HKDF-SHA256 -4. Encrypt vault key with AES-256-GCM -5. Store in user's `encrypted_vault_key` column - -When you connect, noorm decrypts your copy of the vault key using your private identity key. - -**Layer 2: Secret Encryption** - -Individual secrets are encrypted with the shared vault key: - -- AES-256-GCM authenticated encryption -- Random IV per secret -- Stored as JSON in the vault table - -This design means: compromising the database alone doesn't expose secrets. An attacker needs both database access AND a team member's private key. - - -## Initializing the Vault - -The first team member to initialize creates the vault key: +The first developer creates the vault and stores a secret in it: ```bash noorm vault init +printf %s "$KEY" | noorm vault set PAYMENT_GATEWAY_KEY --stdin +noorm vault list ``` -This generates a new vault key and stores it encrypted for your identity. Only one person needs to do this per database. - -If someone else already initialized: - ``` -Vault already initialized but you do not have access. -Ask a team member to propagate. +Vault initialized successfully +Vault secret "PAYMENT_GATEWAY_KEY" set successfully +Vault secrets (1): + PAYMENT_GATEWAY_KEY (set by ana@example.com) ``` +The vault lives in the config's own database, one per config. Templates read it as the last tier of [`$.secrets`](/guide/environments/secrets#which-value-does-a-template-get), so a local value overrides it for one developer. -## Managing Secrets -Once you have vault access, manage secrets with these commands: +## How does a teammate get access? -```bash -# Set a secret -noorm vault set API_KEY "sk-live-..." -noorm vault set DB_PASSWORD "secret123" -# List all secrets (values hidden) -noorm vault list +A new teammate is registered but holds no key. Someone with access grants it: -# Remove a secret -noorm vault rm OLD_API_KEY +```mermaid +sequenceDiagram + participant Ana + participant Database + participant Ben + Ana->>Database: vault init, key sealed to Ana + Ana->>Database: vault set PAYMENT_GATEWAY_KEY + Ben->>Database: vault list, Ben is registered + Database-->>Ben: refused, no key yet + Ana->>Database: vault propagate + Database-->>Ana: pending list, nothing granted + Ana->>Database: vault propagate --yes + Database->>Database: key sealed to Ben's public key + Ben->>Database: vault list + Database-->>Ben: secrets, decrypted with Ben's key ``` -The TUI provides a visual interface at **Vault** (`v` from home): - -- View all vault secrets, each labeled with who set it -- Add a secret with `a`, edit one by selecting it -- See pending users who need access -- One-key propagation (`p`) - -Removing a vault secret is CLI-only: `noorm vault rm <KEY>`. - - -## Granting Team Access - -New team members register their identity but don't automatically get vault access. Someone with access must propagate the key: +Ben needs an identity first: `noorm identity init`. Without `--yes`, `propagate` lists who is waiting and grants nothing. With it: ```bash -noorm vault propagate +noorm vault propagate --yes ``` -This encrypts the vault key for each pending user's public key. After propagation, they can decrypt vault secrets on their next connection. - -The Vault screen counts them in its header: - ``` -3 secret(s) | 4 user(s) with access (2 pending) +Granted vault access to 1 identities + Ben Ortiz <ben@example.com> 4a5c14af... ``` -Press `p` there to propagate. noorm lists the recipients it is about to grant access to and asks you to confirm before it writes anything. +In the TUI, `v` from home opens the vault. The header counts waiting teammates, `p` grants after listing them, and `a` adds a secret. -## Copying Secrets Between Configs +## Which commands manage the vault? -Move a secret from one database to another. `vault cp` takes three positional arguments in this order: the key, the source config, and the destination config. All three are required, so copy one key per invocation. -```bash -# Copy one secret -noorm vault cp API_KEY staging production +| Command | Effect | +|---|---| +| `noorm vault init` | Create the vault key and seal it to you. Safe to repeat | +| `noorm vault set <KEY> [value] [--stdin]` | Add or replace a secret | +| `noorm vault list` | Keys and who set them, never values | +| `noorm vault rm <KEY> --yes` | Delete a secret. The TUI cannot | +| `noorm vault propagate [--to <hash,...>] --yes` | Grant the key to waiting teammates | +| `noorm vault cp <KEY> <source> <destination>` | Copy one secret between configs | -# Overwrite a key that already exists on the destination -noorm vault cp API_KEY staging production --force +Each takes `--json`, and every command but `cp` takes `--config`. Keys follow the [secret key rule](/guide/environments/secrets#how-do-i-set-a-secret). -# Preview without writing -noorm vault cp API_KEY staging production --dry-run -``` -A dry run performs the same preflight as a real copy (vault access on both ends, source key exists, destination collisions) and only skips the write. +## How is it encrypted? -If the destination vault isn't initialized, noorm initializes it automatically using your identity. +| What | Encrypted with | Stored in | +|---|---|---| +| Each secret | One 256-bit vault key, AES-256-GCM, random IV | `noorm.vault`, column `encrypted_value` | +| The vault key, once per teammate | That teammate's public key: ephemeral X25519, HKDF-SHA256, AES-256-GCM | `noorm.identities`, column `encrypted_vault_key` | -## Using Vault Secrets in Templates +A null `encrypted_vault_key` marks a teammate as waiting. Those are the PostgreSQL and SQL Server names. MySQL and SQLite use `__noorm_vault__` and `__noorm_identities__`. Someone with only the database sees ciphertext. Reading a secret takes the database and a granted teammate's private key. -Vault secrets merge into the template context alongside local secrets: -```sql --- sql/setup/external-api.sql.tmpl --- Vault secret (or local override if set) -INSERT INTO api_config (provider, key) -VALUES ('stripe', {%~ $.quote($.secrets.STRIPE_KEY) %}); -``` +## How do I copy a secret to another config? -The resolution order means you can: - -1. Store production `STRIPE_KEY` in the vault -2. Override with a test key locally for development -3. Templates work identically in both cases - - -## CI/CD Integration - -Vault commands run non-interactively, so you can manage secrets straight from a pipeline: ```bash -# Initialize vault -noorm vault init - -# Set secret (value as argument) -noorm vault set API_KEY "$API_KEY" - -# List with JSON output -noorm vault list --json - -# Copy a secret between environments -noorm vault cp API_KEY staging production +noorm vault cp PAYMENT_GATEWAY_KEY staging prod --dry-run ``` -Pipe the value in with `--stdin` to keep it out of the process table, shell history, and `set -x` traces: - -```bash -echo "$SECRET_VALUE" | noorm vault set MY_SECRET --stdin +``` +Would copy 1 secrets: PAYMENT_GATEWAY_KEY ``` +A dry run checks vault access on both configs, that the key exists, and collisions, then writes nothing. A real copy initializes the destination vault if it has none. An existing key is skipped unless you pass `--force`. -## Security Considerations - -The vault design provides several security properties: - -1. **End-to-end encryption** - Secrets encrypted in memory, transmitted encrypted, stored encrypted -2. **Zero-knowledge database** - Database administrators see only ciphertext -3. **Per-user key distribution** - Propagation re-encrypts the same vault key for each recipient, so every user holds unique ciphertext -4. **No shared secrets in transit** - ECDH derives keys without transmitting them -5. **Authenticated encryption** - AES-256-GCM detects tampering - -Limitations to understand: - -- Team members with vault access can read all vault secrets -- Removing access requires regenerating the vault key (not yet supported) -- Lost private keys mean lost vault access (identity must be re-registered) +## How does a CI runner get access? -## Database Schema -The vault uses two database structures. Their names depend on the dialect: PostgreSQL and SQL Server put noorm's tracking tables in a dedicated `noorm` schema (`noorm.vault`, `noorm.identities`), while MySQL and SQLite have no schemas and keep the prefixed forms (`__noorm_vault__`, `__noorm_identities__`) in the default schema. +A developer with access runs `noorm ci identity enroll --config prod --name "GitHub CI" --email ci@example.com --yes` once. It registers the runner's identity and seals the key to it. [CI/CD](/guide/automation/ci) covers the runner side. -**The vault table:** -| Column | Type | Description | -|--------|------|-------------| -| `id` | int | Primary key | -| `secret_key` | string | Secret name (unique) | -| `encrypted_value` | text | AES-256-GCM ciphertext (JSON) | -| `set_by` | string | Identity who set it | -| `created_at` | timestamp | Creation time | -| `updated_at` | timestamp | Last update time | +## What bites -**The identities table extension:** -| Column | Type | Description | -|--------|------|-------------| -| `encrypted_vault_key` | text | Vault key encrypted for this user (nullable) | +| Symptom | Cause | Fix | +|---|---|---| +| `No vault access. Wait for a team member to propagate access.` | Your identity holds no sealed key yet | Ask a teammate to run `noorm vault propagate --yes` | +| `vault rm` exits 1: cannot be undone | The vault keeps no history, so it needs `--yes` | Check the key, then pass `--yes` | +| A `vault set` or `vault init` asks for confirmation | Your role on the config is `operator` | Pass `--yes` | +| A former teammate who still reaches the database can read the vault | A grant cannot be revoked, and no command rotates the vault key | Change the secret at its source, then `vault set` the new value | +| Everyone with access reads every secret | The vault has one key for all of it | Keep per-person values in local secrets | -A null `encrypted_vault_key` means the user is pending vault access. +## Next -## What's Next? -- [Secrets](/guide/environments/secrets) - Local secret storage -- [Identity](/cli/identity) - The keypair that lets teammates encrypt secrets to you -- [Templates](/guide/sql-files/templates) - Using secrets in dynamic SQL +- [Secrets](/guide/environments/secrets): resolution order and local overrides +- [Configs](/guide/environments/configs): the `vault:*` permissions per role +- [Identity](/cli/identity): create the keypair a teammate is registered with +- [CI/CD](/guide/automation/ci): enroll a runner and load job secrets diff --git a/docs/guide/relational-design.md b/docs/guide/relational-design.md index dd71a016..1d43c75f 100644 --- a/docs/guide/relational-design.md +++ b/docs/guide/relational-design.md @@ -1,299 +1,206 @@ --- title: The case for proper relational design -description: Why inherited keys and basetype-subtypes beat the surrogate-ID-everywhere pattern ORMs push you toward. +description: Why inherited keys and a basetype table let the database refuse bad rows, shown on the order-to-cash schema, with a measured cost for the alternative. --- # The case for proper relational design -noorm does not stop you from designing a database the way an ORM would. It just stops making that the only option you can reach. This page explains what you get back once the tool is out of the way. - +A child table that inherits its parent's key lets the database refuse rows that break a business rule. A surrogate ID on every table leaves that refusal to application code. noorm runs the SQL you write, so which one you get is your decision. + +`party_id` enters at `party` and reaches `payment_allocation` down two branches, so the table stores it once and both parents must agree on it: + +```mermaid +erDiagram + party { + int party_id PK + } + payment { + int party_id PK, FK + int payment_no PK + } + invoice { + int party_id PK, FK + int invoice_no PK + } + invoice_line { + int party_id PK, FK + int invoice_no PK, FK + int line_no PK + } + payment_allocation { + int party_id PK, FK + int payment_no PK, FK + int invoice_no PK, FK + int line_no PK, FK + } + party ||--o{ payment : "makes" + party ||--o{ invoice : "owes on" + invoice ||--|{ invoice_line : "contains" + payment ||--|{ payment_allocation : "is applied as" + invoice_line ||--o{ payment_allocation : "is settled by" +``` -## Inherited keys -ORMs push you toward a pattern: every table gets a surrogate ID, relationships happen through foreign keys, and you join your way back to find what you need. It works—until you're seven joins deep trying to figure out which user owns a deeply nested entity, and your messy left joins are adding NULL rows or creating cartesian products. +## What does an inherited key enforce? -Proper relational design uses **inherited keys**. Instead of giving every entity an independent identity, child entities inherit their parent's key as part of their own. -**Example: A todo list** +`payment_allocation` has two parents, `payment` and `invoice_line`. Both keys start with `party_id`, so the table holds it once and both foreign keys read it: ```sql -CREATE TABLE users ( - user_id serial PRIMARY KEY, - email text NOT NULL UNIQUE, - created_at timestamptz NOT NULL DEFAULT now() -); - -CREATE TABLE todos ( - user_id int NOT NULL REFERENCES users (user_id), - created_at timestamptz NOT NULL, - title text NOT NULL, - - PRIMARY KEY (user_id, created_at) +CREATE TABLE payment_allocation ( + party_id integer NOT NULL, + payment_no integer NOT NULL, + invoice_no integer NOT NULL, + line_no integer NOT NULL, + amount numeric(12,2) NOT NULL, + CONSTRAINT payment_allocation_pk + PRIMARY KEY (party_id, payment_no, invoice_no, line_no), + CONSTRAINT payment_allocation_isdrawnfrom_payment + FOREIGN KEY (party_id, payment_no) + REFERENCES payment (party_id, payment_no), + CONSTRAINT payment_allocation_settles_invoice_line + FOREIGN KEY (party_id, invoice_no, line_no) + REFERENCES invoice_line (party_id, invoice_no, line_no) ); +``` -CREATE TABLE todo_items ( - user_id int NOT NULL, - created_at timestamptz NOT NULL, - item_index int NOT NULL, - body text NOT NULL, - done boolean NOT NULL DEFAULT false, +Payment 1 belongs to party 1. Invoice 1, line 1 belongs to party 2. Settling one with the other needs a row whose `party_id` is both: - PRIMARY KEY (user_id, created_at, item_index), - FOREIGN KEY (user_id, created_at) REFERENCES todos (user_id, created_at) -); +```sql +INSERT INTO payment_allocation (party_id, payment_no, invoice_no, line_no, amount) +VALUES (1, 1, 1, 1, 49.00); ``` -No `todo_id`, no `todo_item_id`. A todo is identified by whose it is and when it was made; an item by which list it sits on and where in that list. - -Two things follow, and neither costs any application code. +``` +ERROR: insert or update on table "payment_allocation" violates foreign key constraint "payment_allocation_settles_invoice_line" +DETAIL: Key (party_id, invoice_no, line_no)=(1, 1, 1) is not present in table "invoice_line". +``` -`todo_items` carries `user_id` on every row, two levels down from where it was introduced. Finding the owner of an item is a column read, not a join. +Two foreign keys enforce the rule. No trigger, no application check. The same shape puts `party_id` on every row below `party`, so finding who owes on an invoice line is a column read. -The composite foreign key `(user_id, created_at)` makes cross-tenant corruption unrepresentable. An item cannot reference a list belonging to a different user, because the `user_id` in the item's own key has to match the one in the list's key. Compare the surrogate version: +A surrogate ID on every table gives the rule no place to live: ```sql --- The ORM shape. todo_id alone says nothing about who owns the list. -CREATE TABLE todo_items ( - todo_item_id serial PRIMARY KEY, - todo_id int NOT NULL REFERENCES todos (todo_id), - body text NOT NULL +CREATE TABLE payment_allocation ( + payment_allocation_id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + payment_id integer NOT NULL REFERENCES payment (payment_id), + invoice_line_id integer NOT NULL REFERENCES invoice_line (invoice_line_id), + amount numeric(12,2) NOT NULL ); ``` -Here `todo_id` points at a list, and nothing in the schema relates that list to a user. Every query that needs the owner joins for it, and every write that must not cross tenants is guarded in application code you have to remember to write. - -*Try working that into your ORM. I'll wait...* - - -### What that model looks like - -The diagram below is generated from the schema above, not drawn. It is a live [ignatius](/modeling/) export: click an entity for its columns, sample rows, and prose. - -<div class="model-embed"> -<iframe src="/models/todo-list.html" title="Inherited keys: users, todos, todo_items" loading="lazy"></iframe> -</div> +Each foreign key checks that its row exists. Nothing relates the payment's party to the invoice line's party, so the database accepts any pair. Every write that must not cross parties is guarded in code you have to remember to write, and finding the party takes a join. -Note the two node shapes. `users` is square because nothing above it supplies its identity. `todos` and `todo_items` are rounded because they cannot be identified without their parent. Nobody labelled those. ignatius derived them from the key shape, which is the same information the `PRIMARY KEY` clauses above carry. -The hash marks on each edge mean the relationship is **identifying**: the parent's key is inside the child's, so the child depends on it for identity rather than merely pointing at it. +## Why is there a party table instead of a customer type and ID? -## ~~Polymorphism~~ Basetype-subtypes - -ORMs love polymorphic associations: a `comments` table with `commentable_type` and `commentable_id`. Fast, flexible—and completely breaks referential integrity. Complex app logic, no foreign keys, slow and awkward statistics, and even more awkward queries. - -Here is the shape they push you toward: +An order is placed by a person or a business. A common shape for one-of-several parents is a type column and an ID column: ```sql --- No foreign key is possible here: the table this --- points at changes from row to row. -CREATE TABLE comments ( - comment_id serial PRIMARY KEY, - -- 'Post', 'Photo', 'Comment', … - commentable_type text NOT NULL, - -- references nothing at all - commentable_id int NOT NULL, - body text NOT NULL +CREATE TABLE sales_order ( + sales_order_id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + customer_type text NOT NULL, -- 'PERSON' or 'BUSINESS' + customer_id integer NOT NULL -- no foreign key is possible ); ``` -`commentable_id` is an integer the database has no opinion about. Point it at a row that was deleted, or at a `Photo` id while `commentable_type` says `Post`, and nothing objects. Counting comments per post means filtering on a string first. - -Modelled out, the whole domain looks like this: - -<div class="model-embed"> -<iframe src="/models/polymorphic.html" title="Polymorphic associations: the same domain in seven tables" loading="lazy"></iframe> -</div> +A foreign key has one target table, and `customer_id` has two. Point it at a deleted person, or at a business ID while `customer_type` says `PERSON`, and nothing objects. -Seven tables and one edge. `profiles` points at `users`, because a profile belongs to exactly one kind of thing and so was allowed to keep a foreign key. Everything else wanted to point at more than one kind, so it gave up pointing at all: `posts`, `photos`, `comments` and `tags` float unconnected, holding a type string and an integer nobody checks. - -That is the diagram the ORM shape produces. Compare it with the one at the end of this section. - -Proper relational design solved this years ago with **basetype-subtypes**: +A basetype gives the order one target. `party` holds the identity every customer shares, and `person` and `business` hold what is specific to each. A subtype's key is its basetype's key: +```mermaid +erDiagram + party_type ||--o{ party : "classifies" + party ||--o| person : "is realized as" + party ||--o| business : "is realized as" + party ||--o{ sales_order : "places" ``` -independent entities: user, group -dependent entities: profile -basetype-subtypes: post → user_post, group_post - photo → user_photo, group_photo, profile_photo, user_post_photo, ... - comment → user_comment, group_comment, post_comment, comment_comment, ... - tag → post_tag, photo_tag, comment_tag, ... -``` - -Polymorphism looks cheaper because it never makes you write this list down: one `comments` table absorbs every case, and the count of things you are actually modelling stays hidden in a string column. Here the count is on the page. Photo alone fans out four ways. - -More tables, and every one of them is a constraint the database enforces. The mechanism is the same for each, so here is one cluster in full: ```sql --- The owner types are rows, not a value list baked into DDL. --- Admitting ORGANIZATION later is an INSERT. -CREATE TABLE post_owner_types ( - owner_type text PRIMARY KEY -); - -INSERT INTO post_owner_types (owner_type) VALUES ('USER'), ('GROUP'); - -CREATE TABLE posts ( - post_id serial PRIMARY KEY, - owner_type text NOT NULL REFERENCES post_owner_types (owner_type), - body text NOT NULL, - posted_at timestamptz NOT NULL DEFAULT now() +CREATE TABLE party_type ( + code text NOT NULL, + description text NOT NULL, + CONSTRAINT party_type_pk PRIMARY KEY (code) ); -CREATE TABLE user_posts ( - post_id int PRIMARY KEY REFERENCES posts (post_id), - user_id int NOT NULL REFERENCES users (user_id) +CREATE TABLE party ( + party_id integer GENERATED ALWAYS AS IDENTITY, + party_type text NOT NULL, + name text NOT NULL, + CONSTRAINT party_pk PRIMARY KEY (party_id), + CONSTRAINT party_isclassifiedby_party_type + FOREIGN KEY (party_type) REFERENCES party_type (code) ); -CREATE TABLE group_posts ( - post_id int PRIMARY KEY REFERENCES posts (post_id), - group_id int NOT NULL REFERENCES groups (group_id) +CREATE TABLE person ( + party_id integer NOT NULL, + given_name text NOT NULL, + family_name text NOT NULL, + CONSTRAINT person_pk PRIMARY KEY (party_id), + CONSTRAINT person_isa_party + FOREIGN KEY (party_id) REFERENCES party (party_id) ); ``` -Two things to notice. - -The set of owner types is a table. A `CHECK (owner_type IN (…))` list would have made admitting `ORGANIZATION` a constraint rewrite on the basetype; here it is a row. - -Each subtype's key *is* the basetype's key, so a post gets at most one row in any subtype table and every one of those rows is anchored to a real `posts` row. Keeping a post out of two different subtype tables at once is the one rule this shape leaves to you if you're using postgres. Enforcement is fully possible in TSQL via cross-table constraints. - -`user_id` and `group_id` are real foreign keys to real tables, which the polymorphic version gives up. Counting a group's posts is a join, not a join plus a string comparison. - -Repeat that for photo, comment and tag and you get the list above. It is more tables than the polymorphic version. The tables are where the rules live, so they are not also living in application code you have to keep correct. - -Each relationship gets its own table with proper constraints against its parent. A `user_post` has a foreign key to `user` and `post`. A `group_photo` has a foreign key to `group` and `photo`. No nulls, no type string you have to read to know what you are holding, no ambiguity. - -A row exists or it does not, so nothing has to be computed to find out. Statistics are straightforward. Queries are clean. The database enforces integrity at every level. Illegal states become impossible. The trade is more tables for less application logic. - - -### What that model looks like - -<div class="model-embed tall"> -<iframe src="/models/social.html" title="Basetype-subtypes: the full social graph" loading="lazy"></iframe> -</div> - -All twenty entities, exactly as listed above. Four diamonds, one per cluster, each with an X marking it **exclusive**: a post is one or the other, never both. That X is the model stating the one rule the tables leave to you. - -`post_owner_types` and its siblings are not among the twenty. A table whose only job is to enumerate the legal values of a column is a domain rather than an entity: it describes what a value may be, not a thing the business has. It belongs in the data dictionary, not on the graph. - -Solid lines run to a basetype, dashed ones to an owner. Dashed means non-identifying, so a photo's subject is a fact *about* it rather than part of what identifies it. - -Count the edges. Every line is a foreign key the database enforces. - -Scroll back to the polymorphic diagram. Twenty entities bound by constraints, against seven that float. Fewer tables did not remove the relationships. It removed the database's knowledge of them, and moved every one into code you have to write, test, and keep correct. - - - -## Metrics, and the truth at 1M rows - -The usual claim is that polymorphism is slower. Here is a metric both designs have to answer, on a real database, so we can stop guessing. - -**Comments per group, across every group.** Polymorphic: +`business` has the same shape. The kinds of party are rows in `party_type`, so admitting a new kind is an `INSERT`, where a `CHECK` list would be a constraint rewrite. `sales_order.party_id` is now a real foreign key: ```sql -SELECT p.owner_id AS group_id, count(*) AS comments -FROM posts p -JOIN comments c - ON c.commentable_id = p.post_id - AND c.commentable_type = 'POST' -WHERE p.owner_type = 'GROUP' -GROUP BY p.owner_id -ORDER BY comments DESC -LIMIT 10; +INSERT INTO sales_order (party_id, order_no) VALUES (999, 1); ``` -Two string comparisons before any counting starts, and the join predicate has to carry the type or it silently counts photos as posts. Basetype-subtypes: - -```sql -SELECT gp.group_id, count(*) AS comments -FROM group_posts gp -JOIN post_comments pc ON pc.post_id = gp.post_id -GROUP BY gp.group_id -ORDER BY comments DESC -LIMIT 10; +``` +ERROR: insert or update on table "sales_order" violates foreign key constraint "sales_order_isplacedby_party" +DETAIL: Key (party_id)=(999) is not present in table "party". ``` -No type filter anywhere. `group_posts` is already only group posts, `post_comments` is already only comments on posts. The tables did the filtering when the rows were written. - +One rule stays open: nothing in these keys stops a party from having both a `person` row and a `business` row. The model marks the cluster exclusive, and enforcing that takes one more constraint. -### What the numbers say +[ignatius](/modeling/) draws the cluster from the same key shape: a diamond joins the basetype to its members, and an X marks it exclusive. -PostgreSQL 17, 1,000,000 posts and 5,000,000 comments, both designs indexed equivalently, best of three runs. +<ModelViewer class="tall" model="order-to-cash" hash="view=graph" :tabs="false" label="order-to-cash graph" title="Order-to-cash model graph rendered by ignatius" /> -Answering the metric above, one join: -| | Polymorphic | Basetype-subtypes | -|---|---|---| -| Time | 149.9 ms | 134.6 ms | -| Pages read from disk | 7,852 | 5,022 | +## What does a type-and-ID column cost on reads? -Close enough that nobody should pick a schema over it. -Now add a hop. Comments on a group's posts, plus the tags on those comments: +The measurement used a separate social-graph schema, built both ways: comments and tags in shared tables keyed by a type column and an ID column, against per-parent tables such as `post_comments` and `comment_tags`. PostgreSQL 17, 1,000,000 posts and 5,000,000 comments, both designs indexed equivalently, best of three runs. -```sql --- Polymorphic. Every hop re-states its type filter. -SELECT p.owner_id, count(DISTINCT c.comment_id) AS comments, count(t.tag_id) AS tags -FROM posts p -JOIN comments c ON c.commentable_id = p.post_id AND c.commentable_type = 'POST' -LEFT JOIN tags t ON t.target_id = c.comment_id AND t.target_type = 'COMMENT' -WHERE p.owner_type = 'GROUP' -GROUP BY p.owner_id; -``` +| Query | Design | Time | Pages read | +|---|---|---|---| +| Comments per group (one join) | Type and ID | 149.9 ms | 7,852 | +| Comments per group (one join) | Subtype tables | 134.6 ms | 5,022 | +| Comments on a group's posts, plus their tags (two joins) | Type and ID | 638.2 ms | 47,765 | +| Comments on a group's posts, plus their tags (two joins) | Subtype tables | 452.7 ms | 4,638 | -```sql --- Basetype-subtypes. No type filter anywhere. -SELECT gp.group_id, count(DISTINCT pc.comment_id) AS comments, count(ct.tag_id) AS tags -FROM group_posts gp -JOIN post_comments pc ON pc.post_id = gp.post_id -LEFT JOIN comment_tags ct ON ct.comment_id = pc.comment_id -GROUP BY gp.group_id; -``` +At one join the times are 15 ms apart. The second join took the type-and-ID query from 7,852 to 47,765 pages read and left the subtype query flat. -| | Polymorphic | Basetype-subtypes | -|---|---|---| -| Time | 638.2 ms | 452.7 ms | -| Pages read from disk | **47,765** | **4,638** | +The likely reason is that each type-and-ID join re-derives the row kind at read time, by sifting a large shared table for the right type. A subtype table holds only the right rows, because the write path sorted them once. A third join would pay the sift again. That is inference from where the work happens, and no third join was measured. -Adding one hop took the polymorphic query from 7,852 pages to 47,765, a six-fold jump. The same hop left the relational query flat, 5,022 to 4,638. +The subtype design used more disk, 803 MB against 638 MB, because it has more tables and indexes and stores inherited keys more than once. The two-join gap is 1.4x in wall-clock time because the data mostly fits in cache. -The reason is mechanical. Every polymorphic join has to re-derive the same fact at read time: sift a large shared table for the fraction of rows that are the right *kind*. Two hops means doing that twice, over five million comments and two million tags, and the intermediate results are large enough that both designs spill to temp files. -The subtype version never sifts anything. `post_comments` is already only comments on posts. `comment_tags` is already only tags on comments. Each join lands on a table that is exactly the rows wanted, because the write path sorted that out once instead of every query sorting it out again. +## What happens on delete? -That cost is per hop, so a third or fourth join pays it again while the subtype side does not. Only the two measurements above were taken; the rest follows from where the work happens. -Two caveats. The relational design uses **more** disk, 803 MB against the polymorphic 638 MB, because more tables mean more indexes and inherited keys are stored more than once. And on wall clock the two-hop gap is 1.4x, not 10x, because this data still mostly fits in cache. The read counts are what tell you which way that goes when it stops fitting. - - -### The part that has no benchmark - -Delete a post that has comments. +Delete an invoice line that an allocation settles: ```sql -DELETE FROM posts WHERE post_id = 2; +DELETE FROM invoice_line WHERE party_id = 2 AND invoice_no = 1 AND line_no = 1; ``` -Polymorphic: `DELETE 1`. It works. The comment now points at a post that does not exist, and the metric still reports **1,000,000** post comments, exactly as before. The number is wrong and nothing anywhere says so. - -Basetype-subtypes: the database refuses. - ``` -ERROR: update or delete on table "posts" violates foreign key -constraint "group_posts_post_id_fkey" +ERROR: update or delete on table "invoice_line" violates foreign key constraint "payment_allocation_settles_invoice_line" on table "payment_allocation" +DETAIL: Key (party_id, invoice_no, line_no)=(2, 1, 1) is still referenced from table "payment_allocation". ``` -You can find the orphans in the polymorphic design. It costs an anti-join across all five million comments, 154 ms here: +A type-and-ID pair cannot carry a foreign key, so the equivalent delete succeeds and the child row points at nothing. Totals built from it are wrong and nothing reports it. Finding such orphans takes an anti-join per pair, 154 ms across the five million comments above, and it reports damage already done. -```sql -SELECT count(*) FROM comments c -WHERE c.commentable_type = 'POST' - AND NOT EXISTS (SELECT 1 FROM posts p WHERE p.post_id = c.commentable_id); -``` -But you have to know to ask, on every polymorphic column, forever, and the answer only tells you about damage already done. That query does nothing for your product; it is rent. +## Next -So you spend more disk and get back two things: queries whose cost grows more slowly as they get deeper, and the guarantee that the number on the dashboard is true. The polymorphic version spends less disk and pays for it per query, per hop, and in the reconciliation jobs it obliges you to write. -You pay for bad relational design later, in complexity and bugs. Sometimes you pay for it in metrics nobody knew to distrust. +- [From model to SQL](/modeling/model-to-sql): how each entity file becomes a `CREATE TABLE` +- [Entities and key inheritance](/modeling/entities): the entity format and subtype clusters +- [Concepts](/getting-started/concepts): how `db/sql/` and `db/changes/` divide the work diff --git a/docs/guide/sql-files/execution.md b/docs/guide/sql-files/execution.md index 4f7fad81..487101fd 100644 --- a/docs/guide/sql-files/execution.md +++ b/docs/guide/sql-files/execution.md @@ -1,265 +1,178 @@ # Execution -## How Execution Works - -When you run SQL files with noorm, every file goes through a checksum-based change detection process. This is the core value proposition: files that haven't changed don't run again. - -Here's what happens under the hood: - -1. noorm computes a SHA-256 checksum of the SQL that will actually run — for a `.sql.tmpl`, that is the *rendered* output, not the file on disk -2. It checks the tracking database for a previous execution record -3. If the file is new, changed, previously failed, or belonged to an operation a teardown marked stale, it runs -4. If unchanged and successful, it's skipped -5. After execution, the result and new checksum are recorded - -Hashing the rendered output is what makes a template re-run when its data file, config, or secrets change even though its own bytes did not. The flip side: a template whose render is genuinely non-deterministic — one calling `$.now()` or `$.uuid()` — produces new SQL every time and therefore re-runs every build. That is the correct answer for a file whose output really is different each time; use a fixed value if you want it to settle. - -This makes builds idempotent. Run the same command twice—unchanged files won't re-execute. - - -## run build - -The primary command for executing your schema: +Run your SQL files against a database and re-run only the ones whose SQL changed. ```bash noorm run build ``` -This executes all SQL files in your schema directory (defined in settings). Files are processed alphabetically by path, which means you can control execution order with prefixes: - ``` -sql/ -├── 00_extensions/ -│ └── 001_extensions.sql # Runs first -├── 01_tables/ -│ ├── 001_users.sql # Runs second -│ └── 002_posts.sql # Runs third -└── 02_views/ - └── 001_active_users.sql # Runs last +• log Build completed status=success filesRun=5 filesSkipped=0 filesFailed=0 durationMs=53.388000000000034 ``` -Output shows what ran and what was skipped. Every line carries a timestamp and a level marker ahead of the message: +Run it again and every file is skipped: -``` -Starting schema build (3 files) -Executed sql/01_tables/001_users.sql (14ms) -Skipped sql/01_tables/002_posts.sql (unchanged) -Executed sql/02_views/001_active_users.sql (9ms) -Build complete: 2 run, 1 skipped (412ms) +```bash +noorm run build ``` -Add `--json` to get the same events as newline-delimited JSON on stderr and the batch result on stdout. +``` +• log /srv/shop/db/sql/01_tables/01_party.sql (skipped: unchanged) +• log /srv/shop/db/sql/01_tables/02_sales_order.sql (skipped: unchanged) +• log /srv/shop/db/sql/01_tables/03_product.sql (skipped: unchanged) +• log /srv/shop/db/sql/02_views/01_open_orders.sql (skipped: unchanged) +• log /srv/shop/db/sql/09_seeds/01_products.sql.tmpl (skipped: unchanged) +• log Build completed status=success filesRun=0 filesSkipped=5 filesFailed=0 durationMs=8.991874999999993 +``` +Each line also starts with a `[YY-MM-DD HH:mm:ss]` timestamp, left out here. -## run file -Execute a single SQL file: +## Which command runs what? -```bash -noorm run file sql/01_tables/001_users.sql -``` -Useful for testing a specific file or re-running one file after a quick fix. The same change detection applies—if the file hasn't changed since its last successful run, it won't execute unless you use `--force`. +| Command | Runs | Applies `build.include`, `build.exclude`, rules | +|---|---|---| +| `noorm run build` | every file under `paths.sql` | yes | +| `noorm run file <path>` | one file | no | +| `noorm run dir <path>` | every file under a directory | no | +| `noorm run exec <dir-or-glob>` | a directory, or the files a glob matches | no | +| `noorm run files --paths a.sql,b.sql` | the listed files, in the order listed | no | +| `noorm run preview <file>` | nothing: prints a template's rendered SQL | n/a | +| `noorm run inspect <file>` | nothing: lists what a template can read from `$` | n/a | +Directories and globs run in path order, the order [Organization](/guide/sql-files/organization) describes. The five commands that run SQL take these flags: -## run dir +| Flag | Effect | +|---|---| +| `--force`, `-f` | Run every file without the checksum test | +| `--dry-run` | Render to `tmp/`. Run nothing and record nothing | +| `--config`, `-c` | Use a config other than the active one | +| `--json` | Print the result as JSON on stdout | -Execute all files in a specific directory: +`preview` and `inspect` take `--config` and `--json` only. Exit codes and JSON shapes are in [`noorm run`](/cli/run). -```bash -noorm run dir sql/02_views -``` - -Processes all `.sql` and `.sql.tmpl` files in that directory (and subdirectories), alphabetically. This is helpful when you want to rebuild just views or just a subset of your schema. +## How does noorm decide to skip a file? -## Force Mode -Skip change detection entirely with `--force`: +A file skips only when its newest record succeeded on the same SQL. Everything else runs: -```bash -noorm run build --force +```mermaid +flowchart TD + A["render, then hash the<br/>SQL with SHA-256"] --> B{"--force?"} + B -->|yes| RUN[run] + B -->|no| C{"record for this path<br/>and this config?"} + C -->|no| RUN + C -->|yes| D{"last attempt failed<br/>or never started?"} + D -->|yes| RUN + D -->|no| E{"marked stale by<br/>a teardown?"} + E -->|yes| RUN + E -->|no| F{"checksum differs?"} + F -->|yes| RUN + F -->|no| SKIP[skip] ``` -Every file executes, regardless of whether it changed. Use this when: +The hash covers the SQL the database would receive. For a `.sql.tmpl` that is the rendered output, so a changed data file, config, or secret re-runs the template while its own bytes stay the same. -- You need to rebuild everything after a database restore -- Troubleshooting an issue and want fresh execution -- External changes happened that noorm doesn't know about +One lookup decides the whole batch before anything runs. A skipped file leaves no row, and a batch where every file skips creates no operation record. Records are per config and per file path, whichever command wrote them, so `run file` after `run build` skips an unchanged file. -In the TUI, the force option appears when you select a run command. +## What does --dry-run do? -## Dry Run Mode -Preview what would execute without actually running anything: +It renders every file, templates included, and writes the result under `tmp/`, mirroring the source path and dropping `.tmpl`. Nothing runs and nothing is recorded. The skip test does not apply, so unchanged files render too: ```bash noorm run build --dry-run ``` -This renders all SQL files (including templates) and writes them to a local `tmp/` directory that mirrors your source structure: - ``` -Source: Dry run output: -sql/02_views/001_my_view.sql → tmp/sql/02_views/001_my_view.sql -sql/03_seeds/001_users.sql.tmpl → tmp/sql/03_seeds/001_users.sql +• log /srv/shop/db/sql/09_seeds/01_products.sql.tmpl (success, dry-run) -> /srv/shop/tmp/db/sql/09_seeds/01_products.sql +• log Build (dry-run) completed status=success filesRun=5 filesSkipped=0 filesFailed=0 durationMs=11.70675 dryRun=true ``` -Templates are fully rendered with your current config context. The `.tmpl` extension is stripped from output files. Nothing is written to the database and nothing is tracked. - -::: warning Dry run output contains your secrets -Rendering resolves every secret the template reads, in plaintext, into the files under `tmp/`. noorm writes them owner-only (`0600`, in `0700` directories), but `noorm init` does not add `tmp/` to `.gitignore`. Add it yourself, and delete the output when you're done reading it. +::: warning Dry-run files hold your secrets +Rendering resolves every secret a template reads and writes it to `tmp/` in plain text. The files are owner-only (`0600`, in `0700` directories), but `noorm init` does not add `tmp/` to `.gitignore`. Add it, and delete the output when you finish. ::: -**When to use dry run:** +## What gets tracked -| Scenario | Command | -|----------|---------| -| Inspect rendered templates | `noorm run build --dry-run` | -| Review before production deploy | `noorm run build --dry-run -c production` | -| Debug one template | `noorm run preview seed.sql.tmpl` | -| CI/CD validation step | `noorm run build --dry-run` | +Each batch that runs writes to two tables in your database. PostgreSQL and SQL Server keep them in a `noorm` schema, as `noorm.change` and `noorm.executions`. MySQL and SQLite name them `__noorm_change__` and `__noorm_executions__`. -## What Gets Tracked +| Table | Columns | +|---|---| +| change (one row per batch) | `id`, `name`, `change_type`, `direction`, `checksum`, `executed_at`, `executed_by`, `config_name`, `cli_version`, `status`, `error_message`, `duration_ms` | +| executions (one row per file) | `id`, `change_id`, `filepath`, `file_type`, `checksum`, `cli_version`, `status`, `error_message`, `skip_reason`, `duration_ms` | -noorm maintains two tables in your database to track execution history. On PostgreSQL and SQL Server they live in a dedicated `noorm` schema; MySQL and SQLite have no schemas, so they keep prefixed names in the default schema. +`change_type` is `build`, `run`, or `change`. `filepath` is relative to the project root. Before the first file runs, every file in the run set gets a `pending` row, so an interrupted build still shows what it intended. The next build treats a `pending` row as never started. -**`noorm.change`** (`__noorm_change__` on MySQL and SQLite) - Operation records: -| Field | Description | -|-------|-------------| -| `name` | Operation identifier (e.g., `build:2024-01-15T10:30:00.000Z`) | -| `change_type` | `'build'`, `'run'`, or `'change'` | -| `executed_by` | Identity string (who ran it) | -| `config_name` | Which config was used | -| `status` | `'pending'`, `'success'`, `'failed'`, `'reverted'`, `'stale'` | +## What happens when a file fails? -**`noorm.executions`** (`__noorm_executions__` on MySQL and SQLite) - Individual file records: -| Field | Description | -|-------|-------------| -| `change_id` | FK to parent operation | -| `filepath` | File that was executed, relative to the project root | -| `checksum` | SHA-256 of the SQL that actually ran (the rendered output for a `.sql.tmpl`) | -| `status` | `'pending'`, `'success'`, `'failed'`, `'skipped'` | -| `skip_reason` | `Skipped: failure in <file>` when an earlier file in the batch failed | -| `duration_ms` | Execution time | +The batch stops at the first failure. Files after it are not attempted: -Every file that will run gets a `'pending'` row before the first one executes, so an interrupted build still shows you what it was going to do. Rows move to their final status as the run proceeds. A file that change detection skips gets no row. +| File | Recorded as | Next build | +|---|---|---| +| ran before the failure | `success` | skipped | +| the one that failed | `failed` | runs | +| after the failure | `skipped`, with `Skipped: failure in <file>` | runs | -These tables let noorm answer: "Has this exact file content been executed before, and did it succeed?" +The command exits `3` when some files ran and `1` when none did. -## Execution Order +## How do I watch a slow file? -Files execute **alphabetically by full path**. This is deterministic and predictable, but it means you need to name files carefully when order matters. See [Organization](/guide/sql-files/organization) for detailed naming strategies. -Common patterns: - -``` -sql/ -├── 00_extensions/ -│ └── 001_uuid.sql # Extensions first -├── 01_types/ -│ └── 001_enums.sql # Types before tables -├── 02_tables/ -│ ├── 001_users.sql # Independent tables first -│ ├── 002_profiles.sql # Tables with FKs after their dependencies -│ └── 003_posts.sql -└── 03_views/ - └── 001_active_users.sql # Views last (they depend on tables) -``` - -The numbered prefixes ensure: -1. Extensions load before anything uses them -2. Types exist before tables reference them -3. Tables exist before views reference them -4. Within each category, files run in predictable order - -::: tip File Naming -Use `001_`, `002_` prefixes rather than `1_`, `2_` for consistent sorting. Leading zeros ensure `002` comes before `010` alphabetically. -::: +A file still running after 10 seconds logs `Still running <file> (Ns)` every 10 seconds. The entry carries the progress the database reports (an index build's phase and block count) and any session the file waits on for a lock, which tells a slow file from a stuck one. To cancel a run, use the TUI: [Long-Running Files](/tui#long-running-files). -## MSSQL: Multiple Statements per File +## Multiple statements per file -SQL Server has a particular rule: `CREATE PROCEDURE`, `CREATE FUNCTION`, `CREATE TRIGGER`, `CREATE VIEW`, and `CREATE TYPE` (for table-valued parameters) must each be the only statement in their batch. SQL Server expresses this with `GO` — a separator that ends a batch. - - -`GO` is not part of the SQL language. It's a sqlcmd / SSMS directive. Tools that pass raw file content directly to the driver get back `Incorrect syntax near 'GO'`. noorm handles this for you: on MSSQL connections, the runner splits files on `GO` (anchored to its own line) and executes each batch in order. +| Dialect | A file with several statements | +|---|---| +| PostgreSQL | Runs as one call, in one implicit transaction | +| SQL Server | Splits on a line that holds only `GO`, then runs each batch in order | +| SQLite | Splits on statement boundaries. Semicolons inside strings, quoted identifiers, comments, and trigger bodies do not count | +| MySQL | The driver rejects it. Use one statement per file | +SQL Server needs `GO` because `CREATE VIEW`, `CREATE PROCEDURE`, `CREATE FUNCTION`, `CREATE TRIGGER`, and `CREATE TYPE` must each be alone in their batch: ```sql --- sql/09_procedures/checkout.sql -CREATE PROCEDURE checkout_begin - @CustomerId INT -AS -BEGIN - SET NOCOUNT ON; - INSERT INTO orders (customer_id, status) VALUES (@CustomerId, 'pending'); -END +-- db/sql/02_views/02_order_views.sql +CREATE OR ALTER VIEW open_orders AS +SELECT party_id, order_no FROM sales_order WHERE status = 'open'; GO -CREATE PROCEDURE checkout_commit - @OrderId INT -AS -BEGIN - SET NOCOUNT ON; - UPDATE orders SET status = 'paid' WHERE id = @OrderId; -END -GO - -CREATE PROCEDURE checkout_cancel - @OrderId INT -AS -BEGIN - SET NOCOUNT ON; - UPDATE orders SET status = 'cancelled' WHERE id = @OrderId; -END +CREATE OR ALTER VIEW invoiced_orders AS +SELECT party_id, order_no FROM sales_order WHERE status = 'invoiced'; GO ``` +When a batch fails, the error starts with `[batch 2 of 2]`. SQLite errors start with `[statement N of M]`. Files in `db/changes/` split the same way. Splitter rules are in the [runner internals](/dev/runner#mssql-batch-handling). -Three related procedures, one file. If batch 2 fails, noorm reports `[batch 2 of 3]` in the error and skips batch 3 — so you know exactly which `CREATE PROCEDURE` blew up without re-reading the file in your head. - - -`GO` recognition is line-oriented: the literal token `GO` must be the entire trimmed content of a line (case-insensitive). `GO;`, `GOLANG`, or `GO` mid-line do not split. Keep `GO` tokens out of string literals and `/* ... */` block comments — the splitter does not parse SQL, so an inadvertent `GO` inside a string on its own line will still be treated as a separator. This matches sqlcmd behavior. - - -PostgreSQL runs a multi-statement file as-is, in one implicit transaction. SQLite gets its own splitter: its driver compiles the first statement of a string and discards the rest, so noorm splits on statement boundaries before handing anything over — semicolons inside string literals, quoted identifiers, comments, and trigger bodies are not treated as boundaries. MySQL's driver rejects multi-statement strings outright; put each statement in its own file, or use a dialect that supports them. - - -For the gory details on what the runner does under the hood, see [MSSQL Batch Handling](/dev/runner#mssql-batch-handling). - - -## Long-Running Files - -A file that is still running after 10 seconds gets a status report every 10 seconds until it finishes: elapsed time, the progress the database reports for the running command (an index build's phase and block count, for example), and any session it is waiting on for a lock. That tells a slow file apart from a stuck one. The TUI shows the report under the running file, and the log records a `Still running <file> (Ns)` line with the full report attached. - -The report comes from a second pooled connection, opened only when a file first passes 10 seconds and kept until the run ends. With `connection.pool.max: 1` there is no second connection to take, so reports carry elapsed time only. Behind a transaction-mode pooler (PgBouncer transaction mode, RDS Proxy, Supabase port 6543) the report can describe the wrong session; connect directly for builds you want to watch. - -To cancel a build from the TUI, see [Long-Running Files](/tui#long-running-files). +## What bites -## Summary -| Command | Purpose | -|---------|---------| -| `noorm run build` | Execute entire schema directory | -| `noorm run file <path>` | Execute single file | -| `noorm run dir <path>` | Execute files in directory | -| `--force` | Re-run regardless of changes | -| `--dry-run` | Render to `tmp/` without executing (`run build`, `run dir`) | +| Mistake or symptom | Cause | Fix | +|---|---|---| +| A build skips files after you dropped tables by hand | The records still say `success` and the checksums match | `noorm run build --force`. To empty a database, use `noorm db teardown`, which marks the records stale | +| `run build` fails with `already exists`, and the error ends with `1 prior successful execution` | A `CREATE TABLE` file changed, so it ran again against a table that exists | `run build` is for new databases. Bring an existing one up to date with `noorm change ff` | +| A template runs on every build | Its SQL changes on every render: `$.now()`, `$.uuid()` | Use a fixed value | +| The first build on a second config runs every file | Records are per config name | None needed. The config has not run them yet | +| Progress reports show only elapsed time, or describe another session | `connection.pool.max: 1` leaves no second connection. A transaction-mode pooler (PgBouncer, RDS Proxy, Supabase port 6543) can route the report to another session | Raise the pool max, and connect to the database directly for builds you want to watch | +| A line that is only `GO` inside a string or block comment splits the file | The splitter reads lines, not SQL | Keep `GO` out of strings and `/* */` comments | -The checksum system means you can run `noorm run build` as often as you want—only changed or new files execute. This is the foundation of noorm's approach to database schema management. +## Next -## What's Next? -- [Organization](/guide/sql-files/organization) - Structure your SQL files for predictable execution order -- [Templates](/guide/sql-files/templates) - Add dynamic content to `.sql.tmpl` files -- [Changes](/guide/changes/overview) - One-time operations with rollback support +- [Organization](/guide/sql-files/organization): folder layout and which files take part +- [Templates](/guide/sql-files/templates): generate SQL from data files +- [Forward and revert](/guide/changes/forward-revert): bring existing databases up to date +- [`noorm run`](/cli/run): exit codes and JSON output diff --git a/docs/guide/sql-files/organization.md b/docs/guide/sql-files/organization.md index c09aa5cc..9d9f5635 100644 --- a/docs/guide/sql-files/organization.md +++ b/docs/guide/sql-files/organization.md @@ -1,351 +1,119 @@ # Organizing SQL Files -How you structure your SQL files determines how they execute. noorm processes files in alphabetical order by path, so a well-organized directory structure gives you predictable, repeatable builds. +Name your folders and files so the order they sort in is the order they run in. - -## Recommended Directory Structure - -Start with this layout: +`noorm run build` finds every `.sql` and `.sql.tmpl` file under `paths.sql` (`./db/sql` after `noorm init`), sorts the paths as text, and runs them in that order. Every other file is ignored. ``` -sql/ +db/sql/ ├── 01_tables/ -│ ├── 001_users.sql -│ ├── 002_posts.sql -│ └── 003_comments.sql +│ ├── 01_party.sql # runs 1st +│ ├── 02_sales_order.sql # 2nd +│ ├── 03_product.sql # 3rd +│ ├── 04_order_line.sql # 4th +│ └── 05_app_setting.sql # 5th ├── 02_views/ -│ ├── 001_active_users.sql -│ └── 002_recent_posts.sql +│ └── 01_open_orders.sql # 6th ├── 03_functions/ -│ └── 001_calculate_score.sql -└── 04_seeds/ - └── 001_default_roles.sql +│ └── 01_place_order.sql # 7th +├── 09_seeds/ +│ ├── products.yml # data file, never runs +│ ├── 01_products.sql.tmpl # 8th +│ └── 02_gateway.sql.tmpl # 9th +└── archive/ # excluded in settings, never runs ``` -This structure works because: - -- **Numbered folder prefixes** - Guarantee execution order between folders (`01_` before `02_`) -- **Numbered file prefixes** - Guarantee execution order within each folder -- **Logical grouping** - Easy to find and maintain related files - +`09_seeds` leaves `04` to `08` free for groups you add later, such as procedures and triggers. -## Naming Conventions -noorm doesn't enforce naming rules, but consistent naming makes life easier. +## How are files ordered? -### Use Numeric Prefixes for Order +noorm compares whole paths as text, one character at a time. The folder comes first in the path, so a folder prefix orders the folders and a file prefix orders the files inside one. -Files execute alphabetically. Without prefixes, `accounts.sql` runs before `users.sql`—even if users should exist first. +| Names, in the order you want | Run order | Why | +|---|---|---| +| `tables/`, `views/`, `functions/` | `functions/`, `tables/`, `views/` | `f` sorts before `t` before `v` | +| `2_party.sql`, `10_order.sql` | `10_order.sql`, `2_party.sql` | `1` sorts before `2` as text | +| `accounts.sql`, `Users.sql` | `Users.sql`, `accounts.sql` | every capital letter sorts before every lowercase one | -``` -# Without prefixes (unpredictable) -accounts.sql # Runs first (a < u) -users.sql # Runs second - -# With prefixes (explicit order) -001_users.sql # Runs first -002_accounts.sql # Runs second -``` +Pad the prefixes in a folder to one width: `01_` through `99_`, or `001_` past that. Name each file for the object it creates. -Recommended formats: -| Format | Example | Best For | -|--------|---------|----------| -| `001_` | `001_users.sql` | Most projects (up to 999 files per folder) | -| `01_` | `01_users.sql` | Smaller projects (up to 99 files) | -| `0001_` | `0001_users.sql` | Large projects with many files | - - -### Use Descriptive Names - -The name should tell you what the file does without opening it: - -``` -# Good -001_create_users.sql -002_add_user_indexes.sql -003_create_posts.sql - -# Avoid -001.sql -002_update.sql -users_v2_final_FINAL.sql -``` +## Which files take part in a build? -### Template Files - -Files ending in `.sql.tmpl` are processed through the template engine before execution: - -``` -seeds/ -├── 001_default_roles.sql # Static SQL -└── 002_environment_config.sql.tmpl # Dynamic SQL with variables -``` - - -## Execution Order - -noorm walks your schema directory recursively, collects every `.sql` and `.sql.tmpl` file, and sorts the **full paths as strings**. That sorted list is the execution order. This is the critical rule for understanding how your schema executes. It means: - -1. The directory part of the path decides the order first -2. File names break ties inside the same directory -3. The comparison is by character code, not dictionary order, so every uppercase letter sorts before every lowercase one: `Users.sql` runs before `accounts.sql` - -Without numeric folder prefixes, alphabetical order causes problems: - -``` -sql/ -├── functions/ # f comes before t—runs first! -│ └── 001_helpers.sql -├── tables/ # t comes after f—runs second -│ ├── 001_users.sql -│ └── 002_posts.sql -└── views/ # v is last - └── 001_summary.sql -``` - -This is wrong—functions often depend on tables. Use numeric prefixes on folders: +A file runs unless an exclude entry covers it, or an include list exists and does not cover it: +```mermaid +flowchart TD + A["db/sql/<br/>.sql and .sql.tmpl files"] --> B{"under a<br/>build.exclude<br/>entry?"} + B -->|yes| X[skipped] + B -->|no| C{"build.include<br/>empty?"} + C -->|yes| R[runs] + C -->|no| D{"under a<br/>build.include<br/>entry?"} + D -->|yes| R + D -->|no| X ``` -sql/ -├── 01_tables/ # runs first -│ ├── 001_users.sql -│ └── 002_posts.sql -├── 02_views/ # runs second -│ └── 001_summary.sql -└── 03_functions/ # runs third (can reference tables/views) - └── 001_helpers.sql -``` - -Execution order: -1. `sql/01_tables/001_users.sql` -2. `sql/01_tables/002_posts.sql` -3. `sql/02_views/001_summary.sql` -4. `sql/03_functions/001_helpers.sql` - -## Using Settings for Build Order - -The `settings.yml` file lets you control which folders run and in what order: +An entry is a folder or one file, relative to `paths.sql`. Both keys filter and neither reorders. `exclude` wins over `include`. ```yaml +# .noorm/settings.yml build: - include: - - 01_tables - - 02_views - - 03_functions - - 04_seeds exclude: - - archive # Never runs -``` - -Folder names are relative to your `paths.sql` directory, not the project root. With the default `paths.sql: ./sql`, an entry of `01_tables` means `./sql/01_tables` — so you do **not** repeat the `sql/` prefix. - -::: warning A wrong include path still exits 0 -`sql/01_tables` resolves to `./sql/sql/01_tables` and matches nothing. `noorm run build` names the offending entries in a warning: - -``` -Ignored 1 build.include entry that matched no files: sql/01_tables -Include paths are relative to paths.sql — use `01_tables`, not `sql/01_tables`. + - archive ``` -The build still reports success and exits `0`, so a pipeline that only checks the exit code will not catch it. Read the warnings. -A mistyped `exclude` entry gets the same treatment and is the more dangerous half: nothing is excluded, so the files you meant to fence off run against the target database. +## Can a config change which files run? -A leading `./` or a trailing `/` matches nothing for the same reason. -::: -The `include` array is a **filter**—it controls which folders are included, not their order. Execution order is always alphanumeric. Use numeric prefixes to control the sequence: +A rule adds include or exclude entries when the active config matches. Rules run in order, every key in `match` must hold, and a later rule overrides an earlier one for the same entry. -``` -sql/ -├── 01_tables/ ← runs first -├── 02_views/ ← runs second -├── 03_functions/ ← runs third -├── 04_seeds/ ← runs last -└── archive/ ← excluded, never runs -``` - -This keeps the filesystem self-documenting. Anyone can see the execution order without consulting settings. - - -### Conditional Includes with Rules - -Different environments often need different files. Use rules to include or exclude folders based on config properties: +Suppose test configs have no `PAYMENT_GATEWAY_KEY`. A template that reads a missing secret throws, so this rule keeps the gateway seed out of test builds: ```yaml +# .noorm/settings.yml rules: - # Only run seeds in test environments - - match: + - description: Test configs have no gateway key + match: isTest: true - include: - - 04_seeds - - 05_test-fixtures - - # Skip heavy seeds for CI - - match: - name: ci-test exclude: - - 04_seeds/003_large-dataset.sql + - 09_seeds/02_gateway.sql.tmpl ``` -This means: -- Production builds skip test data entirely -- CI builds skip slow seed files -- Development gets everything - -::: warning A rule `include` narrows the whole build -An empty `build.include` means "every folder under `paths.sql`". A matching rule does not add to that set, it replaces it: its `include` entries are merged into `build.include`, and the moment that list is non-empty only the listed folders run. +| `match` key | Holds when | +|---|---| +| `name` | the config's name equals the value | +| `isTest` | the config's `isTest` flag equals the value | +| `type` | the config's type is `local` or `remote` | +| `protected` | `true` when the config's `user` role is below `admin` | -So on a project with no `build.include`, the rule above turns a test build into `04_seeds` and `05_test-fixtures` only. Your tables never get created. If you use rule `include` at all, list every folder you want in `build.include` too, and use rule `exclude` for the conditional part. -::: +Matching rules are merged into `build.include` and `build.exclude` before the build filters its files. `noorm run build --dry-run --config test` lists the files that config would run, and writes them rendered to `tmp/`. -## Organizing by Feature vs by Type +## Can I group by domain instead of type? -There are two common approaches to organizing SQL files. +Yes. Number the domain folders by dependency (`01_parties/`, `02_sales/`, `03_billing/`), and number `01_tables/` and `02_views/` inside each one. A view in `01_parties` runs before any `02_sales` table exists, so objects that reach across domains are easier to order by type. -### By Type (Recommended for Most Projects) -Group files by what they are: - -``` -sql/ -├── 01_tables/ -├── 02_views/ -├── 03_functions/ -├── 04_procedures/ -└── 05_seeds/ -``` +## What bites -**Pros:** -- Clear execution order from numeric prefixes -- Filesystem is self-documenting -- Matches how databases organize objects -**Cons:** -- Related files are spread across folders - - -### By Domain (For Projects with SDK) - -If you're building a [typed SDK](/getting-started/building-your-sdk), organize SQL to mirror your domain classes: - -``` -sql/ -├── 01_core/ -│ ├── tables/ # Base entities (users, accounts) -│ └── functions/ # Shared utilities -├── 02_auth/ -│ ├── tables/ # Sessions, tokens, permissions -│ ├── views/ # Active sessions, user permissions -│ └── functions/ # Authentication helpers -├── 03_billing/ -│ ├── tables/ # Invoices, payments, subscriptions -│ ├── views/ # Revenue reports, aging -│ └── procedures/ # Billing transactions -└── 04_seeds/ - └── defaults.sql # Reference data -``` - -This aligns your database layer with your application layer. The SDK's `auth` domain class maps to `sql/02_auth/`, making it easy to find related code. - -**Pros:** -- SQL and SDK stay in sync -- Clear ownership boundaries -- Domain changes are localized - -**Cons:** -- Need numbered folders to control cross-domain dependencies -- More complex include configuration - - -### Hybrid Approach - -Start simple and add domains as needed: - -``` -sql/ -├── 01_tables/ # All tables (simple start) -├── 02_views/ -├── 03_functions/ -└── 04_seeds/ -``` - -When a domain grows complex, extract it: - -``` -sql/ -├── 01_core/ # Moved tables, views, functions -├── 02_billing/ # Extracted billing domain -├── 03_functions/ # Shared functions -└── 04_seeds/ -``` - - -## Common Patterns - - -### Environment-Specific Files - -Use rules to include environment-specific folders: - -``` -sql/ -├── 01_tables/ -├── 02_views/ -├── 03_seeds-dev/ # Development data -├── 03_seeds-staging/ # Staging data -└── 03_seeds-prod/ # Production defaults only -``` - -```yaml -rules: - - match: - name: dev - include: - - 03_seeds-dev - - - match: - name: staging - include: - - 03_seeds-staging - - - match: - name: prod - include: - - 03_seeds-prod -``` - - -### Archive Pattern - -Keep old files without running them: - -``` -sql/ -├── 01_tables/ -├── 02_views/ -└── archive/ # Old files, excluded from builds - └── deprecated_table.sql -``` - -```yaml -build: - exclude: - - archive -``` +| Mistake or symptom | Cause | Fix | +|---|---|---| +| The build reports `success` with `filesRun=0`, or a file you excluded ran | An entry matched nothing. `sql/01_tables`, `./01_tables`, and `01_tables/` all match nothing, because entries start inside `paths.sql` | Write `01_tables`. The build warns `Ignored 1 build.include entry that matched no files` and still exits `0`, so read the warnings in CI | +| A test build runs the seeds but creates no tables | A matching rule's `include` joins an empty `build.include`, so the list is no longer empty and only the rule's folders run | List every folder in `build.include`, and use rule `exclude` for the conditional part | +| `run dir` runs a file that `build.exclude` lists | Only `run build` applies include, exclude, and rules | Name the files you want, or use `run build` | -## What's Next? +## Next -Now that your files are organized: -- [Templates](/guide/sql-files/templates) - Add dynamic content to your SQL -- [Execution](/guide/sql-files/execution) - Understand builds and change detection -- [Settings](/guide/environments/stages) - Configure build order with settings.yml +- [Execution](/guide/sql-files/execution): what a build skips, runs, and records +- [Templates](/guide/sql-files/templates): generate seeds from data files +- [Concepts](/getting-started/concepts): how `db/sql/` and `db/changes/` fit together +- [Settings reference](/dev/settings): every key in `settings.yml` diff --git a/docs/guide/sql-files/templates.md b/docs/guide/sql-files/templates.md index ed2a8e68..fed67e62 100644 --- a/docs/guide/sql-files/templates.md +++ b/docs/guide/sql-files/templates.md @@ -1,508 +1,210 @@ # SQL Templates -Static SQL handles most database work. But sometimes you need SQL generated from structured data: +Generate repeated SQL from data files, config, and secrets instead of writing every statement by hand. -- Seeding reference data from CSV or JSON files -- Generating similar objects (triggers, policies, grants) from configuration -- Inserting secrets like API keys into config tables -- Adapting SQL based on environment or stage +A data file sits beside the template, and its name becomes a variable: -Custom scripts work, but then you're maintaining two systems. Templates keep everything in one place: structured data files plus SQL templates that read them. - -Any file ending in `.sql.tmpl` is processed before execution. Templates look like SQL with embedded logic. They auto-load data files, inherit helpers, and have access to secrets and config. - - -## Basic Syntax - -Templates use [Eta](https://eta.js.org/) with custom delimiters designed to avoid conflicts with SQL syntax: - -| Syntax | Purpose | Example | -|--------|---------|---------| -| `{% %}` | JavaScript code block | `{% for (const x of $.items) { %}` | -| `{%~ %}` | Output a value | `{%~ user.name %}` | -| `-- {% %}` | Code block on a line of its own, disguised as a SQL comment | `-- {% for (const x of $.items) { %}` | -| `$` | Context variable | `$.config`, `$.secrets`, `$.uuid()` | - -A line whose entire content is `-- {% ... %}` gets its `-- ` prefix and its newline stripped before Eta parses the file, so the directive produces no output. Writing loops and conditionals that way keeps the `.sql.tmpl` file valid SQL, which means your editor still highlights it and a SQL linter still reads it. Inline tags and real `--` comments are untouched. - -Here's a simple example: - -```sql --- seed_roles.sql.tmpl -{% for (const role of $.roles) { %} -INSERT INTO roles (name) VALUES ('{%~ role %}'); -{% } %} -``` - -When rendered with `$.roles = ['admin', 'editor', 'viewer']`, this produces: - -```sql -INSERT INTO roles (name) VALUES ('admin'); -INSERT INTO roles (name) VALUES ('editor'); -INSERT INTO roles (name) VALUES ('viewer'); -``` - - -## How Rendering Works - -When noorm encounters a `.sql.tmpl` file: - -1. Scans the template's directory for data files (JSON5, YAML, CSV, `.dt`, and referenced JS/TS modules) -2. Walks up the directory tree collecting `$helpers.ts` files -3. Builds a context object (`$`) with helpers, data, config, and secrets -4. Renders the template with Eta -5. Returns the SQL string for execution - - -## The Template Context ($) - -Everything available in your template lives on the `$` object: - -| Property | Description | -|----------|-------------| -| `$.<datafile>` | Auto-loaded data from files in the same directory | -| `$.<helper>` | Functions from `$helpers.ts` files | -| `$.config` | Active database configuration | -| `$.secrets` | Decrypted secrets for active config | -| `$.globalSecrets` | Decrypted global secrets | -| `$.env` | `process.env` | - -Data files win the name: a `config.json` or `config.yml` sitting beside the template claims the `config` key, and `$.config` then holds that file instead of the active database configuration. Helpers lose to data files the same way, so keep helper exports and data file base names distinct. - - -## Built-in Helpers - -These helpers are always available in every template: - -### $.quote(value) - -Escapes a value and wraps it in single quotes. Use this for string values: - -```sql -INSERT INTO users (name) VALUES ({%~ $.quote(user.name) %}); --- Input: O'Reilly --- Output: INSERT INTO users (name) VALUES ('O''Reilly'); -``` - -`null` renders as the SQL keyword `NULL`. `undefined` throws, because no rendering of it is correct and the old behavior wrote the literal six-character string `undefined` into the database. Pass `null` when you mean SQL `NULL`. - -### $.escape(value) - -Escapes a string by doubling single quotes, without adding surrounding quotes: - -```sql -UPDATE users SET bio = '{%~ $.escape(user.bio) %}' WHERE id = 1; --- Input: It's great --- Output: UPDATE users SET bio = 'It''s great' WHERE id = 1; -``` - -### $.uuid() - -Generates a UUID v4: - -```sql -INSERT INTO tokens (id) VALUES ('{%~ $.uuid() %}'); --- Output: INSERT INTO tokens (id) VALUES ('550e8400-e29b-41d4-a716-446655440000'); -``` - -### $.now() - -Returns the current ISO timestamp: - -```sql -INSERT INTO logs (created_at) VALUES ('{%~ $.now() %}'); --- Output: INSERT INTO logs (created_at) VALUES ('2024-01-15T10:30:00.000Z'); +```yaml +# db/sql/09_seeds/products.yml +- { sku: HW-DONGLE, name: USB security key, list_price: 49.00 } +- { sku: SW-SEAT, name: "Desktop seat, yearly", list_price: 89.00 } ``` -### $.json(value) - -JSON stringifies a value: - ```sql -INSERT INTO config (data) VALUES ('{%~ $.escape($.json(settings)) %}'); +-- db/sql/09_seeds/01_products.sql.tmpl +-- {% for (const p of $.products) { %} +INSERT INTO product (sku, name, list_price) +VALUES ({%~ $.quote(p.sku) %}, {%~ $.quote(p.name) %}, {%~ p.list_price %}); +-- {% } %} ``` -### $.include(path) - -Includes another SQL file. The path is relative to the current template: - -```sql --- changes/2025-01-15-setup/change/001_setup.sql.tmpl - --- Load shared functions -{%~ await $.include('../lib/uuid_function.sql') %} - --- Load triggers -{%~ await $.include('../lib/audit_triggers.sql') %} +```bash +noorm run preview db/sql/09_seeds/01_products.sql.tmpl ``` -An included `.sql` file is inserted verbatim. An included `.sql.tmpl` file is rendered first, with the same config and secrets, and its output is inserted. - - -## Auto-Loading Data Files - -Data files in the same directory as the template are automatically loaded into `$`: - ``` -sql/users/ -├── 001_seed.sql.tmpl # Template -├── users.json # -> $.users -├── roles.yml # -> $.roles -└── seed-data.csv # -> $.seedData +INSERT INTO product (sku, name, list_price) +VALUES ('HW-DONGLE', 'USB security key', 49); +INSERT INTO product (sku, name, list_price) +VALUES ('SW-SEAT', 'Desktop seat, yearly', 89); ``` -File names are converted to camelCase: - -| Filename | Context Key | -|----------|-------------| -| `my-config.json` | `$.myConfig` | -| `seed_data.yml` | `$.seedData` | -| `API_KEYS.json` | `$.apiKeys` | - -Subdirectories are not scanned, and neither are `.sql` files (those are for `$.include()`), dotfiles, `$helpers.*`, or other `.tmpl` files. +Four inputs meet in one `$` object. The rendered SQL is what `run build` hashes and runs, and what `run preview` prints: - -### Executable Side-Cars Load Only When Referenced - -`.js`, `.mjs`, and `.ts` data files are an exception to auto-loading. Parsing a `.csv` cannot do anything, but importing a `.ts` runs whatever is in it. So noorm loads an executable side-car only when the template text actually names its key, as `$.key` or `$['key']`: - -```sql --- 001_seed.sql.tmpl, sitting next to build-rows.ts --- The mention of $.buildRows is what makes build-rows.ts load and run. -{% for (const row of $.buildRows) { %} -INSERT INTO rows (label) VALUES ({%~ $.quote(row.label) %}); -{% } %} +```mermaid +flowchart TD + D["products.yml<br/>(same folder)"] --> C["$ context"] + H["$helpers.ts<br/>(this folder and up)"] --> C + S["$.secrets<br/>(config, global, vault)"] --> C + G["$.config and $.env"] --> C + C --> T["01_products.sql.tmpl<br/>rendered by Eta"] + T --> Q[plain SQL] + Q --> B["run build<br/>hash, then skip or run"] + Q --> P["run preview<br/>print to stdout"] ``` -A `build-rows.ts` that no template mentions is skipped, including during `run preview`, `run inspect`, and `--dry-run`, the three commands whose whole point is not to act. The check is syntactic, so a key you reach some other way (a computed lookup, a helper that reads it) will not trigger the load. Reference it by name once and it works. +## What does the syntax look like? -### Naming Collisions -Different naming patterns can produce the same context key: +Templates use [Eta](https://eta.js.org/) with delimiters that do not collide with SQL. Output is never HTML-escaped. -| Filename | Context Key | -|----------|-------------| -| `user_roles.json` | `$.userRoles` | -| `user-roles.yaml` | `$.userRoles` | -| `user.roles.csv` | `$.userRoles` | +| Syntax | Does | Example | +|---|---|---| +| `{% %}` | runs JavaScript | `{% for (const p of $.products) { %}` | +| `{%~ %}` | writes a value into the SQL | `{%~ p.sku %}` | +| `-- {% %}` | a tag alone on a line, behind a SQL comment | `-- {% } %}` | +| `$` | the context object | `$.products` | -If two files resolve to the same key, one silently overwrites the other. Use distinct base names to avoid conflicts. +noorm removes the `-- ` and the line break from a line that holds only `-- ` and a tag, before Eta runs. The directive prints nothing, and the file stays valid SQL for your editor and linter. The result loses its leading and trailing blank lines. -### Supported Formats +## What can a template read? -| Extension | Notes | -|-----------|-------| -| `.json`, `.json5` | JSON5 supports comments and trailing commas | -| `.yaml`, `.yml` | Full YAML support | -| `.csv` | Parsed into array of row objects | -| `.dt`, `.dtz` | Exported database tables (see [Data Transfer](/guide/database/transfer)) | -| `.js`, `.mjs`, `.ts` | Module's default export or exports object (`.ts` requires Node >= 22.13) | +| Name | Holds | +|---|---| +| `$.<file>` | each data file in the template's folder, under its camelCased name | +| `$.<export>` | each export of the `$helpers` files from the template's folder up | +| `$.config` | the active config: `name`, `type`, `isTest`, `connection`, `access` | +| `$.secrets` | this config's secrets, merged with global and vault ones. A missing key throws | +| `$.globalSecrets` | global secrets only. A missing key reads as `undefined` | +| `$.env` | `process.env` | +| `$.quote`, `$.escape`, `$.json`, `$.uuid`, `$.now`, `$.include` | the built-ins | -### JSON5 Example - -JSON5 is easier to write by hand than strict JSON: - -```json5 -// users.json5 - Comments are allowed -{ - users: [ - { name: 'Alice', role: 'admin' }, - { name: 'Bob', role: 'user' }, // Trailing commas are fine - ] -} -``` - -### CSV Example - -Given `users.csv`: - -```csv -email,name,role -alice@example.com,Alice,admin -bob@example.com,Bob,user -``` - -Access in your template as an array of objects: +When names collide, the last wins: helpers, data files, then `$.config`, `$.secrets`, `$.globalSecrets`, `$.env`, and the built-ins. A data file named `config.*` is the exception: it replaces `$.config`. -```sql --- seed.sql.tmpl -{% for (const user of $.users) { %} -INSERT INTO users (email, name, role) VALUES - ({%~ $.quote(user.email) %}, {%~ $.quote(user.name) %}, {%~ $.quote(user.role) %}); -{% } %} -``` +## Where does data come from? -### .dt Files (Exported Database Tables) -`.dt` and `.dtz` files exported from [Data Transfer](/guide/database/transfer) can be used directly as seed data. The loader reads the embedded schema, maps positional values to column names, and exposes the data as an array of row objects—just like CSV. +noorm loads the data files in the template's own folder. It skips subfolders, dotfiles, `.sql` files, `$helpers` files, and other `.tmpl` files. `seed_data.yml` becomes `$.seedData`, and `my-config.json` becomes `$.myConfig`. -``` -sql/users/ -├── 001_seed.sql.tmpl # Template -└── users.dt # -> $.users (exported from another database) -``` +| Extension | `$.<file>` holds | +|---|---| +| `.json`, `.json5` | the parsed value. JSON5 allows comments and trailing commas | +| `.yaml`, `.yml` | the parsed value | +| `.csv` | an array of row objects keyed by the header row. Every value is a string | +| `.dt`, `.dtz` | an array of row objects from a [data transfer](/guide/database/transfer) export. `.dtzx` is not supported | +| `.js`, `.mjs`, `.ts` | the module's default export, or its exports | -```sql --- seed.sql.tmpl -{% for (const user of $.users) { %} -INSERT INTO users (id, email, name) VALUES - ({%~ user.id %}, {%~ $.quote(user.email) %}, {%~ $.quote(user.name) %}); -{% } %} -``` +A `.js`, `.mjs`, or `.ts` file runs code, so noorm loads it only when the template names its key, as `$.buildRows` or `$['buildRows']`. -This lets you export data from one environment and use it as seed data in templates—without converting formats. Compressed `.dtz` files work too. Encrypted `.dtzx` files are not supported as template data since there's no way to provide a passphrase in the template context. +## How do helpers work? -## Helper Inheritance -Helper files named `$helpers.ts` (or `$helpers.js`) are automatically loaded and inherited up the directory tree: +A `$helpers.ts`, `$helpers.js`, or `$helpers.mjs` file exports functions and values for every template in its folder and below. The nearest file wins a name: ``` -sql/ -├── $helpers.ts # Project-wide helpers -├── users/ -│ ├── $helpers.ts # Overrides/extends parent -│ ├── 001_create.sql.tmpl # Has access to both -│ └── roles.json -└── products/ - └── 001_create.sql.tmpl # Only has project-wide helpers +db/sql/ +├── $helpers.ts # price() for every template +└── 09_seeds/ + ├── $helpers.ts # its own price() wins here + ├── products.yml + └── 01_products.sql.tmpl # sees both files ``` -Child helpers override parent helpers with the same name. This lets you define project-wide utilities at the root and specialize them in subdirectories. - +```ts +// db/sql/$helpers.ts +export function price(amount: number): string { -### Writing Helpers - -Helper files can be written in TypeScript (`.ts`) or JavaScript (`.js`, `.mjs`). TypeScript helpers require Node >= 22.13, which supports type stripping natively. On older Node versions, use `.js` instead. - -```typescript -// sql/$helpers.ts -export function padId(id: number, length = 6): string { - - return String(id).padStart(length, '0') -} - -export function formatDate(date: Date): string { - - return date.toISOString().split('T')[0] -} - -export const defaults = { - pageSize: 50, - maxRetries: 3, + return amount.toFixed(2); } ``` -Use in templates: +`{%~ $.price(p.list_price) %}` now writes `49.00`. The search for helper files stops at the project root, not at `paths.sql`. -```sql -INSERT INTO users (id, created_at) VALUES - ('{%~ $.padId(1) %}', '{%~ $.formatDate(new Date()) %}'); +`noorm run inspect` lists what a template can read, without rendering it: --- Output: INSERT INTO users (id, created_at) VALUES ('000001', '2024-01-15'); +```bash +noorm run inspect db/sql/09_seeds/01_products.sql.tmpl ``` - -## Common Patterns - - -### Seeding from Data Files - -```yaml -# roles.yml -- name: admin - permissions: ["read", "write", "delete"] -- name: editor - permissions: ["read", "write"] -- name: viewer - permissions: ["read"] ``` +Template: db/sql/09_seeds/01_products.sql.tmpl +Config: dev -```sql --- seed_roles.sql.tmpl -{% for (const role of $.roles) { %} -INSERT INTO roles (name, permissions) VALUES - ({%~ $.quote(role.name) %}, {%~ $.quote($.json(role.permissions)) %}); -{% } %} -``` +Data Files: + $.products Array [2] +Helpers ($helpers): + $.price Function -### Environment-Aware SQL - -The active configuration is available through `$.config`. Use it to adapt SQL based on stage, config name, or database: - -```sql --- setup_monitoring.sql.tmpl -{% const db = $.config.connection.database %} -{% const stage = $.config.stage %} - --- Tag all entries with the source database -INSERT INTO monitoring.activity_log (source_db, event, created_at) VALUES - ({%~ $.quote(db) %}, 'schema_deployed', '{%~ $.now() %}'); - -{% if (stage === 'prod') { %} --- Only production stage gets strict constraints -ALTER TABLE orders ADD CONSTRAINT chk_amount CHECK (amount > 0); -{% } %} - -{% if ($.config.isTest) { %} --- Test databases get sample data -INSERT INTO users (name, email) VALUES ('Test User', 'test@example.com'); -{% } %} -``` - -Common conditions: - -| Condition | Use Case | -|-----------|----------| -| `$.config.stage === 'prod'` | Production-only constraints or settings | -| `$.config.isTest` | Test fixtures and sample data | -| `$.config.name === 'dev'` | Local development overrides | -| `$.config.connection.database` | Database-specific configuration | - - -### Using Secrets - -Never hardcode credentials. Access them through `$.secrets`: - -```sql --- config_setup.sql.tmpl -INSERT INTO app_config (key, value) VALUES - ('stripe_key', {%~ $.quote($.secrets.STRIPE_KEY) %}), - ('api_endpoint', {%~ $.quote($.env.API_URL || 'https://api.example.com') %}); +Built-ins: include, escape, quote, json, now, uuid +Secrets: 1 resolved, 0 global ``` -Secrets resolve through a three-tier hierarchy: config-specific local secrets override global local secrets, which override team-shared [vault](/guide/environments/vault) secrets. This means you can store production credentials in the vault and override them locally for development without affecting teammates. See the [secret resolution hierarchy](/guide/environments/vault#secret-resolution-hierarchy) for details. - -A key that isn't set in any of the three tiers has no correct SQL rendering, so reading it -throws instead of quietly resolving to `undefined`: -```sql --- Throws: Secret "STRIPE_KEY" not found (searched: config-local, global-local, vault) -{%~ $.secrets.STRIPE_KEY %} -``` +## What do the built-ins do? -This is deliberate — a missing secret used to render as the literal six-character string -`undefined`, which is valid SQL and easy to miss in a review. `$.secrets.KEY ?? fallback` -can't paper over a missing key either, because `??` still evaluates the throwing read on its -left side. To probe for a secret that's genuinely optional, check for it first with `in`, -which never throws: - -```sql -{% const hasKey = 'STRIPE_KEY' in $.secrets; %} -{%~ hasKey ? $.quote($.secrets.STRIPE_KEY) : 'NULL' %} -``` -`in` before read is how a template says "this secret is allowed to be missing" on purpose, -rather than by accident. See -[`examples/todo-db/sql/10_seeds/feature_flags.sql.tmpl`](https://github.com/noormdev/noorm/blob/master/examples/todo-db/sql/10_seeds/feature_flags.sql.tmpl) -for a full example that forces a feature flag off when its backing secret isn't configured. +| Call | Returns | +|---|---| +| `$.quote(v)` | `v` as a SQL string literal: single quotes around it, quotes inside doubled. `null` gives `NULL`, `undefined` throws, and a number is quoted too (`'49'`) | +| `$.escape(s)` | `s` with single quotes doubled, and no surrounding quotes | +| `$.json(v)` | `JSON.stringify(v)` | +| `$.uuid()` | a UUID v4 | +| `$.now()` | the current time as an ISO 8601 string | +| `await $.include(path)` | the file at `path`, relative to the template. A `.sql` file is inserted as is. A `.sql.tmpl` file is rendered with the same context first. The path cannot leave the project root | -### Generating Multiple Similar Objects +## How do secrets and config reach a template? -When you have many similar tables, views, or procedures: -```yaml -# audit_tables.yml -tables: - - users - - posts - - comments - - orders -``` +Read them off `$`. This template writes a secret and picks a URL by config name: ```sql --- create_audit_triggers.sql.tmpl -{% for (const table of $.auditTables.tables) { %} -CREATE TRIGGER trg_{%~ table %}_audit -AFTER INSERT OR UPDATE OR DELETE ON {%~ table %} -FOR EACH ROW EXECUTE FUNCTION audit_log_changes(); - -{% } %} +-- db/sql/09_seeds/02_gateway.sql.tmpl +INSERT INTO app_setting (name, value) VALUES + ('payment_gateway_key', {%~ $.quote($.secrets.PAYMENT_GATEWAY_KEY) %}), + ('payment_gateway_url', {%~ $.quote($.config.name === 'prod' ? 'https://api.gateway.example' : 'https://sandbox.gateway.example') %}); ``` +On a config named `dev` with the secret set, `run preview` prints: -### Role-Based Permissions - -Define permissions in YAML and generate all the grants: - -```yaml -# permissions.yml -roles: - WebApp: - views: [Users_V, Posts_V, Settings_V] - procs: [Login_trx, Signup_trx] - Worker: - views: [Queue_V, Jobs_V] - procs: [ProcessJob_trx] ``` - -```sql --- grant_permissions.sql.tmpl -{% for (const [role, perms] of Object.entries($.permissions.roles)) { %} --- Grants for {%~ role %} -{% for (const view of perms.views || []) { %} -GRANT SELECT ON {%~ view %} TO {%~ role %}; -{% } %} -{% for (const proc of perms.procs || []) { %} -GRANT EXECUTE ON {%~ proc %} TO {%~ role %}; -{% } %} - -{% } %} +INSERT INTO app_setting (name, value) VALUES + ('payment_gateway_key', 'gw_live_123'), + ('payment_gateway_url', 'https://sandbox.gateway.example'); ``` +[Secrets](/guide/environments/secrets#which-value-does-a-template-get) explains which of the three tiers supplies `$.secrets.PAYMENT_GATEWAY_KEY`. `$.config.isTest` and `$.config.connection.database` are the other keys templates branch on. -### Composing from Fragments - -Large table schemas are easier to review when split into logical groups. Use `$.include()` to compose them into a single change: - -``` -changes/2025-01-15-full-setup/ -├── change/ -│ └── 001_schema.sql.tmpl -└── lib/ - ├── core_tables.sql - ├── content_tables.sql - └── seed_defaults.sql -``` - -```sql --- changes/2025-01-15-full-setup/change/001_schema.sql.tmpl --- Core tables (users, roles, user_roles) -{%~ await $.include('../lib/core_tables.sql') %} +## How do I check what a template renders? --- Content tables (posts, comments, tags, post_tags) -{%~ await $.include('../lib/content_tables.sql') %} --- Default seed data -{%~ await $.include('../lib/seed_defaults.sql') %} -``` +`noorm run preview <file>` prints the rendered SQL to stdout, so `> rendered.sql` works. `--json` returns `{ filepath, sql, durationMs }`. Both `preview` and `inspect` take `--config` and `--json`, and a `viewer` role cannot use either. -Each fragment is plain SQL that could run on its own. The template just stitches them together so the change stays atomic while the source stays organized. +A change folder renders `.sql.tmpl` files the same way. [Changes](/guide/changes/overview) covers where they go. +::: warning Helpers and scripts run as you +A `$helpers` file runs whenever a template at or below its folder renders, and a referenced `.js` or `.ts` data file runs too. Both execute in the noorm process with your environment, including during `preview`. Read them before you run a template from a branch you did not write. +::: -## Security Notes -**SQL Injection**: Always use `$.quote()` or `$.escape()` for values from data files. Even trusted data files can contain characters that break SQL syntax. +## What bites -**Secret Exposure**: The rendered SQL may contain sensitive values from `$.secrets`. Don't log rendered templates in production. -**Code Execution**: `$helpers.ts` files execute arbitrary code whenever a template in or under their directory renders. `.js`, `.mjs`, and `.ts` data files execute only when a template names their key. Both run in the noorm process with your environment. Only use trusted sources. +| Mistake or symptom | Cause | Fix | +|---|---|---| +| `49.00` renders as `49` | YAML and JSON parse it as the number 49 | Quote it in the data file (`"49.00"`), or format it in a helper such as `$.price()` | +| `Secret "PAYMENT_GATEWAY_KEY" not found (searched: config-local, global-local, vault)` | No tier holds the key. `$.secrets.KEY ?? fallback` cannot catch it, because the read throws first | Set the secret, or test first: `'PAYMENT_GATEWAY_KEY' in $.secrets ? $.quote($.secrets.PAYMENT_GATEWAY_KEY) : 'NULL'` | +| `sqlQuote() received undefined` | A field the row lacks, a typo, or an unset `$.globalSecrets` key | Fix the name. Pass `null` when you mean SQL `NULL` | +| `$.config.name` is undefined, or holds a file's fields | A `config.yml` beside the template replaced `$.config` | Rename the data file | +| `[object Promise]` in the SQL | `$.include()` is async | Write `{%~ await $.include('...') %}` | +| A string with `'` breaks the statement | The value went in with `{%~ %}` alone, which writes raw text | Wrap every string from a data file in `$.quote()` | +| One of two data files is ignored | `user_roles.json`, `user-roles.yaml`, and `user.roles.csv` all become `$.userRoles`, and the last one loaded wins | Give each file a distinct base name | +| A `.ts` data file never loads, even in `preview` | The template does not name its key | Write `$.key` in the template | +| A `$helpers.ts` fails to load on Node | Node older than 22.13 cannot strip TypeScript types | Use `$helpers.js`. `run inspect` lists the failure under `Helper Errors:` | -**Path Traversal**: `$.include()` resolves relative paths and cannot escape the project root. +## Next -## What's Next? -- [Organization](/guide/sql-files/organization) - Control execution order with file naming -- [Secrets](/guide/environments/secrets) - Securely store credentials -- [Vault](/guide/environments/vault) - Team-shared secrets with resolution hierarchy -- [Changes](/guide/changes/overview) - Versioned changes with template support -- [Data Transfer](/guide/database/transfer) - Export/import .dt files for seeding templates +- [Secrets](/guide/environments/secrets): set the values a template reads +- [Organization](/guide/sql-files/organization): where templates sit in the build order +- [Execution](/guide/sql-files/execution): how a rendered template is hashed and skipped +- [Data transfer](/guide/database/transfer): export `.dt` files to use as seed data diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index f3949bbb..c92917f3 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -1,146 +1,42 @@ -# Troubleshooting common surprises +# Troubleshooting -A short list of the issues users hit most often when first reaching -for `noorm`. Each entry is a search target plus a one-paragraph fix -and a link to the deeper doc. +Find the message you see, read the cause, and apply the fix. Exit code `2` means the command never ran, `1` means it failed and a retry is safe, and `3` means some work landed ([exit codes](/headless#exit-codes)). -## `noorm --config prod run build` fails or targets the wrong directory +## Why does a command fail or surprise me? -`--config` (long form) is per-subcommand only — it has no global meaning. Neither do -`--dry-run`, `--json`, or `--yes`: the only flag with a root-level meaning is `-c`/`--cwd`. -Placed before the subcommand, `--config` (and everything else) is rejected outright rather -than silently ignored: - # rejected: --config has no global form - noorm --config prod run build +| Message or symptom | Cause | Fix | +|---|---|---| +| `Unrecognized flag '--config' before the subcommand` | Only `-c`/`--cwd` is a root flag | `noorm run build --config prod` | +| `Error: --cwd path is not a directory: …/prod` | In `noorm -c prod run build`, `-c` means `--cwd` | `noorm run build -c prod` | +| `Unknown command help` | There is no `noorm help` | `noorm db create --help` | +| `db create --name foo` creates nothing named `foo` | `db create` has no `--name`. The config names the database | `noorm config import dev-config.json`, then `noorm db create -c dev` | +| `Unknown command` from `noorm sql "…"` | The `sql query` rewrite needs the query to start with a SQL verb, so a leading comment defeats it | `noorm sql query "…"` | +| `Error: noorm init requires an interactive terminal.` | No terminal, no `--yes` | `noorm identity init …`, then `noorm init --yes` | +| `--dry-run` leaves files in `tmp/` | It renders SQL there with secrets resolved, and `tmp/` is not gitignored | Delete `tmp/`, or use `noorm run preview <path>` | - # works - noorm run build --config prod - noorm run build -c prod -`-c` is the sharper trap: `noorm -c prod run build` is *not* the same command — before the -subcommand, `-c` means `--cwd`, not `--config`. See [CLI flag conventions](../cli/flags.md) -for the full rule. +## Why does the database refuse or misbehave? -## `noorm help db create` errors with "Unknown command help" +Each server sends one error for a wrong password and an unknown account. noorm rewords it, and the server's own text is in `.noorm/state/noorm.log` (`connection:error`, fields `serverCode` and `serverMessage`). -There is no `noorm help <subcommand>` form — `noorm`'s CLI framework -(citty) doesn't ship one. Use `--help` (or `-h`) instead: +| Message | Also sent for | +|---|---| +| `Login failed for user …` (SQL Server) | Unknown login, login denied `CONNECT SQL`, Windows-only authentication | +| `Password authentication failed for user …` (PostgreSQL) | Unknown role, password past `VALID UNTIL` | +| `Access denied for user …` (MySQL) | Unknown user, no account for this client host, `REQUIRE SSL` account | - noorm db create --help - noorm change ff --help - noorm sql query --help +| Symptom | Cause | Fix | +|---|---|---| +| A right password fails from `NOORM_CONNECTION_PASSWORD` | The compiled binary does not read `.env` files | Export the variable, or set it in CI | +| An MSSQL procedure's `DEFAULT` is not applied | The SDK sends `undefined` as `NULL`, and MSSQL applies a `DEFAULT` only to an omitted parameter | `.default(value)` in the Zod schema, or omit the key ([details](/reference/sdk#parameter-handling-and-null-semantics)) | -See [Discovering command help](../cli/help.md). +## Next -## `noorm db create --name foo` doesn't work -`noorm db create` has no `--name` flag. The database name lives in the -stored config; create the config first, then point `db create` at it. -The intended workflow is: - - noorm config import dev-config.json - noorm db create -c dev - noorm config use dev - -See [Creating a database](./database/create.md) for the reasoning and -the full CI bootstrap. - - -## My MSSQL proc's `DEFAULT` value isn't applied - -A Zod schema with `.optional()` parses missing keys to `undefined`. -The SDK serializes `undefined` (and `null`) to SQL `NULL` when -building a named-parameter `EXEC`. MSSQL applies a parameter's -`DEFAULT` only when the parameter is *omitted* from the call, not when -it's explicitly `NULL`. Two fixes: - -1. Use `.default(<value>)` in your Zod schema so the parsed input - always carries a concrete value. -2. Or, build the params object without the key entirely (e.g. - conditionally `delete obj.key` before calling `ctx.proc(...)`). - The SDK only emits `@key = …` for keys present in the object. - -See [SDK reference — Parameter handling and NULL semantics](../reference/sdk.md#parameter-handling-and-null-semantics). - - -## `noorm sql "SELECT ..."` errors with "Unknown command" - -The bare `noorm sql "<SQL>"` form is supported, but only when the -first token of the query is one of the SQL verbs noorm rewrites on -(`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `WITH`, etc.). Citty resolves -`sql`'s subcommands first, so noorm rewrites `sql <verb> …` to -`sql query <verb> …` before citty ever sees it. For anything -that starts with a leading comment, a CTE-style preamble, or a -non-recognized verb, use the explicit subcommand: - - noorm sql query "<your SQL>" - noorm sql query -f path/to/file.sql - -See [`noorm sql`](../cli/sql.md) for the full list of recognized -keywords and when each form fires. - - -## `noorm change ff --dry-run` wrote files I didn't expect - -`change ff` does accept `--dry-run`, and it does not touch the -database. What it does do is *render* every pending change to `tmp/` -so you can read the SQL that would have run, including any secrets -the templates resolved, in plaintext. `tmp/` is not gitignored by a -project noorm scaffolds, so clean it up. - - noorm change ff --dry-run - noorm run build --dry-run - -To render a single template without the change machinery, use -`noorm run preview <path>`. From the SDK the same preview is -`changes.apply(name, { dryRun: true })` or -`changes.ff({ dryRun: true })`. - - -## `noorm init` errors in CI with "requires an interactive terminal" - -The bootstrap path needs an existing identity at -`~/.noorm/identity.{key,pub,json}`, then `--yes` (or `NOORM_YES=1`) -skips every other prompt. Two commands: - - noorm identity init --name "$CI_USER" --email "$CI_EMAIL" - noorm init --yes - -See [Non-interactive operation](./automation/non-interactive.md). - - -## Login fails with a password I know is right - -SQL Server, PostgreSQL, and MySQL each send one error for a wrong password and -an unknown account, so a client cannot learn which accounts exist. noorm says so -and lists what that error covers; the server's own log names the cause: - -| Server | Error | Also sent for | -|------------|----------------------------------|------------------------------------------------------------------------| -| SQL Server | `Login failed` (18456) | unknown login, login denied `CONNECT SQL`, Windows-only authentication | -| PostgreSQL | `password authentication failed` | unknown role, password past `VALID UNTIL` | -| MySQL | `Access denied` (1045) | unknown user, no account for this client host, `REQUIRE SSL` account | - -The message noorm shows is reworded for the user. What the server itself -sent is in the log (`.noorm/state/noorm.log`): the `connection:error` -entry carries `serverCode` (error number, SQLSTATE, or driver code) and -`serverMessage` (the server's text, including any wrapped socket error). - -A password that reaches noorm through a `NOORM_CONNECTION_PASSWORD` -environment variable arrives exactly as the shell exported it. The -compiled `noorm` binary does not read `.env` files, so export the -variable or set it in CI. - - -## Related - -- [CLI flag conventions](../cli/flags.md) -- [Discovering command help](../cli/help.md) -- [Creating a database](./database/create.md) -- [Non-interactive operation](./automation/non-interactive.md) -- [SDK reference](../reference/sdk.md) +- [CLI flag conventions](/cli/flags): where each flag goes +- [Non-interactive operation](/guide/automation/non-interactive): `--yes` and what still refuses diff --git a/docs/headless.md b/docs/headless.md index 7d7dbd2d..729d375c 100644 --- a/docs/headless.md +++ b/docs/headless.md @@ -1,479 +1,486 @@ # CLI Reference -Every `noorm` command runs as a non-interactive CLI by default. The Ink/React TUI lives behind a separate subcommand: +Run noorm from a script or CI job, read one JSON result, and branch on the exit code. + +With one pending change, `change ff` writes a single JSON document to stdout and its log to stderr. The exit code matches `success`: ```bash -noorm run build # CLI: build schema headlessly -noorm change ff --json # CLI: fast-forward with JSON output -noorm ui # TUI: launch the interactive terminal UI +noorm change ff --json > ff.json 2> ff.log; echo "exit $?" +jq -c '{success, status, executed, failed}' ff.json ``` -There is no mode detection, no `--headless` or `--tui` flag, no CI heuristic. If you want the wizard, run `noorm ui` explicitly. +```text +exit 0 +{"success":true,"status":"success","executed":1,"failed":0} +``` +`ui` and `sql repl` open the TUI, and `settings edit` and `settings secret` need a terminal. `init` needs a terminal or `--yes`. Every other command runs headless. -## Discovering Commands -Every command exposes native `--help` rendered by [citty](https://github.com/unjs/citty). The root help lists every subcommand; per-command help shows arguments, options, and examples. +## How a command runs -```bash -noorm --help # Top-level command list -noorm change --help # Subcommands of `change` -noorm change ff --help # Args, options, and examples for `change ff` -``` -For shell completion, run `noorm complete <shell>` and source the script it prints. Supported shells: `zsh`, `bash`, `fish`, `powershell`, `fig`. A bare `noorm complete` is the runtime completion hook, not a generator, and errors without `--`. +### Where do flags go? -## Global Flags +Only `-c`/`--cwd` is read before the subcommand. Every other flag goes after it. -A flag goes on the command that uses it. `-c`/`--cwd <path>` is the one exception: it's -recognized **before** the subcommand instead, because it's consumed before dispatch — it sets -the working directory everything else (config discovery, project root) resolves against. An -unrecognized or misplaced flag typed before the subcommand exits non-zero naming the correct -form, rather than being silently dropped. +| Position | Flags | Effect | +|---|---|---| +| Before the subcommand | `-c <path>`, `--cwd <path>`, `--cwd=<path>` | Run in `<path>` instead of walking up to the nearest `.noorm/`. A path that is not a directory exits 2 | +| Anywhere | `--help`, `-h` | Print help | +| As the only argument | `--version`, `-v` | Print the version | +| After the subcommand | all other flags | Per-command flags | -| Flag | Short | Description | -|------|-------|-------------| -| `--cwd <path>` | `-c <path>` | Run the subcommand inside `<path>`. Skips the walk-up that normally finds the nearest `.noorm/`. | +Any other flag before the subcommand exits 2 and names where to move it. ```bash -# Run the subcommand at packages/db, not the repo root -noorm -c packages/db run build - -# Initialize a nested project from elsewhere -noorm -c packages/db init - -# every other flag belongs on the subcommand -noorm run build --dry-run +noorm -c packages/db run build # --cwd: build inside packages/db +noorm run build -c staging # --config: build against the staging config ``` -`--cwd`/`-c` only has this global meaning *before* the subcommand — after it, `-c` is the -per-command `--config` alias instead (see the note below). +These flags recur across commands. Each command section lists the ones it accepts: +| Flag | Alias | Effect | +|---|---|---| +| `--config <name>` | `-c` | Use this stored config instead of the active one | +| `--json` | | Print one JSON document on stdout | +| `--dry-run` | | Preview without writing. `run` and `change` commands render to `tmp/` | +| `--force` | `-f` | `run` and `change`: skip the checksum check. `init`, `ci init`, `settings init`, `config import`, `vault cp`, `identity init`: overwrite what exists | +| `--yes` | `-y` | Confirm a command the config's role marks `confirm`. `NOORM_YES` does the same | -## Common Flags +Run `noorm <command> --help` for a command's flags and an EXAMPLES block. Shell completion is [`complete`](#complete). -These options are reused across most commands. Run `<command> --help` to see exactly which apply. -| Flag | Short | Description | -|------|-------|-------------| -| `--json` | — | Emit machine-readable JSON instead of human-friendly text | -| `--config` | `-c` | Use a specific stored configuration (overrides the active one) | -| `--force` | `-f` | Skip checksum / safety checks | -| `--yes` | `-y` | Skip confirmation prompts | -| `--dry-run` | — | Preview without executing | -| `--help` | `-h` | Show command help (citty native) | +### Which database does a command use? -> **Note:** `-c` is overloaded by position. `noorm -c <path> run` is the global cwd flag; `noorm run -c <name>` is the per-command config alias. The first non-flag token (the subcommand) is the boundary. Every flag other than `-c`/`--cwd` — `--config`, `--force`, `--json`, `--dry-run`, `--yes`, etc. — has no global meaning; it only works after the subcommand. See [CLI flag conventions](./cli/flags.md) for the full reasoning. For per-command help, see [Discovering command help](./cli/help.md). +A command resolves one config by name, or builds one from the environment when no name resolves: -## Configuration +```mermaid +flowchart TD + N{"Name from flag,<br/>env, or active?"} + N -->|"name, stored"| ST["Stored config"] + N -->|"name, not stored"| E1["Error: Config not found"] + N -->|"no name"| V{"Dialect and database<br/>set in environment?"} + V -->|yes| EO["Env-only config"] + V -->|no| E2["Error: no config<br/>available"] + ST --> R["Resolved config"] + EO --> R +``` +The resolved config merges three layers, and a later layer wins. Then a `settings.yml` stage named like the config, if it has `protected: true`, caps the config's access: -### Using a Stored Config +1. Built-in defaults +2. The stored config +3. `NOORM_*` environment variables -```bash -# Use active config -noorm run build +A `NOORM_CONNECTION_HOST` therefore overrides the host of a stored config. CLI flags select the config but never change its connection fields. -# Use specific config -noorm run build --config staging - -# Or via environment -export NOORM_CONFIG=staging -noorm run build -``` +### Environment-only mode -### Environment-Only Mode -No stored config needed. Set connection via environment variables: +Set `NOORM_CONNECTION_DIALECT` and `NOORM_CONNECTION_DATABASE` and skip stored configs. The config is named `__env__`. Host defaults to `localhost`. ```bash export NOORM_CONNECTION_DIALECT=postgres export NOORM_CONNECTION_HOST=db.example.com -export NOORM_CONNECTION_DATABASE=myapp +export NOORM_CONNECTION_DATABASE=app export NOORM_CONNECTION_USER=deploy export NOORM_CONNECTION_PASSWORD=$DB_PASSWORD - noorm run build ``` -**Minimum required:** -- `NOORM_CONNECTION_DIALECT` (postgres, mysql, sqlite, mssql) -- `NOORM_CONNECTION_DATABASE` +Env-only mode applies only when no name resolves. A runner with an active stored config, such as one set up by `ci init`, uses that config and layers the variables over it. A clean checkout has no `state.enc`, so env-only mode needs no identity. -### All Environment Variables +### What does a command print? -**Connection:** -```bash -NOORM_CONNECTION_DIALECT # postgres, mysql, sqlite, mssql -NOORM_CONNECTION_HOST # Database host -NOORM_CONNECTION_PORT # Database port -NOORM_CONNECTION_DATABASE # Database name -NOORM_CONNECTION_USER # Username -NOORM_CONNECTION_PASSWORD # Password -``` -**Identity (CI):** -```bash -NOORM_IDENTITY_PRIVATE_KEY # X25519 private key, hex PKCS8 DER (96 hex chars) -NOORM_IDENTITY_NAME # Display name (e.g. "CI Bot") -NOORM_IDENTITY_EMAIL # Email (e.g. "ci@example.com") -NOORM_CI_CONFIG_NAME # Default config name for `ci init` (override: --name) -``` +| Stream | Carries | +|---|---| +| stdout | The result: summary text, or with `--json` exactly one JSON document | +| stderr | Progress lines, warnings, log events (JSON lines under `--json`), and error diagnostics | +| exit code | The outcome. See [Exit codes](#exit-codes) | -**Paths:** -```bash -NOORM_PATHS_SQL # Schema directory (default: ./sql) -NOORM_PATHS_CHANGES # Changes directory (default: ./changes) -``` +Every `--json` payload is an object with a boolean `success`. Three rules hold for all commands: -**Behavior:** -```bash -NOORM_CONFIG # Config name to use -NOORM_YES # Skip confirmations (1 or true) -NOORM_DEBUG # Enable debug logging +1. `jq -e '.success'` works on the `--json` output of every command except `dev test-helpers`, which prints a text report first. +2. `success` is `true` if and only if the exit code is `0`. A partial result is `success: false` with `"status": "partial"` and exit `3`. +3. A failed payload carries `error` as a message string, never a stack trace. + +```json +{"success":false,"error":"Template not found: db/sql/nope.sql.tmpl"} ``` +A command with nothing to report still writes a result: `change list` on a project with no changes prints `No changes found.`, or `{"success":true,"changes":[],"pending":0}` under `--json`. Lists sit under a named key (`changes`, `tables`, `configs`), never at the top level. + -### Environment Variable Overrides +## Exit codes -All `NOORM_*` environment variables are processed through a nesting convention that maps underscores to object depth. This lets you override any config property from the environment without editing files. -**Meta variables** are handled separately and control CLI behavior rather than config values: +Defined in `src/cli/_exit.ts`. The split between `1` and `3` matters most in a pipeline. -| Variable | Purpose | -|----------|---------| -| `NOORM_CONFIG` | Select which stored config to use | -| `NOORM_YES` | Skip confirmations (`1` or `true`) | -| `NOORM_CHANNEL` | Force the policy channel: `user` or `agent` (default: detected from the environment) | +| Code | Name | Meaning | Remedy | +|---|---|---|---| +| `0` | success | Everything asked for happened | Continue | +| `1` | failure | The operation ran and no unit of work succeeded, or it failed before starting: connection error, policy denial, missing `--yes`, unreadable state | Read `error` or stderr, fix the cause, retry | +| `2` | usage | A usage error the command checks for itself: an invalid flag value, a named target that does not exist (directory, glob, config, change, secret or vault key), a TTY-only command run without a terminal, or `lock force` with nothing to release. Nothing was attempted. A missing required flag (reported by the argument parser), `run file` on a missing file, and `secret set` on a missing config exit `1` | Fix the invocation. Retrying unchanged fails the same way | +| `3` | partial | Some units succeeded and some failed. The target is in a mixed state | Do not retry blindly. Inspect first with `change list`, `change history-detail`, or `vault list`, then repair or revert | -These are excluded from config nesting and consumed directly by the CLI. +Commands that can exit `3`: -**Nesting rules** -- underscore separates object levels after stripping the `NOORM_` prefix: +- `run build`, `run dir`, `run exec`, `run files` +- `change ff`, `change next`, `change rewind` +- `db transfer` (database to database), `ci secrets`, `vault propagate` +- `dev test-helpers`, `dev test-workers` -``` -NOORM_CONNECTION_HOST → connection.host -NOORM_CONNECTION_PORT → connection.port -NOORM_PATHS_SQL → paths.sql -NOORM_PATHS_CHANGES → paths.changes -``` +Policy denials exit `1`. They have no dedicated code. -**Type conversion** -- string values are auto-converted: -- Numbers: `NOORM_CONNECTION_PORT=5432` becomes `5432` (number) -- Booleans: `NOORM_DEBUG=true` becomes `true` (boolean) -- Password fields skip conversion and remain strings regardless of content -**Priority order** (highest to lowest): -1. CLI flags (`--config`, `--json`, `--yes`) -2. Environment variables (`NOORM_*`) -3. Settings file (`.noorm/settings.yml`) -4. Config defaults +## Environment variables -## Access Roles +Names after the `NOORM_` prefix nest on underscores: `NOORM_CONNECTION_HOST` sets `connection.host`. Numeric values become numbers, `true` and `false` become booleans, and values whose name contains `password` stay strings. Other unreserved `NOORM_*` names overlay the matching key in `settings.yml`. -Each config carries a per-channel access grant: `access: { user, agent }`. The `user` role governs a human at the CLI, TUI, or SDK; `agent` governs an AI agent, over MCP or the CLI (see [MCP](./guide/automation/mcp.md)). Roles are fixed — `viewer`, `operator`, `admin` — and hard-coded to this matrix (cells: allow / confirm / deny): +| Variable | Effect | +|---|---| +| `NOORM_CONNECTION_DIALECT` | `postgres`, `mysql`, `sqlite`, or `mssql`. With `NOORM_CONNECTION_DATABASE`, the minimum for env-only mode | +| `NOORM_CONNECTION_DATABASE` | Database name | +| `NOORM_CONNECTION_HOST`, `_PORT`, `_USER`, `_PASSWORD`, `_FILENAME` | Server, credentials, and the SQLite file | +| `NOORM_CONFIG` | Stored config to use. `--config` wins | +| `NOORM_YES` | Any value except empty, `0`, or `false` (case-insensitive). Confirms every `confirm` cell on the `user` channel, like `--yes`. `identity init --force` and `sql clear` read only the literal flag | +| `NOORM_CHANNEL` | `user` or `agent`. Forces the access channel. Other values are ignored | +| `NOORM_IDENTITY_PRIVATE_KEY` | X25519 private key, hex PKCS8 DER, 96 characters | +| `NOORM_IDENTITY_NAME`, `NOORM_IDENTITY_EMAIL` | Display name and email. All three `NOORM_IDENTITY_*` variables must be valid or the CLI ignores them | +| `NOORM_IDENTITY` | Audit identity as `Name <email>`. Used for change and build records when no `NOORM_IDENTITY_*` set is present and the config has no `identity` override. Falls back to the git user, then the system user | +| `NOORM_CI_CONFIG_NAME` | Default config name for `ci init`. `--name` wins, then `ci` | +| `NOORM_PATHS_SQL`, `NOORM_PATHS_CHANGES` | Override `paths.sql` and `paths.changes`. Defaults are `./db/sql` and `./db/changes` | +| `NOORM_INSECURE` | Same as `noorm update --insecure` | +| `NOORM_DEBUG` | Truthy enables event tracing and connection debug output | +| `NOORM_DEV` | `true` raises the default log level from `info` to `verbose` | +| `NOORM_AGENT` | Marks the session as an agent. See [Which channel am I on?](#which-channel-am-i-on) | +| `NOORM_HEADLESS`, `NOORM_LOGGER_DEBUG`, `NOORM_JSON` | Reserved. Excluded from config nesting. The logger reads the first two. Use `--json` for JSON output | + + +## Access roles + + +Each config carries `access: { user, agent }`. The `user` role governs a human at the CLI, TUI, or SDK. The `agent` role governs an AI agent over MCP or the CLI. Roles are `viewer`, `operator`, and `admin`, and `agent` can also be `false`, which hides the config from agents. A config with no explicit access gets `{ user: admin, agent: viewer }`. A legacy `protected: true` maps to `{ user: operator, agent: viewer }`, and a stage with `protected: true` caps its configs at that pair. + + +### Which channel am I on? + + +noorm resolves the channel from who is driving, not from which binary ran: + +1. `NOORM_CHANNEL`, when set to exactly `user` or `agent`. +2. A non-empty harness marker variable: `CLAUDECODE`, `CLAUDE_CODE`, `CLAUDE_CODE_ENTRYPOINT` (Claude Code), `CODEX_SANDBOX` (OpenAI Codex), `CURSOR_AGENT` (Cursor), `GEMINI_CLI` (Gemini CLI), `AI_AGENT` or `NOORM_AGENT` (any other agent). The channel is `agent`. +3. Otherwise `user`. + +`noorm mcp serve` is always `agent`. `noorm info` reports the detected harness and the marker variables that triggered it, which is the first place to look when a denial has no visible cause. -| Command class | viewer | operator | admin | -|---|---|---|---| -| explore, `sql` reads | allow | allow | allow | -| `sql` writes (INSERT/UPDATE/DELETE) | deny | allow | allow | -| `sql` DDL (CREATE/ALTER/DROP/...) | deny | deny | allow | -| `change run`/`ff`/`revert`, `run build`/`file`/`dir` | deny | confirm | allow | -| `db create`, `db reset`, `db transfer --to` | deny | confirm | allow | -| `db truncate` | deny | confirm | confirm | -| `db teardown` | deny | deny | confirm | -| `db drop` | deny | deny | confirm | -| `config rm`, `config import --force` | deny | confirm | confirm | -| `change rm` | deny | confirm | confirm | -| `vault list`, `secret list`/`config export` | deny | allow | allow | -| `vault set`/`rm`, `secret set`/`rm` | deny | confirm | allow | -| `vault propagate` | deny | confirm | confirm | -| `lock force` | deny | confirm | confirm | -Raw SQL (`noorm sql`) is gated by what the statement actually does, not by a flag — a multi-statement input is classified by its highest class (a `SELECT` plus a `DROP` classifies as DDL). Unparseable or unrecognized statements classify as DDL, fail closed. +### What happens when a command is gated? -`confirm` means: type the phrase `yes-<config-name>` when prompted, or set `NOORM_YES=1` to skip the prompt in CI. There is no `--force` override for a denied permission — `--force` only skips file checksums (see [Common Flags](#common-flags)). -**Migration note:** the old `Config.protected: boolean` maps automatically on first load — `protected: true` becomes `{ user: 'operator', agent: 'viewer' }`, `protected: false` (or absent) becomes the default `{ user: 'admin', agent: 'viewer' }`. A config that already stores an explicit `access` keeps it untouched. The legacy field is still accepted on `config import` for one version, then dropped. +Each command maps to a permission. The config's role for the channel picks a cell: `allow`, `deny`, or `confirm`. + +```mermaid +flowchart TD + C{"Role's cell for<br/>the command"} + C -->|allow| RUN["Command runs"] + C -->|deny| FAIL["Exit 1 with reason"] + C -->|confirm| CH{"Channel"} + CH -->|agent| FAIL + CH -->|user| Y{"--yes or<br/>NOORM_YES?"} + Y -->|yes| RUN + Y -->|no| FAIL +``` + +On the `agent` channel a `confirm` cell is a deny, so `--yes` never unlocks it. There is no `--force` override for a denied cell: `--force` only skips checksums. The TUI asks you to type `yes-<config-name>` for the same cells. + +| Permission | Commands | viewer | operator | admin | +|---|---|---|---|---| +| `explore`, `sql:read` | `db explore`, `sql query` with `SELECT`, or `SHOW` and `DESCRIBE` on MySQL | allow | allow | allow | +| `sql:write` | `sql query` with `INSERT`, `UPDATE`, `DELETE`, `MERGE` | deny | allow | allow | +| `sql:ddl` | `sql query` with any other statement | deny | deny | allow | +| `change:run`, `change:revert` | `change run`, `ff`, `next`, `revert`, `rewind` | deny | confirm | allow | +| `run:build`, `run:file`, `run:dir` | `run build`, `file`, `dir`, `exec`, `files`, and `inspect`, `preview` against a resolved config | deny | confirm | allow | +| `db:create`, `db:reset` | `db create`, `db reset`, `db transfer --to`, `db transfer --import` | deny | confirm | allow | +| `db:truncate` | `db truncate` | deny | confirm | confirm | +| `db:teardown` | `db teardown` | deny | deny | confirm | +| `db:destroy` | `db drop` | deny | deny | confirm | +| `change:rm` | `change rm` | deny | confirm | confirm | +| `config:rm`, `config:write` | `config rm`, `config import --force` | deny | confirm | confirm | +| `vault:read`, `secret:read` | `vault list`, `secret list`, `config export` | deny | allow | allow | +| `vault:write`, `secret:write` | `vault init`, `set`, `rm`, `cp`, and `secret set`, `rm` | deny | confirm | allow | +| `vault:propagate` | `vault propagate`, `ci identity enroll` | deny | confirm | confirm | +| `transfer:plan` | `db transfer --to --dry-run` | deny | allow | allow | +| `lock:force` | `lock force` | deny | confirm | confirm | + +Commands not listed have no role gate. + +`sql query` classifies the statement, not a flag. Multiple statements take the highest class (`read` < `write` < `ddl`), `EXPLAIN` takes the class of what it wraps, and anything unparseable or unrecognized is `ddl`. + +Roles catch mistakes such as a teardown against the wrong database. They do not stop someone who holds the credentials and connects with another client. ## Commands -### Project Bootstrap +These commands never connect to a database: + +- `identity`, `settings`, `secret`, `version`, `mcp init` +- `ci init`, `ci secrets`, `ci identity new` +- `change add`, `change edit`, `change rm` +- `sql history`, `sql clear` +- `config`, except `use` and `validate` + +`run inspect` and `run preview` try one short connection to read vault secrets. + + +### Project + #### `init` -Bootstrap a new noorm project interactively. Creates identity (if needed), project structure (`sql/`, `changes/`, `.noorm/`), `settings.yml`, and encrypted `state.enc`. + +Create `.noorm/`, `settings.yml`, an encrypted `state.enc`, and a `# noorm` block in `.gitignore`. On the default layout, also create `db/sql/`, `db/changes/`, `db/AGENTS.md`, and `db/CLAUDE.md`. Create an identity first when none exists. + +``` +noorm init [--force] [--yes] [--here] +``` + +| Flag | Effect | +|---|---| +| `--force`, `-f` | Reinitialize a directory that already has `.noorm/`. Without it, exit 1. Existing `paths` in `settings.yml` are kept, so a project on `./sql` and `./changes` gets no `db/` folder. Existing `db/AGENTS.md` and `db/CLAUDE.md` are not overwritten | +| `--yes`, `-y` | Skip prompts. Needs an identity at `~/.noorm/identity.{key,pub,json}`, else exit 2 | +| `--here` | Initialize in the directory you launched from, ignoring a parent `.noorm/` | ```bash -noorm init # Interactive, requires TTY -noorm init --force # Reinitialize an existing .noorm/ -noorm init --yes # Non-interactive, requires existing identity +noorm identity init --name "CI Bot" --email ci@example.com +noorm init --yes ``` -**TTY required by default.** Fails with exit code 2 (usage error) in CI or when stdin is piped. For scripted bootstrap, set up the identity with `noorm identity init --name ... --email ...` then run `noorm init --yes` (or set `NOORM_YES=1`). For the full non-interactive matrix across all TTY-gated commands, see [Non-interactive operation](./guide/automation/non-interactive.md). For ephemeral CI runners, use `noorm ci init` instead. +Without a terminal and without `--yes` or `NOORM_YES`, `init` exits 2. Ephemeral runners use [`ci init`](#ci-init) instead. See [`noorm init`](/cli/init). -### Configuration +#### `ui` -#### `config list` -List available configurations. +Open the TUI at the home screen. Takes no flags. ```bash -noorm config list -noorm config list --json +noorm ui ``` -**JSON output:** -```json -{ - "success": true, - "configs": [ - {"name": "dev", "type": "local", "dialect": "postgres", "database": "app", "isActive": true, "isTest": false, "access": {"user": "admin", "agent": "viewer"}}, - {"name": "staging", "type": "remote", "dialect": "postgres", "database": "app", "isActive": false, "isTest": false, "access": {"user": "admin", "agent": "viewer"}} - ] -} -``` -Configs whose `access.agent` is `false` are omitted when the caller resolves to the `agent` channel — the same filtering the `list_configs` MCP command applies. +### Config -#### `config use <name>` +Local state in `.noorm/state/state.enc`. `config use` also registers your identity in that config's database when it can connect. -Set the active configuration. + +#### `config list` + + +List stored configs. ```bash -noorm config use dev -noorm config use production --json +noorm config list --json ``` -**JSON output:** -```json -{ - "success": true, - "activeConfig": "production" -} -``` +JSON: `configs[]` with `name`, `type`, `isTest`, `access`, `isActive`, `dialect`, `database`. Configs whose `access.agent` is `false` are omitted on the `agent` channel. -#### `config add` +#### `config use <name>` -Create a new configuration. -> Wizard-only. Launches the TUI wizard. To configure non-interactively, use `NOORM_*` environment variables or `config import`. Exits 1 (message on stderr) — there is no headless equivalent. +Set the active config. An unknown name exits 2. JSON: `activeConfig`. ```bash -noorm config add +noorm config use staging ``` -#### `config edit [name]` +#### `config add` -Edit an existing configuration. -> Wizard-only. Launches the TUI wizard. Use `NOORM_*` env vars or `config export`/`config import` to modify non-interactively. Exits 1 (message on stderr) — there is no headless equivalent. +Wizard only. Exits 1 with a pointer to `noorm ui` and `config import`. -```bash -noorm config edit dev -``` + +#### `config edit [name]` + + +Wizard only. Exits 1 with a pointer to `noorm ui` and `config import <file> --force --yes`. #### `config rm <name>` -Remove a configuration. + +Delete a stored config. ```bash noorm config rm staging --yes -noorm config rm staging --yes --json ``` -**JSON output:** -```json -{ - "success": true, - "name": "staging", - "deleted": true -} -``` +| Condition | Result | +|---|---| +| No `--yes` and no `NOORM_YES` | Exit 1. There is no confirmation prompt | +| Unknown name | Exit 2 | +| Config linked to a locked stage | Exit 1 until the stage is unlocked | +| Role denies `config:rm` | Exit 1 | -::: danger Destructive -Requires `--yes` (or `NOORM_YES=1`) — there is no TTY confirmation prompt, so the flag is mandatory even at an interactive terminal. An unknown config name fails with exit 2. Gated by the config's `config:rm` access (see [Access Roles](#access-roles) above); a config linked to a locked stage cannot be removed until the stage is unlocked. -::: +JSON: `name`, `deleted`. #### `config cp <src> <dest>` -Copy a configuration to a new name. + +Copy a config under a new name. Unknown `<src>` exits 2. An existing `<dest>` exits 1, and there is no `--force`. JSON: `src`, `dest`. ```bash noorm config cp dev staging -noorm config cp dev prod --json ``` -**JSON output:** -```json -{ - "success": true, - "src": "dev", - "dest": "staging" -} -``` - -A destination name that already exists fails with exit 1; `config cp` has no `--force`. - #### `config export <name>` -Export a configuration to JSON. Useful for sharing configs between machines or backing up before edits. + +Write a config as JSON, password included. ```bash -noorm config export dev -noorm config export dev --output ./dev-config.json +noorm config export dev > dev.json +noorm config export dev --output ./dev.json ``` -| Flag | Description | -|------|-------------| -| `-o, --output` | Write to file instead of stdout (mode `0600`) | +| Flag | Effect | +|---|---| +| `--output`, `-o` | Write to a file with mode `0600` instead of stdout | -Plain `config export dev` writes the bare config object — the exact shape `config import` reads — so `noorm config export dev > dev.json` round-trips. `--json` wraps it in the envelope instead (`{"success": true, "name": "dev", "config": {…}}`), which is not importable as-is. - -The export carries the connection password in plaintext, so it is gated by `secret:read`: a `viewer` is denied. +Plain output is the bare config object that `config import` reads, so redirecting round-trips. `--json` wraps it as `{success, name, config}`, which `config import` does not accept. Gated by `secret:read`. #### `config import <path>` -Import a configuration from a JSON file. + +Create a config from JSON. Unreadable or invalid JSON, or a missing `name` or `connection`, exits 2. ```bash -noorm config import ./dev-config.json -noorm config import ./staging-config.json --force --yes +noorm config import ./staging.json +noorm config import ./staging.json --force --yes ``` -| Flag | Description | -|------|-------------| -| `--force` / `-f` | Overwrite existing config with the same name | -| `--yes` / `-y` | Confirm the overwrite. Required with `--force`, because replacing a config rewrites its `access` roles — a `config:write` confirm cell for every role that holds it | +| Flag | Effect | +|---|---| +| `--force`, `-f` | Overwrite an existing config of the same name. Without it, exit 1 | +| `--yes`, `-y` | Confirm the overwrite. An overwrite rewrites `access`, so it is a `config:write` confirm cell | + +JSON: `name`, `overwritten`. #### `config validate <name>` -Validate that a configuration can connect to its database. -```bash -noorm config validate dev -noorm config validate production --json -``` +Test that a config connects. -**JSON output:** -```json -{ - "success": true, - "config": "dev", - "valid": true, - "checks": [ - {"label": "Connection", "status": "success", "detail": "Connection successful"}, - {"label": "Name", "status": "success", "detail": "dev"} - ] -} +```bash +noorm config validate prod --json ``` -Exit code 0 if valid, 1 if any check fails, 2 if the config name is unknown. +Exit 0 when valid, 1 when a check fails, 2 for an unknown name. JSON: `config`, `valid`, `checks[]` with `label`, `status`, `detail`. ### Identity -Manage your cryptographic identity. Identity is used for change attribution, lock ownership, and vault encryption. + +Your keypair lives at `~/.noorm/identity.{key,pub,json}`. It signs change history, owns locks, and encrypts state and the vault. + #### `identity init` -Create a new cryptographic identity. Generates an X25519 keypair and stores it at `~/.noorm/identity.{key,pub,json}`. + +Create an X25519 keypair. ```bash -noorm identity init --name "Alice" --email "alice@example.com" -noorm identity init --name "Alice" --email "alice@example.com" --force --yes -noorm identity init --name "Alice" --email "alice@example.com" --json +noorm identity init --name "Ana Ruiz" --email ana@example.com ``` -| Flag | Description | -|------|-------------| -| `--name` | Display name (required) | -| `--email` | Email address (required) | -| `--force` | Replace an existing identity. Requires `--yes` | -| `--yes` / `-y` | Confirm the replacement. The flag is read literally here, so an ambient `NOORM_YES` does not satisfy it | +| Flag | Effect | +|---|---| +| `--name` | Display name. Required | +| `--email` | Email address. Required | +| `--force` | Replace an existing identity. Needs `--yes` | +| `--yes`, `-y` | Confirm the replacement. Only the literal flag counts, so an ambient `NOORM_YES` does not | -::: danger Destructive -The state encryption key derives from the private key, so replacing an identity orphans every `state.enc` on the machine — configs, secrets, and database passwords become permanently undecryptable. Nothing re-encrypts existing state under the new key. The old keypair is backed up first, and its path lands in `backedUpTo` on the result; restoring `identity.key` from it is the only way back. -::: +An existing identity without `--force` exits 1. JSON: `name`, `email`, `fingerprint`, `publicKey`, and on replacement `backedUpTo` and `warning`. -**JSON output:** -```json -{ - "success": true, - "name": "Alice", - "email": "alice@example.com", - "fingerprint": "...", - "publicKey": "..." -} -``` +::: danger +Replacing an identity re-keys state encryption. Every `state.enc` on the machine becomes unreadable: configs, secrets, and database passwords. Nothing re-encrypts them under the new key. The old keypair is backed up first, and restoring `identity.key` from `backedUpTo` is the only way back. +::: #### `identity edit` -Update the display name or email on the existing identity. + +Change the display name or email. At least one is required, else exit 2. No identity exits 2. ```bash -noorm identity edit --name "Alice Cooper" -noorm identity edit --email "alice@newjob.com" -noorm identity edit --name "Alice" --email "alice@new.com" --json +noorm identity edit --email ana@newjob.com ``` -At least one of `--name` or `--email` must be provided. +The fingerprint derives from name and email, so editing changes it. Vault access was granted to the old fingerprint, and a teammate with access must run `vault propagate` again. JSON: `name`, `email`, `fingerprint`, `publicKey`, `previousFingerprint`, `fingerprintChanged`. #### `identity export` -Print your public key so teammates can add you to encrypted vaults. + +Print your public key and fingerprint for a teammate. ```bash -noorm identity export noorm identity export --json ``` +No identity exits 2. JSON: `name`, `email`, `fingerprint`, `publicKey`. + #### `identity list` -List all known users discovered from database syncs (the audit trail of who has touched shared state). + +List users discovered from database syncs. ```bash -noorm identity list noorm identity list --json ``` +JSON: `users[]` with `name`, `email`, `identityHash`, `machine`, `source`, `lastSeen`. + ### Settings -Manage project settings stored in `.noorm/settings.yml`. Settings control paths, build rules, stages, strict mode, logging, and teardown behavior. + +Project settings in `.noorm/settings.yml`. + #### `settings init` -Initialize `settings.yml` with defaults. + +Write `settings.yml` with defaults. An existing file exits 1 unless you pass `--force`. JSON: `path`. ```bash -noorm settings init -noorm settings init --force # Overwrite existing +noorm settings init --force ``` +`--force` keeps the `paths` already in the file. + #### `settings build` -Reload `settings.yml` and re-save it. Applies missing defaults and normalizes formatting. + +Reload and re-save `settings.yml` to apply missing defaults and normalize formatting. JSON: `path`. ```bash noorm settings build @@ -482,770 +489,613 @@ noorm settings build #### `settings edit` -Interactive editor for settings sections: paths, build, strict, logging, stages, rules, teardown. - -```bash -noorm settings edit # Requires TTY -``` -**TTY required.** `--yes` / `NOORM_YES` errors with a redirect hint — edit `settings.yml` directly. See [Non-interactive operation](./guide/automation/non-interactive.md). +Interactive editor for paths, build, strict, logging, stages, rules, and teardown. Exits 2 with a pointer to editing the file directly when you pass `--yes` or `NOORM_YES`, or when stdin is not a terminal. Run `noorm settings build` to validate a hand edit. See [`settings edit`](/cli/settings-edit). #### `settings secret` -Interactive editor for secret requirement declarations: which secrets are required for each stage. -```bash -noorm settings secret # Requires TTY -``` +Interactive editor for the secret requirements each stage declares, not their values. Exits 2 under `--yes` or without a terminal. Edit the `secrets:` section of `settings.yml` directly and set values with `secret set`. See [`settings secret`](/cli/settings-secret). + -**TTY required.** This manages requirement declarations, not secret values. Use `noorm secret set` or `noorm vault set` for values. `--yes` / `NOORM_YES` errors with a redirect hint pointing at direct YAML edits. +### Secrets -### Secrets (Config-Scoped) +Config-scoped secrets, encrypted in your `state.enc` and never shared. A template reads `$.secrets.KEY` from the first match of: this config's secret, a global secret, then the [vault](/guide/environments/vault). + +All three take `--config` and `--json`, and `set` and `rm` take `--yes`. With no `--config`, they use the active config. An unknown config exits 1. -Local secrets that override vault secrets for a specific config. These are stored on disk per-user (not shared with the team). #### `secret list` -List secret keys for a config. + +List key names, never values. JSON: `configName`, `keys`. ```bash -noorm secret list noorm secret list --config prod -noorm secret list --json ``` -#### `secret set <key> <value>` +#### `secret set <key> [value]` + -Set a local secret for a config. +Set a secret. `--stdin` reads the value from stdin so it stays out of the process table, shell history, and `set -x` traces. Pass the value or `--stdin`, not both. ```bash -noorm secret set API_KEY "sk-live-..." -noorm secret set DB_PASSWORD "secret123" --config prod -noorm secret set API_KEY "sk-live-..." --json +printf %s "$KEY" | noorm secret set PAYMENT_GATEWAY_KEY --stdin ``` +An `operator` role needs `--yes`. JSON: `configName`, `key`. + #### `secret rm <key>` -Remove a local secret from a config. + +Delete a secret. `--yes` or `NOORM_YES` is required. An unknown key exits 2. JSON: `configName`, `key`. ```bash -noorm secret rm OLD_KEY --yes noorm secret rm OLD_KEY --yes --config prod ``` -Requires `--yes` to prevent accidental deletion. +### CI provisioning -### CI/CD Provisioning -The `ci` namespace bridges the gap between developer machines (which have `.noorm/` state on disk) and CI runners (which start from scratch on every job). +A CI runner starts with no `.noorm/` and no identity on disk. `ci` bridges that. The full pipelines are in [CI/CD automation](/guide/automation/ci). + #### `ci init` -Bootstrap ephemeral `state.enc` from `NOORM_IDENTITY_*` + `NOORM_CONNECTION_*` env vars. Run inside the CI job. -```bash -noorm ci init -noorm ci init --name staging -noorm ci init --force # Overwrite existing state.enc -noorm ci init --json +Build `state.enc` from `NOORM_IDENTITY_*` and `NOORM_CONNECTION_*`, create a `remote` config with default access, and make it active. + +``` +noorm ci init [--name <name>] [--force] [--test] [--yes] [--json] ``` -| Flag | Description | -|------|-------------| -| `--name` | Config name (default: `$NOORM_CI_CONFIG_NAME` or `"ci"`) | -| `--force` | Overwrite existing state.enc | +| Flag | Effect | +|---|---| +| `--name` | Config name. Defaults to `$NOORM_CI_CONFIG_NAME`, then `ci` | +| `--force`, `-f` | Replace an existing `state.enc`. The old one is copied to `state.enc.bak-<timestamp>` first | +| `--test` | Mark the config `isTest: true`, which satisfies the SDK's `requireTest` guard | +| `--yes`, `-y` | Needed with `--force` on a terminal. Without a terminal, `--force` alone is enough | -Creates a config, marks it active, sets `isTest: true`, and leaves state on disk so later commands (`run build`, `change ff`, `ci secrets`) work normally. +```bash +noorm ci init --name prod +``` -Fails fast with exit 1 if required env vars are missing or malformed. +| Condition | Result | +|---|---| +| Missing or invalid `NOORM_IDENTITY_*`, missing dialect or database | Exit 2 | +| `state.enc` exists without `--force` | Exit 1 | -**JSON output:** -```json -{ - "success": true, - "identity": { - "name": "CI Bot", - "email": "ci@example.com", - "publicKey": "...", - "identityHash": "...", - "source": "env" - }, - "config": {"name": "ci", "dialect": "postgres", "database": "app", "isTest": true}, - "stateFile": "/path/to/.noorm/state/state.enc" -} -``` +JSON: `identity` (`name`, `email`, `publicKey`, `identityHash`, `source`), `config` (`name`, `dialect`, `database`, `isTest`), `stateFile`, and `backedUpTo` after a replacement. #### `ci secrets` -Batch-load secrets from a dotenv-style file into the active vault. Run after `ci init`. + +Load a dotenv file into the config's secrets. Run it after `ci init`. ```bash -noorm ci secrets --file ./ci-secrets.env noorm ci secrets --file ./ci-secrets.env --overwrite -noorm ci secrets --file ./ci-secrets.env --config prod --json ``` -| Flag | Description | -|------|-------------| -| `--file` | Path to dotenv file (required) | -| `--config` | Target config (default: active) | -| `--overwrite` | Replace existing keys | +| Flag | Effect | +|---|---| +| `--file` | Dotenv file. Required | +| `--config` | Target config. Defaults to the active one | +| `--overwrite` | Replace keys that exist. Without it, existing keys are skipped | -**File format:** `KEY=value` per line; blank lines and `#` comments ignored; surrounding quotes stripped. +The file takes `KEY=value` lines. Blank lines and `#` lines are ignored, the line splits on the first `=`, one matched pair of quotes is stripped, and keys match `[A-Za-z_][A-Za-z0-9_]*`. -**Exit codes:** `0` all loaded; `1` total failure; `2` bad invocation (unknown config, unreadable or malformed `--file`); `3` partial (some keys set, some errored). +| Exit | Meaning | +|---|---| +| `0` | Every key loaded or skipped | +| `1` | State unreadable, or every write failed | +| `2` | No active config, unknown config, or an unreadable or malformed file | +| `3` | Some keys set, some failed | -**JSON output:** -```json -{ - "success": true, - "config": "ci", - "set": 3, - "skipped": 0, - "errors": 0 -} -``` +JSON: `config`, `set`, `skipped`, `errors`, and the first five `errorDetails`. #### `ci identity new` -Generate a test-CI keypair locally. No database contact, no state written. Prints an env block to copy into CI secrets. + +Generate a CI keypair locally. No database contact and nothing written to disk. ```bash noorm ci identity new --name "CI Bot" --email ci@example.com -noorm ci identity new --name "CI Bot" --email ci@example.com --json ``` -| Flag | Description | -|------|-------------| -| `--name` | Display name (required) | -| `--email` | Email address (required) | - -The private key is shown once and never stored. Save it immediately. - -**JSON output:** -```json -{ - "name": "CI Bot", - "email": "ci@example.com", - "publicKey": "...", - "identityHash": "...", - "privateKey": "...", - "envBlock": { - "NOORM_IDENTITY_PRIVATE_KEY": "...", - "NOORM_IDENTITY_NAME": "CI Bot", - "NOORM_IDENTITY_EMAIL": "ci@example.com" - } -} -``` +`--name` and `--email` are required. The private key prints once. JSON: `name`, `email`, `publicKey`, `identityHash`, `privateKey`, `envBlock`. #### `ci identity enroll` -Register a CI identity in the target database and grant vault access. Run once by a developer who already has vault access. + +Register a CI identity in a config's database and grant it vault access. A developer with vault access runs it once per config. ```bash -noorm ci identity enroll --config prod --name "CI Bot" --email ci@example.com -noorm ci identity enroll --config prod --name "CI Bot" --email ci@example.com --public-key <hex> +noorm ci identity enroll --config prod --name "CI Bot" --email ci@example.com --yes ``` -| Flag | Description | -|------|-------------| -| `--config` | Target configuration (required) | -| `--name` | Display name (required) | -| `--email` | Email address (required) | -| `--public-key` | Pre-generated public key hex (air-gapped flow) | +| Flag | Effect | +|---|---| +| `--config`, `-c` | Target config. Required | +| `--name`, `--email` | Identity to enroll. Required | +| `--public-key` | Hex X25519 public key from `ci identity new`, 88 characters. Without it, enroll mints a keypair and prints the private key once | +| `--yes`, `-y` | Needed to grant access, which cannot be undone. Skipped when the identity already holds access | + +Re-running is safe for the same key. An identity already registered under the same hash with a different public key is refused. JSON: `name`, `email`, `publicKey`, `identityHash`, `enrolledIn`, `alreadyEnrolled`, and `privateKey` and `envBlock` when it minted the keypair. -Without `--public-key`, mints a keypair and returns the private key once. With `--public-key`, only enrolls the provided public half (for keys minted via `ci identity new`). -Idempotent on identityHash. Re-running ensures vault access. +### Schema runs -### Schema Operations +Executes SQL files against the database. A file whose checksum is unchanged is skipped unless `--force`. Every `run` command accepts `--config` and `--json`, and all but `inspect` and `preview` accept `--force` and `--dry-run`. `--dry-run` renders to `tmp/` and writes nothing to the database. + #### `run build` -Execute all SQL files in the schema directory. Uses checksums to skip unchanged files. -```bash -noorm run build -noorm run build --force # Skip checksums, run everything -noorm run build --dry-run # Preview without executing +Execute every SQL file under `paths.sql`, filtered by `build.include` and `build.exclude`. + +``` +noorm run build [--config <name>] [--dry-run] [--force] [--json] ``` -**JSON output:** -```json -{ - "success": true, - "status": "success", - "files": [ - {"filepath": "sql/01_tables/users.sql", "status": "success", "checksum": "…", "durationMs": 12} - ], - "filesRun": 2, - "filesSkipped": 1, - "filesFailed": 0, - "durationMs": 234 -} +```bash +noorm run build --dry-run ``` -**Exit codes:** `0` success, `1` complete failure, `3` partial failure (some files ran, some failed). +| Exit | Meaning | +|---|---| +| `0` | Every file ran or was skipped as unchanged | +| `1` | No file succeeded | +| `3` | Some files ran, some failed | + +JSON: `status`, `files[]` (`filepath`, `checksum`, `status`, `durationMs`, `outputPath` on a dry run), `filesRun`, `filesSkipped`, `filesFailed`, `durationMs`, `unmatchedInclude`, `unmatchedExclude`. An include or exclude entry that matches nothing warns and does not fail. Entries are relative to `paths.sql`, so write `01_tables`, not `db/sql/01_tables`. #### `run file <path>` -Execute a single SQL or `.sql.tmpl` file. -```bash -noorm run file sql/01_tables/001_users.sql -noorm run file seed.sql.tmpl -noorm run file sql/init.sql --json -``` +Execute one `.sql` or `.sql.tmpl` file. JSON is the file result: `filepath`, `status`, `checksum`, `durationMs`. -**JSON output:** -```json -{ - "success": true, - "filepath": "seed.sql", - "status": "success", - "checksum": "…", - "durationMs": 45 -} +```bash +noorm run file db/sql/01_tables/02_sales_order.sql ``` #### `run dir <path>` -Execute all SQL files in a directory in alphabetical order. + +Execute every SQL file in a directory in path order. ```bash -noorm run dir sql/01_tables/ -noorm run dir seeds/ -noorm run dir migrations/ --json +noorm run dir db/sql/01_tables/ ``` -A directory that does not exist, or that holds no SQL files, exits `2` — a directory with nothing in it is a mistyped path, not a successful no-op. +A missing directory, or one with no SQL files, exits 2. A directory with nothing in it is a mistyped path, not a successful no-op. #### `run exec <path>` -Batch-execute SQL files matching a glob pattern or directory. More flexible than `run dir`. + +Execute the files a glob or directory matches. A glob keeps only `.sql` and `.sql.tmpl` files, sorted. No matches exits 2. ```bash -noorm run exec sql/ -noorm run exec "sql/**/*.sql" -noorm run exec "seeds/*.sql" --force -noorm run exec migrations/ --dry-run +noorm run exec "db/sql/**/*.sql" --dry-run ``` -**Exit codes:** `0` success, `1` complete failure, `2` bad invocation, `3` partial failure. - #### `run files --paths <path,...>` -Execute multiple specific SQL files in order. + +Execute specific files in the order given. An empty list exits 1. ```bash -noorm run files --paths seed.sql,fixtures.sql -noorm run files --paths sql/001_tables.sql,sql/002_indexes.sql --json +noorm run files --paths db/sql/01_tables/01_party.sql,db/sql/01_tables/02_sales_order.sql ``` #### `run inspect <path>` -Inspect the template context for a `.sql.tmpl` file without executing. Shows data files, helpers, config, secrets, and built-in functions. + +Show what a `.sql.tmpl` file can see, without running it: data files, helpers, built-ins, config keys, and secret counts. ```bash -noorm run inspect sql/users/001_create.sql.tmpl -noorm run inspect sql/core/Crons.sql.tmpl --json +noorm run inspect db/sql/09_seeds/01_products.sql.tmpl --json ``` -**JSON output:** -```json -{ - "success": true, - "filepath": "sql/users/001_create.sql.tmpl", - "context": { - "dataFiles": [{"key": "roles", "type": "Array [3]"}], - "helpers": [{"key": "padId", "type": "Function"}], - "helperErrors": [], - "builtins": ["quote", "escape", "uuid", "now", "json", "include"], - "configKeys": ["name", "connection"], - "secretCount": 2, - "globalSecretCount": 0 - } -} -``` +JSON: `filepath`, `context` (`dataFiles`, `helpers`, `helperErrors`, `builtins`, `configKeys`, `secretCount`, `globalSecretCount`). A missing template exits 2. Building the context executes the template's helper scripts, so a resolved config must allow `run:file`. #### `run preview <path>` -Render a `.sql.tmpl` file and output the resulting SQL. Does **not** execute against the database. -```bash -noorm run preview sql/schema.sql.tmpl -noorm run preview sql/schema.sql.tmpl > rendered.sql -noorm run preview sql/seed.sql.tmpl --json -noorm run preview sql/migrations/002.sql.tmpl --config staging -``` +Render a `.sql.tmpl` file and print the SQL without running it. It tries a short, no-retry connection to read vault secrets. When the database is unreachable it renders with local secrets only and says so: stderr in text mode, `vaultProbeFailed` and `notice` in JSON. -**JSON output:** -```json -{ - "success": true, - "filepath": "sql/schema.sql.tmpl", - "sql": "CREATE TABLE ...", - "durationMs": 12 -} +```bash +noorm run preview db/sql/09_seeds/01_products.sql.tmpl > rendered.sql ``` -A missing template exits `2`. In text mode the rendered SQL is the only thing on stdout, so `noorm run preview x.sql.tmpl > rendered.sql` stays pipeable; warnings go to stderr. +Text mode writes only the SQL to stdout, so redirecting works. JSON: `filepath`, `sql`, `durationMs`. A missing template exits 2. A render error exits 1. The output contains resolved secrets in plaintext, so it needs `run:file` like `inspect`. -### Change Operations +### Changes -Bare `noorm change` renders citty's help output and does not connect to the database. Use `noorm change list` for the status table. -**Interactive prompts (TTY only).** When you omit the change name on a TTY, `run`, `revert`, `rewind`, `edit`, `rm`, and `history-detail` open a clack picker filtered to the relevant subset: +Bare `noorm change` prints help and does not connect. Every `change` command that connects accepts `--config` and `--json`. -| Command | Picker includes | -|---------|-----------------| -| `change run` | pending + reverted + stale (not orphaned) | -| `change revert` | success | -| `change rewind` | success | -| `change edit` | every directory under `changes/` (filesystem) | -| `change rm` | every directory under `changes/` (filesystem) | -| `change history-detail` | anything with an execution record (non-pending) | +When you omit the name on a terminal, these commands open a picker. Without a terminal they exit 1 and ask for the name. -Non-TTY callers (CI, scripts) must pass the name or the command errors out. +| Command | Picker lists | +|---|---| +| `change run` | pending, reverted, and stale changes, and never-run changes that cannot be parsed. Not orphaned ones | +| `change revert`, `change rewind` | applied changes | +| `change edit`, `change rm` | every directory under `paths.changes` | +| `change history-detail` | every change with an execution record | +| `change add` | prompts for a description | #### `change list` -List every known change with its status. + +List every known change and its status: `pending`, `success`, `failed`, `reverted`, or `stale`, plus `orphaned` when the folder is gone. ```bash -noorm change list noorm change list --json -noorm change list -c staging ``` -**JSON output:** -```json -{ - "success": true, - "changes": [ - {"name": "2024-01-15-init-schema", "status": "success"}, - {"name": "2024-02-01-notifications", "status": "pending"} - ], - "pending": 1 -} -``` +JSON: `changes[]` and `pending`. An unreadable change that never ran shows as `failed`, with the parse message after it in text and in `errorMessage` under `--json`. #### `change ff` -Fast-forward: apply all pending changes in order. + +Apply every pending change in order. + +``` +noorm change ff [--config <name>] [--dry-run] [--force] [--json] +``` + +| Flag | Type | Effect | +|---|---|---| +| `--config`, `-c` | string | Config to run against. Defaults to the active config | +| `--dry-run` | boolean | Render the changes to `tmp/` without writing to the database | +| `--force`, `-f` | boolean | Skip the checksum check and run every file of each pending change | +| `--json` | boolean | Print the result as JSON | ```bash noorm change ff +noorm change ff -c staging --json noorm change ff --dry-run -noorm change ff --force ``` -**JSON output:** -```json -{ - "success": true, - "status": "success", - "executed": 2, - "skipped": 0, - "failed": 0, - "durationMs": 90, - "changes": [ - {"name": "2024-02-01-notifications", "status": "success", "durationMs": 45} - ] -} -``` +| Exit code | Meaning | +|---|---| +| `0` | Every pending change applied, or none were pending | +| `1` | A change failed, or the command could not run | +| `3` | Partial: some changes applied before one failed | + +JSON fields: + +| Field | Meaning | +|---|---| +| `success`, `status` | `status` is `success`, `partial`, or `failed` | +| `executed`, `skipped`, `failed` | Change counts | +| `durationMs` | Total time | +| `changes[]` | Per change: `name`, `direction`, `status`, `files[]`, `durationMs`, `error` | +| `warnings[]` | Non-fatal problems, such as a missing `db/changes/` directory | +| `error` | Why the batch failed before any change ran | +| `dryRun` | `true` under `--dry-run` | + +A missing `db/changes/` directory is a warning, not a failure. A change folder that cannot be parsed, for example a `.txt` manifest naming a file no longer in `db/sql/`, is handled by its status: + +| The unreadable change | `ff` and `next` | +|---|---| +| Never ran, `reverted`, or `stale` | Fails at that change's position, so later changes do not run out of order. The result is `failed` (exit `1`), or `partial` (exit `3`) when earlier changes applied, and `error` carries the parse message | +| Applied | Keeps its status and adds a message to `warnings` | -`--dry-run` adds `"dryRun": true` to the payload. A missing `changes/` directory is a warning, not a failure: it lands in `warnings` rather than making the batch fail. +A folder with neither `change/` nor `revert/` is not a change and is ignored. On a database built from `db/sql/`, the first `ff` runs every change, so write changes that are safe to run twice. See [Add a column](/guide/changes/add-a-column). #### `change next` -Apply the next pending change. An optional positional count applies the next `n` instead of one. + +Apply the next pending change, or the next `n` with a positional count. ```bash -noorm change next -noorm change next 3 -noorm change next --json +noorm change next 3 --dry-run ``` +Accepts `--config`, `--force`, `--dry-run`, and `--json`. A count that is not a positive integer exits 1. Failed and unreadable changes behave as in `ff`. JSON matches `ff`. + #### `change run [name]` -Apply a specific change by name. Omit the name on a TTY to pick interactively. -```bash -noorm change run # interactive picker (TTY) -noorm change run 2024-02-01-notifications -noorm change run 2024-02-01-notifications --json -``` +Apply one change by name. Without a name on a terminal, pick from the list. -**JSON output:** -```json -{ - "success": true, - "name": "2024-02-01-notifications", - "direction": "change", - "status": "success", - "files": [ - {"filepath": "change/001_create-table.sql", "checksum": "a1b2c3", "status": "success", "durationMs": 23} - ], - "durationMs": 45 -} +```bash +noorm change run 2026-10-20-order-status --json ``` -`direction` is `change` or `revert`. Per-file `status` is one of `pending`, `success`, `failed`, `skipped`; the change-level `status` adds `reverted` and `stale`. +Accepts `--config`, `--force`, `--dry-run`, and `--json`. A change whose folder cannot be parsed reports the parse error instead of "Change not found". JSON: `name`, `direction`, `status`, `files[]` (`filepath`, `checksum`, `status`, `durationMs`), `durationMs`. `direction` is `change` or `revert`. #### `change revert [name]` -Revert a previously applied change by running its rollback SQL. Omit the name on a TTY to pick interactively. + +Run one applied change's `revert/` scripts. ```bash -noorm change revert # interactive picker (TTY) -noorm change revert 2024-02-01-notifications -noorm change revert 002_users --json +noorm change revert 2026-10-20-order-status ``` +Accepts `--config`, `--force`, `--dry-run`, and `--json`. + #### `change rewind [name]` -Revert the given change **and every change applied after it**, in reverse order. Omit the name on a TTY to pick interactively. + +Revert the named change and every change applied after it, newest first. A bare integer reverts the last `n` instead. ```bash -noorm change rewind # interactive picker (TTY) -noorm change rewind 2024-02-01-notifications -noorm change rewind 001_init --json +noorm change rewind 2026-10-20-order-status +noorm change rewind 3 ``` +Accepts `--config`, `--force`, `--dry-run`, and `--json`. A name that matches no applied change fails with `error` set. + #### `change add [description]` -Scaffold a new change directory. Omit the description on a TTY to be prompted. + +Scaffold `<paths.changes>/<YYYY-MM-DD>-<slug>/` with `change/` and `revert/`, each holding `001_<slug>.sql`, plus `changelog.md`. No database connection. ```bash -noorm change add # prompts for description (TTY) -noorm change add add-users-table -noorm change add "notification queue" --json +noorm change add order-status --json ``` -Creates `changes/<YYYY-MM-DD>-<slug>/` with `change/` and `revert/` directories, each -holding a `001_<slug>.sql` stub (a single comment naming what belongs there — "Add the SQL -statements that apply this change" / "...undo this change"), plus `changelog.md`. The stub -comments alone don't count as content: running the change before editing them fails with an -actionable message ("Files are empty or contain only template placeholders"), not "change not -found". +JSON: `name`, `path`. The stub files hold one comment each, so running the change before you edit them fails with "Files are empty or contain only template placeholders". An existing folder of the same name fails. #### `change edit [name]` -Open a change folder in `$EDITOR`. Omit the name on a TTY to pick interactively. + +Open a change folder in `$EDITOR`, then `$VISUAL`, then `code`. The command exits with the editor's exit code. An unknown change exits 2. Takes no flags besides the name. ```bash -noorm change edit # interactive picker (TTY) -noorm change edit 2024-02-01-notifications -EDITOR=vim noorm change edit 001_init +EDITOR=vim noorm change edit 2026-10-20-order-status ``` #### `change rm [name]` -Remove a change directory from disk. Does **not** touch database state. Gated by the config's `change:rm` access (see [Access Roles](#access-roles) above): `viewer` is denied; `operator` and `admin` confirm via `--yes` or `NOORM_YES=1`. On a TTY, omit the name to pick interactively. + +Delete a change folder from disk. The database is untouched. The role gate applies when a config resolves, and `--yes` or `NOORM_YES` confirms. With no config at all, confirmation is required. An unknown change exits 2. ```bash -noorm change rm # picker + confirm (TTY) -noorm change rm 2024-02-01-notifications --yes # CI +noorm change rm 2026-10-20-order-status --yes ``` +JSON: `name`, `deleted`. + #### `change history` -View execution history with timestamps, direction, and duration. + +Show execution records. ```bash -noorm change history -noorm change history --json -noorm change history --count 50 +noorm change history --count 50 --json ``` -| Flag | Description | -|------|-------------| -| `--count` | Show the last N records (default: 20) | +| Flag | Effect | +|---|---| +| `--count` | Last N records. Default 20 | -**JSON output:** -```json -{ - "success": true, - "history": [ - { - "id": 1, - "name": "001_init", - "direction": "change", - "status": "success", - "executedAt": "2024-01-15T10:30:00Z", - "executedBy": "Alice <alice@example.com>", - "durationMs": 45, - "errorMessage": null, - "checksum": "abc123def456" - } - ] -} -``` +JSON: `history[]` with `id`, `name`, `direction`, `status`, `executedAt`, `executedBy`, `durationMs`, `errorMessage`, `checksum`. #### `change history-detail [name]` -Per-file execution detail for a single change. Omit the name on a TTY to pick interactively. + +Show per-file results for each operation on one change. ```bash -noorm change history-detail # interactive picker (TTY) -noorm change history-detail 001_init -noorm change history-detail 002_users --json +noorm change history-detail 2026-10-20-order-status --count 5 ``` +| Flag | Effect | +|---|---| +| `--count` | Last N operations. Default 10 | + +JSON: `name` and `operations[]`, each with the history `record` and its `files[]`. + + +### Database -### Database Operations #### `db create` -Create the target database if it does not exist. Connects to the dialect's system database (`postgres`, `master`, etc.) to issue the CREATE statement. + +Create the database if it does not exist and bootstrap noorm's tracking tables. It connects to the dialect's system database to issue the statement. An existing, initialized database reports `alreadyExists` and exits 0. ```bash -noorm db create -noorm db create -c dev -noorm db create --yes -noorm db create --json +noorm db create --config dev --yes ``` -Gated by the config's `db:create` access: `viewer` is denied, `operator` needs `--yes` (or `NOORM_YES=1`), `admin` runs unconfirmed. The gate fires before any probe touches the server, so a denied role never gets a SQLite file created as a side effect of the existence check. +Accepts `--config`, `--yes`, and `--json`. The role gate runs before any probe, so a denied role never gets a SQLite file created as a side effect. JSON: `config`, `database`, `created`, `trackingInitialized`. #### `db drop` -Drop the entire database. + +Drop the whole database. ```bash -noorm db drop --yes -noorm db drop -c dev --yes --json +noorm db drop --config dev --yes ``` -::: danger Destructive -Drops the entire database. Requires `--yes`. Gated by the config's `db:destroy` access: `viewer`/`operator` are denied outright — there is no flag that overrides this — and `admin` still requires confirmation. See [Access Roles](#access-roles) above. -::: +Accepts `--config`, `--yes`, and `--json`. When `NOORM_CONNECTION_*` points the config at another database than the one it stores, stderr warns before the drop, and JSON adds `targetOverridden` and `storedDatabase`. JSON: `config`, `database`, `dropped`. #### `db reset` -Teardown + build (idempotent rebuild). Drops all user objects then rebuilds from SQL files. + +Tear down every user object, then build from `db/sql/`. `--yes` or `NOORM_YES` is required up front, else exit 1. ```bash -noorm db reset --yes -noorm db reset -c dev --yes --json +noorm db reset --config dev --yes ``` -::: danger Destructive -Equivalent to `db teardown` followed by `run build`. Requires `--yes`. -::: +Accepts `--config`, `--yes`, and `--json`. JSON: `reset`. #### `db explore` -Explore database schema. + +Overview counts, with a subcommand per object type. ```bash -noorm db explore # Overview counts -noorm db explore tables # List all tables -noorm db explore tables detail users # Describe specific table -noorm db explore views # List all views -noorm db explore functions # List all functions -noorm db explore procedures # List all procedures -noorm db explore indexes # List all indexes -noorm db explore types # List custom types -noorm db explore fks # List foreign keys -noorm db explore triggers # List triggers -noorm db explore --json # JSON overview +noorm db explore +noorm db explore tables detail sales_order +noorm db explore fks --schema public ``` -**JSON output (overview):** -```json -{ - "success": true, - "tables": 12, - "views": 3, - "procedures": 0, - "functions": 2, - "types": 4, - "indexes": 8, - "foreignKeys": 6, - "triggers": 1, - "locks": 0, - "connections": 3 -} -``` +| Subcommand | Lists | JSON key | +|---|---|---| +| `tables` | tables. `tables detail <name>` describes one | `tables` | +| `views [name]` | views, or one view | `views` | +| `procedures [name]` | stored procedures, or one | `procedures` | +| `functions [name]` | functions, or one | `functions` | +| `types [name]` | custom types, or one | `types` | +| `indexes` | indexes | `indexes` | +| `fks` | foreign keys | `foreignKeys` | +| `triggers [name]` | triggers, or one | `triggers` | -`locks` and `connections` are runtime state, not schema, so they appear only in `--json` — the text overview lists the eight counters that have a drill-down subcommand. +Every subcommand accepts `--config` and `--json`, plus `--schema <name>` to restrict the listing or name the schema of the object you describe. A name that matches nothing exits 2. The overview JSON has `tables`, `views`, `procedures`, `functions`, `types`, `indexes`, `foreignKeys`, `triggers`, `locks`, and `connections`. `locks` and `connections` are runtime state, so only `--json` shows them. #### `db truncate` -Wipe all data from application tables. Schema and noorm tracking tables are preserved. + +Delete all rows from application tables. Schema and noorm tracking tables stay. ```bash -noorm db truncate -y -noorm db truncate --dry-run -noorm db truncate --preserve seeds,lookups --yes -noorm db truncate --only users,posts --yes +noorm db truncate --preserve party_type,product --yes ``` -| Flag | Description | -|------|-------------| +| Flag | Effect | +|---|---| | `--preserve` | Comma-separated tables to leave untouched | -| `--only` | Comma-separated tables to truncate, to the exclusion of all others | -| `--dry-run` | List the statements without executing them | +| `--only` | Comma-separated tables to truncate, and no others | +| `--dry-run` | List the statements without running them | +| `--yes`, `-y` | Confirm | -::: warning Access Roles -Gated by the config's `db:truncate` access: `viewer` is denied, `operator` and `admin` both confirm (`--yes`, or `NOORM_YES=1` in CI). There is no `--force` override — see [Access Roles](#access-roles) above. -::: +Also accepts `--config` and `--json`. JSON: `truncated`, `preserved`, `count`, and under `--dry-run` `dryRun` and `statements`. #### `db teardown` + Drop all database objects. ```bash -noorm db teardown -y -noorm db teardown --dry-run noorm db teardown --preserve-schemas app_private --yes ``` -| Flag | Description | -|------|-------------| -| `--preserve-schemas` | Comma-separated schemas to leave untouched (teardown otherwise reaches every non-system schema) | -| `--dry-run` | Report what would be dropped without dropping it | - -A teardown whose post-teardown script failed exits `1` even though the objects are already gone — a half-finished teardown must not report success. +| Flag | Effect | +|---|---| +| `--preserve-schemas` | Comma-separated schemas to leave untouched. Teardown otherwise reaches every non-system schema | +| `--dry-run` | Report what would drop | +| `--yes`, `-y` | Confirm | -::: warning Access Roles -Gated by the config's `db:teardown` access: `viewer` and `operator` are denied outright, `admin` confirms (`--yes`, or `NOORM_YES=1` in CI). There is no `--force` override — see [Access Roles](#access-roles) above. -::: +Also accepts `--config` and `--json`. JSON: `dropped` (`tables`, `views`, `functions`, `types`), `count`, `postScriptResult`, `dryRun`. A post-teardown script that fails exits 1 even though the objects are gone, so a half-finished teardown does not report success. #### `db transfer` -Transfer data between database configurations, or export/import `.dt` files. Three mutually exclusive modes. -**Database-to-database:** -```bash -noorm db transfer --to backup -noorm db transfer --to backup --tables users,posts -noorm db transfer --to backup --on-conflict update -noorm db transfer --to backup --truncate -noorm db transfer --to backup --dry-run -``` +Move data between configs, or to and from `.dt` files. Exactly one of `--to`, `--export`, and `--import`; none or several exit 2. -**Export to files:** ```bash -noorm db transfer --export ./backup/ +noorm db transfer --to backup --tables party,sales_order --on-conflict update noorm db transfer --export ./backup/ --compress -noorm db transfer --export ./backup/ --passphrase "correct-horse-battery" -noorm db transfer --export ./backup/users.dt --tables users +noorm db transfer --import ./backup/party.dtz --on-conflict skip ``` -**Import from files:** -```bash -noorm db transfer --import ./backup/users.dt -noorm db transfer --import ./backup/users.dtzx --passphrase "correct-horse-battery" --on-conflict skip -``` +| Flag | Applies to | Effect | +|---|---|---| +| `--to <config>` | database to database | Destination config. Unknown config exits 2 | +| `--export <path>` | export | A file for one table, or a directory for several | +| `--import <path>` | import | A `.dt`, `.dtz`, or `.dtzx` file | +| `--compress` | export | Write `.dtz`. Any other mode exits 2 | +| `--passphrase <text>` | export, import | Encrypt to `.dtzx`, or decrypt one. Export needs at least 12 characters. Omit it on a terminal for a masked prompt. In a non-interactive session or under `--json`, omitting it exits 2 | +| `--tables <a,b>` | database to database, export | Comma-separated tables. Default all | +| `--on-conflict <s>` | database to database, import | `fail`, `skip`, `update`, or `replace`. Default `fail` | +| `--batch-size <n>` | all | Rows per batch. Default 1000 | +| `--truncate` | database to database, import | Truncate destination tables first | +| `--no-fk` | database to database | Leave foreign key checks enforced. By default they are disabled during the transfer | +| `--no-identity` | database to database | Let the destination assign identity values. By default they are preserved | +| `--dry-run` | database to database | Print the plan and exit 0 | -`--passphrase` implies encryption, so the export lands as `.dtzx` whether or not you also pass `--compress`. On export it must be at least 12 characters; on import there is no floor, so archives encrypted with an older, shorter passphrase still open. Omit it on a TTY and noorm prompts with masked input; in a non-interactive session or under `--json`, omitting it exits `2`. +`--config` selects the source, and `--json` is accepted. The export extension follows `--passphrase` (`.dtzx`), then `--compress` (`.dtz`), then `.dt`. -**Transfer flags:** +| Mode | Exit | JSON | +|---|---|---| +| Database to database | 0 success, 1 failed, 3 partial | `status`, `tables[]`, `totalRows`, `durationMs`, `fkChecksRestored` | +| `--dry-run` | 0 | `sameServer`, `tableCount`, `estimatedRows`, `tables[]`, `warnings[]` | +| Export | 0, or 1 on error | `mode: "export"`, `tables[]`, `totalRows`, `totalBytes`, and `filepath` or `directory` | +| Import | 0, or 1 on error | `mode: "import"`, `filepath`, `rowsImported`, `rowsSkipped` | -| Flag | Description | -|------|-------------| -| `--to` | Destination config name (db-to-db mode) | -| `--export` | Export path: file or directory | -| `--import` | Import from `.dt`/`.dtz`/`.dtzx` file | -| `--compress` | Compress export as `.dtz` | -| `--passphrase` | Passphrase for `.dtzx` encryption/decryption | -| `--tables` | Comma-separated table list (default: all) | -| `--on-conflict` | `fail`, `skip`, `update`, or `replace` (default: `fail`) | -| `--batch-size` | Rows per batch for cross-server transfers (default: 1000) | -| `--truncate` | Truncate destination tables before transfer | -| `--no-fk` | Do not disable foreign key checks | -| `--no-identity` | Do not preserve identity/auto-increment values | -| `--dry-run` | Show transfer plan without executing | +When foreign key checks are not restored on the destination, stderr warns and `fkChecksRestored` is `false`. -### Lock Operations +### Locks + + +Tool-level mutual exclusion between noorm runs against one config. `change run`, `ff`, `next`, `revert`, and `rewind` take and release their own lock around each change. -Tool-level mutual exclusion. Prevents concurrent schema migrations across noorm instances or CI runners targeting the same database. #### `lock status` -Check if database is locked. + +Show whether the config is locked, and by whom. ```bash -noorm lock status noorm lock status --json ``` -**JSON output:** -```json -{ - "isLocked": true, - "lock": { - "lockedBy": "deploy@ci-runner", - "lockedAt": "2024-01-15T10:30:00Z", - "expiresAt": "2024-01-15T10:35:00Z" - } -} -``` +JSON: `isLocked` and `lock` (`lockedBy`, `lockedAt`, `expiresAt`), or `lock: null`. #### `lock acquire` -Acquire an exclusive lock. + +Take the config's lock so other noorm runs against it fail until you release it or it expires. ```bash -noorm lock acquire noorm lock acquire --timeout 600000 --reason "nightly migration" -noorm lock acquire --json ``` -| Flag | Description | -|------|-------------| -| `--timeout` | Lock duration in milliseconds before it expires (default: 300000) | +| Flag | Effect | +|---|---| +| `--timeout` | Milliseconds before the lock expires. Default 300000. A non-positive or non-numeric value exits 2 | | `--reason` | Text shown to anyone this lock blocks | -**CI/CD pattern with cleanup:** -```bash -noorm lock acquire -trap "noorm lock release" EXIT -noorm change ff -``` +Another holder fails the command with exit 1. The same identity extends its own lock. JSON: `acquired`, `lockedBy`, `expiresAt`, `reason`. #### `lock release` -Release the current lock. Only the lock holder can release. + +Release your own lock. A lock held by someone else, or no lock, exits 1. JSON: `released`. ```bash noorm lock release @@ -1254,501 +1104,320 @@ noorm lock release #### `lock force` -Force release any lock regardless of ownership. Gated by the config's `lock:force` access, which is a confirm cell for both `operator` and `admin`, so `--yes` (or `NOORM_YES=1`) is required. + +Release any holder's lock. Needs `--yes` or `NOORM_YES`. ```bash noorm lock force --yes -noorm lock force -c prod --yes --json ``` -Exits `0` when a lock was actually broken and `2` when there was nothing to release, so a script can tell "evicted a holder" from "no-op" without parsing text. The `--json` payload reports `success` as whether a lock was broken, not whether the command ran. +Exit 0 when a lock was broken and 2 when there was none, so a script can tell an eviction from a no-op. JSON: `success` (whether a lock was broken), `released`, `holder`, `forced`. Breaking a lock can corrupt data if the holder is still running. -::: warning -Force releasing a lock can cause data corruption if the original holder is still running. -::: +### Vault -### Vault Operations -Team-shared encrypted secrets stored in the database. +Team-shared secrets, encrypted in the config's own database. Vault commands load your identity first and exit 1 when it is missing. See [Vault](/guide/environments/vault). -**Secret resolution order:** -1. Config-specific local secrets (highest) -2. Global local secrets -3. Vault secrets (team-shared, lowest) +All but `cp` accept `--config`, `--json`, and `--yes`. #### `vault init` -Initialize the vault for the current database. + +Create the vault and seal its key to your identity. ```bash noorm vault init -noorm vault init --json ``` +```text +Vault initialized successfully +``` + +An initialized vault you can read reports success. One you cannot read exits 1 and asks for a `propagate`. An `operator` role needs `--yes`. + #### `vault set <key> [value]` -Store an encrypted secret in the vault. + +Create or update a secret. `--stdin` reads the value from stdin. Pass the value or `--stdin`, not both. ```bash -noorm vault set API_KEY "sk-live-abc123" -echo "$API_KEY" | noorm vault set API_KEY --stdin -noorm vault set API_KEY "sk-live-abc123" --json +printf %s "$KEY" | noorm vault set PAYMENT_GATEWAY_KEY --stdin ``` -| Flag | Description | -|------|-------------| -| `--stdin` | Read the value from stdin instead of argv, so it never reaches the process table, shell history, or a `set -x` trace. Omit the positional value when you pass it | -| `--yes` / `-y` | Confirm the write on a config whose `vault:write` access is a confirm cell (an `operator` user role) | - -Upserts: creates if new, updates if the key exists. +An `operator` role needs `--yes`. Without vault access, exit 1. JSON: `key`, `action`. #### `vault list` -List all secrets in the vault (keys and metadata only, never values). + +List keys, who set them, and when. Never values. ```bash -noorm vault list noorm vault list --json ``` -**JSON output:** -```json -{ - "success": true, - "secrets": [ - {"key": "API_KEY", "setBy": "alice@example.com", "updatedAt": "2024-01-15T10:30:00Z"} - ], - "status": { - "usersWithAccess": 3, - "usersWithoutAccess": 1 - } -} -``` +JSON: `secrets[]` (`key`, `setBy`, `updatedAt`) and `status` (`usersWithAccess`, `usersWithoutAccess`). #### `vault rm <key>` -Remove a secret from the vault. `--yes` (or `NOORM_YES=1`) is mandatory: the vault has no soft-delete and no history table, so this destroys the team's only copy. + +Delete a secret. `--yes` or `NOORM_YES` is required, because the vault keeps no history and this destroys the team's only copy. A key that was never in the vault exits 2. ```bash noorm vault rm OLD_API_KEY --yes -noorm vault rm OLD_API_KEY --yes --json ``` -A key that was never in the vault exits `2`, which is distinct from a delete that failed (`1`). +JSON: `key`, `deleted`. #### `vault cp <key> <source> <destination>` -Copy one secret from one config's vault to another's. All three arguments are positional and required — there is no bulk mode. + +Copy one secret from one config's vault to another's. All three arguments are positional. ```bash -noorm vault cp API_KEY staging production -noorm vault cp API_KEY staging production --dry-run -noorm vault cp DB_PASSWORD dev staging --force -noorm vault cp API_KEY staging production --json +noorm vault cp PAYMENT_GATEWAY_KEY staging production --dry-run ``` -| Flag | Description | -|------|-------------| -| `--force` / `-f` | Overwrite the key if it already exists in the destination | -| `--dry-run` | Preview without executing | +| Flag | Effect | +|---|---| +| `--force`, `-f` | Overwrite the key if the destination has it. Without it, the key is skipped | +| `--dry-run` | Run the same checks, write nothing | -Without `--force`, a key that already exists in the destination is skipped. An unknown source or destination config exits `2`. +Unknown source or destination exits 2. `--config` is accepted and ignored. JSON: `dryRun`, `copied`, `skipped`, `errors`. #### `vault propagate` -Grant vault access to team members who don't have it yet. Encrypts the vault key with each pending user's public key. -Propagation seals the vault key to a public key and cannot be revoked, so a bare `noorm vault propagate` only *lists* the identities awaiting access and exits `1`. Re-run with `--yes` to grant, or `--to <hash>` to grant to specific identities. +Grant vault access to identities waiting for it. A grant cannot be revoked, so a bare `propagate` only lists who is waiting and exits 1. ```bash -noorm vault propagate # review the pending list -noorm vault propagate --yes # grant to everyone pending -noorm vault propagate --to 4a5c14af... --yes # grant to one identity -noorm vault propagate --json +noorm vault propagate # list who is waiting +noorm vault propagate --to 4a5c14af… --yes # grant one identity +noorm vault propagate --yes # grant everyone waiting ``` -| Flag | Description | -|------|-------------| -| `--to` | Comma-separated identity hashes to grant to (default: every pending identity) | -| `--yes` / `-y` | Perform the grant. Without it the command reports and stops | +| Flag | Effect | +|---|---| +| `--to` | Comma-separated identity hashes. Default: everyone waiting. A hash that is not waiting fails | +| `--yes`, `-y` | Perform the grant | + +JSON: `granted`, `propagatedTo[]` (hashes), `alreadyHadAccess`, `failed[]`, `pending[]` (`identityHash`, `name`, `email`). A grant that lands for some identities and fails for others exits 3. -**JSON output:** -```json -{ - "success": true, - "granted": true, - "propagatedTo": ["4a5c14af…"], - "alreadyHadAccess": 3, - "failed": [], - "pending": [ - {"identityHash": "4a5c14af…", "name": "Alice", "email": "alice@example.com"} - ] -} -``` -`propagatedTo` holds identity hashes; `pending` is what maps them back to a name and email. A grant that lands for some identities and not others exits `3` with `success: false` — the vault is in a mixed state, and the teammate whose update did not land believes they have access. +### SQL -### SQL Operations +#### `sql query [sql]` -#### `sql [query]` -Execute a raw SQL query. +Run one statement. The statement can be positional or read from a file. ```bash -noorm sql "SELECT * FROM users LIMIT 10" -noorm sql query -f query.sql -noorm sql query -c prod "SELECT count(*) FROM orders" -noorm sql "SELECT id, name FROM users" --json +noorm sql query "SELECT party_id, name FROM party ORDER BY party_id" ``` -| Flag | Description | -|------|-------------| -| `-f, --file` | Read SQL from a file | -| `-c, --config` | Use specific configuration | +```text +Columns: party_id, name +{"party_id":1,"name":"Meridian Analytics"} +{"party_id":2,"name":"Ada Lovelace"} +2 rows returned (3ms) +``` -The bare `noorm sql "<SQL>"` form works by rewriting argv to `noorm sql query "<SQL>"`, and it only fires when the first non-flag token after `sql` looks like SQL. A flag that takes a value puts something else there, so `noorm sql -f query.sql` and `noorm sql -c prod "SELECT 1"` print help instead — spell out `query` with either. See [`noorm sql`](./cli/sql.md). +| Flag | Effect | +|---|---| +| `--file`, `-f` | Read the SQL from a file. It replaces a positional statement | +| `--config`, `-c` | Config to run against | +| `--json` | Print the result as JSON | -**JSON output:** -```json -{ - "success": true, - "columns": ["id", "name"], - "rows": [ - {"id": 1, "name": "Alice"}, - {"id": 2, "name": "Bob"} - ], - "durationMs": 12.5 -} -``` +No statement, or an unreadable file, exits 2. A failed statement exits 1. A write prints `Rows affected: n` in place of the row count. JSON: `columns`, `rows`, `durationMs`. `rowsAffected` replaces the rows for a write and is absent on a `SELECT`. The role gate classifies the statement. + +`noorm sql "SELECT 1"` works because the entry point rewrites it to `sql query` when the first non-flag token starts with a SQL verb, such as `SELECT`, `INSERT`, `WITH`, `CREATE`, or `DROP`. `noorm sql -f q.sql` and `noorm sql -c prod "SELECT 1"` print help, because a flag's value sits in that position. Spell out `query` there. See [`noorm sql`](/cli/sql). -`rowsAffected` replaces the row payload for INSERT/UPDATE/DELETE and is absent (not `null`) on a SELECT. +`sql query` records no history, so a build agent never persists query text or rows. #### `sql repl` -Launch the TUI SQL Terminal with multi-line editing, sortable results, and history. + +Open the TUI at the SQL Terminal. Exits 2 under `--yes` or `NOORM_YES`, or without a terminal. `--config` makes that config the active one before it starts. ```bash -noorm sql repl noorm sql repl --config dev ``` -**TTY required.** `--yes` / `NOORM_YES` errors with a redirect hint — use `noorm sql query "<SQL>"` or `noorm sql query --file query.sql` for non-interactive SQL. +See [`sql repl`](/cli/sql-repl). #### `sql history` -Show SQL execution history. Only the interactive SQL terminal records history — `sql query` deliberately writes none, so persisting query text and returned rows on a build agent never happens. + +Show history recorded by the interactive terminal for one config. ```bash -noorm sql history -noorm sql history -n 20 -noorm sql history --json -noorm sql history -c prod +noorm sql history -n 20 --json ``` -| Flag | Description | -|------|-------------| -| `-n, --limit` | Max entries (default: 50) | +| Flag | Effect | +|---|---| +| `--limit`, `-n` | Maximum entries. Default 50. A non-positive value exits 2 | +| `--config`, `-c` | Config whose history to read. Defaults to `NOORM_CONFIG`, then the active config. None resolved exits 2 | + +JSON: `configName` and `entries[]` (`id`, `query`, `executedAt`, `durationMs`, `success`, `errorMessage`, `rowCount`). #### `sql clear` -Clear SQL execution history. + +Delete recorded history and its result files. ```bash -noorm sql clear --yes noorm sql clear --older-than 3 --yes -noorm sql clear -c prod --yes ``` -| Flag | Description | -|------|-------------| -| `--older-than` | Only clear entries older than N months | -| `--yes` / `-y` | Confirm. Without it the command reports what it would clear and exits 0 without clearing | +| Flag | Effect | +|---|---| +| `--older-than` | Only entries older than N months. Must be a positive integer, else exit 2 | +| `--yes`, `-y` | Confirm. Without it, and without `--json`, the command reports what it would clear and exits 0 | + +`NOORM_YES` does not count here. JSON: `configName`, `entriesRemoved`, `filesRemoved`. + -`--json` also counts as a confirmation, since it is only used non-interactively. +### MCP -### MCP (AI Agent Integration) +Expose noorm to AI coding agents over the Model Context Protocol. See [MCP](/guide/automation/mcp). -Model Context Protocol server for AI coding agents. Exposes noorm operations as MCP tools. #### `mcp init` -Generate MCP configuration for a coding agent. + +Write or extend the agent's MCP config with a `noorm` server entry that runs `noorm mcp serve`. Other entries stay. ```bash -noorm mcp init # Default: Claude Code -noorm mcp init --agent cursor # Cursor editor -noorm mcp init --json +noorm mcp init --agent cursor ``` +| Flag | Effect | +|---|---| +| `--agent` | `claude` (default) writes `.mcp.json`. `cursor` writes `.cursor/mcp.json`. Other values write `.mcp.json` | + +JSON: `path`, `created`, `extended`. + #### `mcp serve` -Start the noorm MCP server on stdio transport. Designed to be spawned by an AI agent; stays alive as long as stdin is open. + +Start the MCP server on stdio. It stays alive while stdin is open. Always on the `agent` channel. ```bash noorm mcp serve ``` -### Utility Commands +### Diagnostics and updates + #### `info` -Project and database status overview. + +Report versions, the active config and its connection, your identity, the detected agent harness, and object counts. ```bash -noorm info noorm info --json ``` -**JSON output:** -```json -{ - "success": true, - "cli_version": "1.0.0-alpha.34", - "schema_version": 2, - "state_version": 3, - "settings_version": 1, - "installed_at": "2024-01-15T10:30:00.000Z", - "upgraded_at": null, - "active_config": "dev", - "config_count": 2, - "connection": { - "host": "localhost", - "port": 5432, - "database": "mydb", - "dialect": "postgres" - }, - "identity": { - "name": "Your Name", - "email": "you@example.com", - "machine": "laptop", - "registered_at": "2024-01-15T10:30:00.000Z", - "last_seen_at": "2024-02-01T09:00:00.000Z" - }, - "agent": null, - "objects": { - "tables": 5, - "views": 12, - "functions": 8, - "procedures": 9, - "types": 13 - } -} -``` - -`connection` is `null` and `connection_error` carries the message when the active config can't be reached. `agent` names the AI harness noorm detected, if any, along with the env markers that gave it away — worth checking first when a permission denial has no visible cause, since harness detection is what flips the policy channel from `user` to `agent`. +`--config` is accepted and ignored: `info` always reports the active config. An unreachable database still exits 0, with `connection: null` and `connection_error` set. JSON: `cli_version`, `schema_version`, `state_version`, `settings_version`, `installed_at`, `upgraded_at`, `active_config`, `config_count`, `connection`, `identity`, `agent` (`id`, `name`, `markers`), `objects`. #### `version` -CLI version and diagnostic information. + +Report the CLI version, Node, platform, identity files, and project detection. No database connection. Exits 0. ```bash -noorm version noorm version --json ``` +JSON: `version`, `node`, `platform`, `arch`, `identity`, `project`. + #### `update` -Check for and install noorm updates. + +Check for a newer release and replace the running binary. ```bash noorm update -noorm update --json ``` -| Flag | Description | -|------|-------------| -| `--insecure` | Downgrade an unreachable `checksums.txt` from a failure to a warning. Also settable as `NOORM_INSECURE=1`. It never bypasses a confirmed checksum mismatch, which always fails | +| Flag | Effect | +|---|---| +| `--insecure` | Downgrade an unreachable `checksums.txt` from a failure to a warning. A confirmed checksum mismatch always fails. `NOORM_INSECURE` does the same | +| `--json` | Print only the JSON document | +On a terminal, status lines rewrite in place and the download ends on `Downloaded X MB (100%)`. Piped output prints one line per step. A failed check or install exits 1. JSON: `currentVersion`, `latestVersion`, `updateAvailable`, `installed`, and `error` on failure. -#### `ui` +After a successful update, `update` offers the agent skill through the first installer on `PATH`: -Launch the interactive terminal UI. +| Found | Command | +|---|---| +| `npx` | `npx skills add noormdev/noorm/skills` | +| `gh`, no `npx` | `gh skill install noormdev/noorm skills/noorm`, which needs GitHub CLI 2.90 or newer | +| Neither | Nothing runs. Both commands print, with where to get Node.js and the GitHub CLI | + +On a terminal it asks first and runs the command on yes. Piped output prints the command without asking, and `--json` skips the offer. A failed install prints both commands. A declined or failed skill install does not change the exit code. + + +#### `complete` + + +Print a shell completion script. Takes `zsh`, `bash`, `fish`, `powershell`, or `fig`. ```bash -noorm ui +source <(noorm complete zsh) ``` -**TTY required.** - - -## Exit Codes - -| Code | Meaning | -|------|---------| -| `0` | Success — everything you asked for happened | -| `1` | Total failure — the operation ran and no unit of work succeeded | -| `2` | Usage error — a bad or missing flag, a TTY-only command run non-interactively, or a named target (file, directory, config, change, secret key, glob) that does not exist. Nothing was attempted, so nothing changed | -| `3` | Partial failure — some units succeeded and some failed. The target is in a mixed state and re-running is not automatically safe | +`fig` prints a JSON spec. A bare `noorm complete` is the runtime hook the scripts call, and it exits 1 without `--`. + + +#### `dev test-helpers <template>` and `dev test-workers` -The split between `1` and `3` is the one that matters in a pipeline: a total -failure can be retried, a partial one needs a human. `2` says the command never -got as far as touching the database. -Policy denials currently exit `1`. They do not yet have a dedicated code. - -> **Changed in this release.** `2` previously meant "partial failure" on -> `run build` / `run dir` / `run files` / `run exec` and the five `change` -> execution commands, but the same `2` was also returned for a *total* failure -> on those commands — the two were indistinguishable. Partial now has its own -> code (`3`), total failure reports `1`, and `2` is reserved for a bad -> invocation. A pipeline that tested `[ $? -eq 2 ]` to mean "partially applied" -> must now test `-eq 3`. Checks of the form `if ! noorm …` are unaffected. - - -## CI/CD Examples - - -### Test CI (GitHub Actions) - -Stateless, ephemeral. No identity or `ci init` needed. - -```yaml -jobs: - test: - runs-on: ubuntu-latest - services: - postgres: - image: postgres:16 - env: - POSTGRES_PASSWORD: test - POSTGRES_DB: test_db - ports: - - 5432:5432 - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: '22' - - run: npm ci - - name: Apply schema and changes - env: - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: localhost - NOORM_CONNECTION_DATABASE: test_db - NOORM_CONNECTION_USER: postgres - NOORM_CONNECTION_PASSWORD: test - run: | - npx noorm run build - npx noorm change ff - - run: npm test -``` - - -### Prod CI (GitHub Actions, vault-enabled) - -**One-time developer setup:** -```bash -noorm ci identity enroll --config prod --name "GitHub CI" --email ci@example.com -# → prints NOORM_IDENTITY_PRIVATE_KEY / NAME / EMAIL; copy to GitHub Secrets -``` - -**Pipeline:** -```yaml -name: Deploy Database Changes -on: - push: - branches: [main] - -jobs: - deploy: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: '22' - - run: npm ci - - - name: Bootstrap CI state - env: - NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} - NOORM_IDENTITY_NAME: "GitHub CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} - NOORM_CONNECTION_DATABASE: ${{ secrets.DB_NAME }} - NOORM_CONNECTION_USER: ${{ secrets.DB_USER }} - NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_PASSWORD }} - run: npx noorm ci init --name prod - - - name: Apply changes - run: npx noorm change ff -``` - - -### Prod CI with Batch Secrets - -When templates need secrets beyond the connection: - -```yaml -- name: Bootstrap CI state - env: - NOORM_IDENTITY_PRIVATE_KEY: ${{ secrets.NOORM_CI_KEY }} - NOORM_IDENTITY_NAME: "GitHub CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: ${{ secrets.DB_HOST }} - NOORM_CONNECTION_DATABASE: ${{ secrets.DB_NAME }} - NOORM_CONNECTION_USER: ${{ secrets.DB_USER }} - NOORM_CONNECTION_PASSWORD: ${{ secrets.DB_PASSWORD }} - run: npx noorm ci init --name prod - -- name: Load secrets - env: - API_KEY: ${{ secrets.API_KEY }} - STRIPE_KEY: ${{ secrets.STRIPE_KEY }} - run: | - cat > ./ci-secrets.env <<EOF - API_KEY=$API_KEY - STRIPE_KEY=$STRIPE_KEY - EOF - npx noorm ci secrets --file ./ci-secrets.env - rm ./ci-secrets.env - -- name: Apply changes - run: npx noorm change ff -``` - - -### GitLab CI - -```yaml -migrate: - stage: deploy - image: node:22 - script: - - npm ci - - npx noorm ci init --name prod - - npx noorm change ff - variables: - NOORM_IDENTITY_PRIVATE_KEY: $NOORM_CI_KEY - NOORM_IDENTITY_NAME: "GitLab CI" - NOORM_IDENTITY_EMAIL: "ci@example.com" - NOORM_CONNECTION_DIALECT: postgres - NOORM_CONNECTION_HOST: $DB_HOST - NOORM_CONNECTION_DATABASE: $DB_NAME - NOORM_CONNECTION_USER: $DB_USER - NOORM_CONNECTION_PASSWORD: $DB_PASSWORD +Diagnostics for `$helpers` resolution and worker threads in a compiled binary. + +```bash +noorm dev test-helpers db/sql/09_seeds/01_products.sql.tmpl +noorm dev test-workers --json ``` +Both accept `--json`. Exit 0 when everything passes, 3 when some pass and some fail, 1 when none pass. `test-helpers` prints its text report to stdout even under `--json`, so its output is not a single JSON document. -### Generic CI Script with Locking + +## How do I script a pipeline? + + +A developer enrolls the CI identity once. Every job then builds its own state from environment variables: + +```mermaid +sequenceDiagram + participant Developer + participant CI as CI job + participant Database + Developer->>Database: ci identity enroll --config prod --yes + Database-->>Developer: private key, shown once + Developer->>CI: store the identity variables as CI secrets + CI->>CI: ci init --name prod + CI->>Database: change ff + Database-->>CI: JSON result and exit code +``` + +The same job as a script. `|| true` covers the lock `change ff` already released: ```bash #!/bin/bash @@ -1764,107 +1433,53 @@ export NOORM_CONNECTION_USER=$DB_USER export NOORM_CONNECTION_PASSWORD=$DB_PASSWORD noorm ci init --name prod -noorm lock acquire -trap "noorm lock release" EXIT -noorm change ff -noorm db explore --json +noorm lock acquire --reason "deploy $GITHUB_SHA" +trap 'noorm lock release || true' EXIT +noorm change ff --json ``` - -## Output Streams - -A command's **result** — the human summary, or the `--json` payload — always goes to -**stdout**. Everything else — progress lines, warnings, the logger's event stream — goes to -**stderr**, in both text and `--json` mode. This holds regardless of whether the command -succeeds or fails: a setup failure (bad config, no connection) prints nothing to stdout in -text mode — the diagnostic goes to stderr — and a single JSON error document to stdout in -`--json` mode, e.g. `{"success":false,"error":"..."}`. - -That means `--json` output is always exactly one parseable JSON document on stdout, with no -log noise ahead of or after it — no `tail -1` needed: +GitHub Actions and GitLab versions are in [CI/CD automation](/guide/automation/ci). Snippets for reading results: ```bash -noorm change history --json | jq '.history[0].status' -``` - -A command that finds nothing to report still writes a result — never silence. `noorm change -list` on a project with no changesets prints `No changes found.` on stdout in text mode and -`{"success":true,"changes":[],"pending":0}` in `--json` mode; either way, a CI step can tell -"ran and found nothing" from "didn't run" by checking the exit code and the (always-present) -stdout line, rather than guessing from empty output. +# Apply only when something is pending +[ "$(noorm change list --json | jq '.pending')" -gt 0 ] && noorm change ff +# One failure check for every command +noorm run build --json | jq -e '.success' > /dev/null || echo "build did not succeed" -## The `--json` Envelope +# Grant vault access to anyone waiting +[ "$(noorm vault list --json | jq '.status.usersWithoutAccess')" -gt 0 ] && noorm vault propagate --yes -Every `--json` payload is a **JSON object** carrying a top-level boolean `success`. -It is never a bare array. Command-specific fields sit alongside `success`, and -list results live under a key named for the resource: - -```json -{ "success": true, "changes": [ … ], "pending": 0 } -{ "success": true, "status": "success", "filesRun": 3, "filesSkipped": 0, "filesFailed": 0 } -{ "success": false, "status": "partial", "filesRun": 1, "filesSkipped": 0, "filesFailed": 2 } -{ "success": false, "error": "Template not found: sql/nope.sql.tmpl" } +# Render a template and pipe the SQL onward +noorm run preview db/sql/hotfix.sql.tmpl | psql -h localhost -d app ``` -Three guarantees hold across every command: -1. `jq -e '.success'` works everywhere — success and failure alike. -2. `success` and the exit code always agree: `success` is `true` if and only if - the process exits `0`. A partial result reports `success: false` alongside - `"status": "partial"` and exit `3`. -3. An unsuccessful payload always carries `error` as a string. That string is a - message, never a stack trace — stack traces contain absolute filesystem - paths and stay out of machine-readable output. +## What bites -Commands that report per-unit outcomes also carry a `status` of `"success"`, -`"partial"`, or `"failed"`, which is what `success` and the exit code are derived -from. -> **Changed in this release.** `--json` success payloads previously had four -> incompatible shapes and no shared discriminator. Two changes are breaking: -> `success` is now present on payloads that had no such key, and the commands -> that returned a **top-level array** now return a named object instead — -> `change list` → `.changes`, `change history` → `.history`, `db explore tables` -> → `.tables`, `views` → `.views`, `indexes` → `.indexes`, `fks` → -> `.foreignKeys`, `functions` → `.functions`, `procedures` → `.procedures`, -> `types` → `.types`, `triggers` → `.triggers`. A script doing `jq '.[]'` on any -> of those must now name the key. Everything else gained a field and lost none. - - -## Scripting with JSON - -```bash -# Check for pending changes -pending=$(noorm change list --json | jq '.pending') -if [ "$pending" -gt 0 ]; then - echo "Found $pending pending changes" - noorm change ff -fi -``` +| Mistake or symptom | Cause | Fix | +|---|---|---| +| `noorm -c prod run build` fails: `--cwd path is not a directory` | Before the subcommand, `-c` is `--cwd` | `noorm run build -c prod` | +| `Unrecognized flag '--json' before the subcommand`, exit 2 | Only `-c`/`--cwd` is read before the subcommand | Move the flag after the subcommand | +| A script tests `[ $? -eq 2 ]` for "partially applied" | Before noorm 1.0.0, `2` meant partial and total failure alike. Partial is `3` now | Test `-eq 3`. `if ! noorm ...` is unaffected | +| `jq '.[]'` on `change list` or `db explore tables` fails | Lists sit under a named key since 1.0.0 | `jq '.changes[]'`, `jq '.tables[]'` | +| `noorm init` exits 2 in CI | No terminal, and `--yes` needs an existing identity | `noorm identity init --name ... --email ...`, then `noorm init --yes`, or use `ci init` | +| A `confirm` command fails under Claude Code, Codex, Cursor, or Gemini CLI even with `--yes` | The `agent` channel turns every `confirm` cell into a deny | Raise the config's `agent` role where the command is allowed, or run as a human with `NOORM_CHANNEL=user`. `noorm info` shows the detected harness | +| `noorm sql -c prod "SELECT 1"` prints help | The bare form rewrites only when the first non-flag token is SQL | `noorm sql query -c prod "SELECT 1"` | +| `config add` or `config edit` exits 1 | Both are wizard-only | `noorm ui`, or `config import <file>` with `--force --yes` | +| `noorm lock release` exits 1 after `change ff` | Each change takes and releases its own lock, so the lock from `lock acquire` is gone after the first change | `noorm lock release \|\| true` | +| The SDK's `requireTest` guard rejects the config `ci init` made | `ci init` sets `isTest: true` only with `--test` | Pass `--test` for a throwaway database, not for production | +| `tmp/` holds secrets after `run build --dry-run` | Rendered files contain resolved secrets in plaintext, owner-only, and `tmp/` is not gitignored | Delete `tmp/`, or add it to `.gitignore` | +| `change ff` fails on a freshly built database: object already exists | `run build` leaves no change records, so the first `ff` runs every change | Write changes that are safe to run twice | +| `dev test-helpers --json` does not parse | It prints a text report before the document | Use `run inspect --json` for helper data | -```bash -# One failure check that works against every command -if ! noorm run build --json | jq -e '.success' > /dev/null; then - echo "build did not succeed" -fi -``` -```bash -# Get table count -tables=$(noorm db explore --json | jq '.tables') -echo "Database has $tables tables" -``` +## Next -```bash -# Verify vault access -pending=$(noorm vault list --json | jq '.status.usersWithoutAccess') -if [ "$pending" -gt 0 ]; then - noorm vault propagate --yes -fi -``` -```bash -# Render template and pipe to external tool -noorm run preview sql/hotfix.sql.tmpl | psql -h localhost -d myapp -``` +- [Non-interactive operation](/guide/automation/non-interactive): `--yes`, `NOORM_YES`, and which commands need a terminal +- [CI/CD automation](/guide/automation/ci): pipelines for GitHub Actions and GitLab +- [CLI flag conventions](/cli/flags): why `-c` means two things +- [Stages](/guide/environments/stages): defaults and `protected` caps for configs +- [MCP](/guide/automation/mcp): the same commands for AI agents diff --git a/docs/index.md b/docs/index.md index cdbf5ede..59e400b7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,166 +1,195 @@ --- layout: home -title: noorm — Write SQL. Skip the ORM. +title: noorm — A database change manager for the modern era titleTemplate: false -description: A SQL-first schema and change manager for Postgres, MySQL, SQLite, and MSSQL. Your schema lives in SQL files. noorm builds it, versions it, and keeps every environment in sync. - -hero: - name: noorm - text: "Write SQL. Skip the ORM." - tagline: "The raw SQL manager people keep asking for. Schema in files, changes in git, one CLI to prod." - actions: - - theme: brand - text: "Get started →" - link: /getting-started/installation - - theme: alt - text: View on GitHub - link: https://github.com/noormdev/noorm - -features: - - icon: - src: /icons/database.svg - title: Schema lives in SQL files - details: Your SQL files are the current schema. Fresh databases build from them in seconds. Existing ones catch up through versioned changes. - link: /getting-started/concepts - linkText: How it works - - icon: - src: /icons/cubes.svg - title: Real relational design - details: Compound keys, inherited keys, check constraints, subtype clusters. Model what your data actually is instead of one surrogate ID per table. - link: /guide/relational-design - linkText: The case for it - - icon: - src: /icons/git-branch.svg - title: Procedures, functions, TVFs - details: "Call stored procedures from TypeScript with typed params and result rows. Table-valued parameters included. The bridge ORMs never built." - link: /reference/sdk#stored-procedures-functions-tvfs - linkText: SDK reference - - icon: - src: /icons/fast-forward.svg - title: Data pipelines - details: Move data between databases in foreign-key order, across dialects, or out to portable files. Seed a QA environment from staging in one command. - link: /guide/database/transfer - linkText: Transfer data - - icon: - src: /icons/lock.svg - title: Safe to point an agent at - details: An MCP server exposes noorm to coding agents behind per-channel access roles. Admin at your terminal, read-only for the agent, invisible for prod. - link: /guide/automation/mcp - linkText: AI integration - - icon: - src: /icons/toolbox.svg - title: One CLI, not five tabs - details: Schema explorer, SQL terminal, encrypted vault, dynamic templates. Every command runs headless and emits JSON you can pipe into CI. - link: /headless - linkText: CLI reference +description: noorm keeps one SQL file per table as the schema you have today. Fresh databases build from those files; existing ones catch up through tracked changes. Postgres, MySQL, SQLite, and SQL Server. --- +<section class="nh-hero"> -## Why noorm? +<p class="nh-eyebrow">Postgres · MySQL · SQLite · SQL Server</p> -Migration tools make you describe your schema twice: once in the migrations that built it, and once in your head. The current state only exists if you replay every file in order, and the moment you need a compound key or a trigger you are writing raw SQL inside a wrapper that was designed to keep you away from it. +<h1 class="nh-title">A database change manager<br><span>for the modern era</span></h1> -noorm inverts that. **Your SQL files are the current schema.** A fresh database runs them and is done. An existing database gets to the same place through **changes** — small forward/revert pairs that noorm tracks, checksums, and applies in order. +<div class="nh-row"> +<p class="nh-lead">Your schema lives in <code>db/sql/</code>, one file per table, as it stands today. A fresh database builds straight from those files. An existing one catches up through tracked changes you can revert.</p> +<div class="nh-actions"> +<a class="nh-btn brand" href="/getting-started/installation">Get started →</a> +<a class="nh-btn alt" href="/download">Download</a> +<p class="nh-caption">Write SQL. Skip the ORM. · one binary · Apache 2.0</p> +</div> +</div> -Everything else follows from that split: +<div class="nh-compare"> +<div class="nh-panel nh-bad"> +<div class="nh-panel-head"><span><b class="nh-mark">✕</b> migrations/</span><span>what changed</span></div> +<pre class="nh-panel-body">0001_create_party.sql +0002_create_sales_order.sql +0003_add_email_to_party.sql +0004_create_order_line.sql +… +0212_rename_total_to_amount.sql +0213_add_status_to_sales_order.sql +0214_backfill_sales_order_status.sql</pre> +<p class="nh-panel-foot">To learn what <code>sales_order</code> looks like, read all 214.</p> +</div> +<div class="nh-file nh-good"> +<div class="nh-panel-head"><span><b class="nh-mark">✓</b> db/sql/01_tables/02_sales_order.sql</span><span>what exists</span></div> + +```sql +CREATE TABLE sales_order ( + party_id integer NOT NULL, + order_no integer NOT NULL, + placed_at timestamptz NOT NULL DEFAULT now(), + status text NOT NULL DEFAULT 'open', + CONSTRAINT sales_order_pk + PRIMARY KEY (party_id, order_no), + CONSTRAINT sales_order_isplacedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) +); +``` -- **Stages** keep dev, staging, and production configs apart, with access roles per environment -- **Templates** let one SQL file render differently per environment -- **The SDK** wraps it all in a type-safe client — Kysely queries, stored procedures, and TVFs -- **Headless mode** makes every command scriptable, with `--json` on anything worth parsing +</div> +</div> -noorm is the fifth attempt at this problem, and every feature in it exists because working without an ORM was worse without it. [Why noorm](/why-noorm) is the longer version of that story. +</section> +<section class="nh-section"> -## Quick start +<p class="nh-eyebrow">What changes</p> -```bash -# Install (no sudo needed) -curl -fsSL https://noorm.dev/install.sh | sh +<h2 class="nh-h2">The problems you stop having</h2> -# Or via npm -npm install -g @noormdev/cli +<div class="nh-rows"> +<div class="nh-r"> +<p class="nh-pain">A test database replays the whole migration history before the first test runs.</p> +<p class="nh-fix">It builds from the current files in one command.</p> +<code class="nh-proof">noorm run build</code> +</div> +<div class="nh-r"> +<p class="nh-pain">A stored procedure was hotfixed in a SQL editor, and nobody can say which version is live.</p> +<p class="nh-fix">Procedures, views, and functions re-run through a change. History records who ran it, when, and the error if it failed.</p> +<code class="nh-proof">noorm change history</code> +</div> +<div class="nh-r"> +<p class="nh-pain">A password or API key the database needs sits in a SQL file in git.</p> +<p class="nh-fix">Store it encrypted with noorm. The template fills it in when the SQL runs, so the committed file holds only the reference.</p> +<code class="nh-proof">{%~ $.quote($.secrets.GATEWAY_KEY) %}</code> +</div> +<div class="nh-r"> +<p class="nh-pain">Your tests mock the database because a real one is a chore to set up.</p> +<p class="nh-fix">The SDK runs tests against a real database, and refuses any config not marked as a test one.</p> +<code class="nh-proof">createContext({ requireTest: true })</code> +</div> +<div class="nh-r"> +<p class="nh-pain">You handed an AI agent a connection string and hope it reads before it writes.</p> +<p class="nh-fix">Each config sets a role per caller: read-only for the agent on dev, invisible on prod.</p> +<code class="nh-proof">noorm mcp serve</code> +</div> +</div> -# Bootstrap a project -cd /my/project && noorm init -``` +</section> -::: tip Corporate network? -If `noorm.dev` is blocked, install from the GitHub mirror: -```bash -curl -fsSL https://raw.githubusercontent.com/noormdev/noorm/master/install.sh | sh -``` -::: +<section class="nh-section nh-split"> +<div> -Write a SQL file, then build it: +<p class="nh-eyebrow">How it works</p> -```bash -mkdir -p sql/01_tables -echo "CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT);" > sql/01_tables/001_users.sql +<h2 class="nh-h2">Two folders, two commands</h2> -noorm run build -``` +<p>Edit the table in <code>db/sql/</code>. Add a change for databases that already exist. Every environment ends up in the same state.</p> -``` -✓ Executed 1 file -``` +<pre class="nh-flow" role="img" aria-label="A fresh database runs noorm run build on db/sql/; an existing database runs noorm change ff on db/changes/; both reach the current schema."><span class="nh-dim">fresh database</span> db/sql/ <span class="nh-arrow">── noorm run build ──▶</span> current +<span class="nh-dim">existing database</span> db/changes/ <span class="nh-arrow">── noorm change ff ──▶</span> current</pre> -When the schema evolves, update the SQL file **and** add a change. The file keeps describing what the schema is; the change tells existing databases how to catch up: +<p class="nh-links"><a href="/getting-started/first-build">Quick start</a> · <a href="/guide/changes/add-a-column">Add a column to a live database</a></p> -```bash -# Edit sql/01_tables/001_users.sql to add an email column -# Add changes/2024-01-add-email/forward.sql +</div> +<div> -noorm change ff # fast-forward: apply pending changes +```bash +# a new test database +$ noorm run build +• log Build completed status=success filesRun=11 filesSkipped=0 filesFailed=0 durationMs=96.417 ``` -A fresh database skips all of that — it just runs the files: - ```bash -noorm config use test -noorm run build # fresh DB gets the current schema directly +# staging, which already has data +$ noorm change ff -c staging +• log Fast-forward success executed=1 skipped=0 failed=0 +• log 2026-10-20-order-status (success) ``` -**SQL files = current schema. Changes = how existing databases get there.** +</div> +</section> + +<section class="nh-section nh-band"> +<div class="nh-tool"> -Setup wizards (`config add`, `config edit`, secret management) run in the TUI — launch it with `noorm ui`. Everything else runs headless: +<p class="nh-eyebrow">ignatius · data modeling</p> -```bash -noorm run build # Build the schema from SQL files -noorm change ff # Apply all pending changes -noorm db explore --json # Inspect the database as JSON -noorm vault set API_KEY ... # Push a team secret to the encrypted vault -``` +<h2 class="nh-h2 nh-big">Agree on the data model<br><span>before anyone writes the SQL</span></h2> +<div class="nh-row"> +<p class="nh-lead">ignatius turns a folder of markdown into an IDEF1X entity diagram, a searchable data dictionary, and data flow diagrams. It reads no database and writes no SQL. The model lives in your repo and changes in the same pull request as the schema.</p> +<div class="nh-actions"> +<a class="nh-btn alt" href="/download">Download</a> +<a class="nh-btn alt" href="/modeling/">Explore ignatius</a> +<p class="nh-caption">one binary · works with or without noorm</p> +</div> +</div> -## Plan it before you build it +<div class="nh-dfd"> -Most schemas get designed while they are being built, one migration at a time, and nobody outside the code can see the shape until it is already in production. +<h3 class="nh-h3">Where your data comes from, and where it goes</h3> -[ignatius](/modeling/) is the planning half. Describe your entities and processes in markdown and it renders an IDEF1X entity diagram, a searchable data dictionary, and SSADM data flow diagrams that show how data moves between people, the database, caches, files, and paper. Export it to a single HTML file, get agreement from the people who care, then hand the same markdown to your agent as the specification for the SQL. +<p class="nh-dfd-lead">A data flow diagram shows who sends data, including people outside the technical system, which process moves or transforms it, and which store keeps it. Most teams skip it and draw logic flowcharts, which show decisions, not data. ignatius draws data flow diagrams in the SSADM style.</p> -```bash -curl -fsSL https://raw.githubusercontent.com/noormdev/ignatius/main/install.sh | sh -ignatius serve ./models -o -``` +<!--@include: ./parts/ignatius-dfd.md--> -ignatius decides what the data is. noorm builds it. Both keep the source of truth in files you own. See [Information modeling](/modeling/). +</div> +<!--@include: ./parts/ignatius-viewer.md--> -## Next steps +<!--@include: ./parts/ignatius-problems.md--> -<div class="next-steps"> +<!--@include: ./parts/ignatius-cmds.md--> -[**Installation**](/getting-started/installation) -Get noorm installed and running. +</div> +</section> -[**First Build**](/getting-started/first-build) -Complete the tutorial and see the core value. +<section class="nh-section"> -[**Building Your SDK**](/getting-started/building-your-sdk) -Create a type-safe database package for your apps. +<p class="nh-eyebrow">FAQ</p> -[**Information Modeling**](/modeling/) -Design the data model before you write the schema. +<h2 class="nh-h2">Questions developers ask first</h2> +<div class="nh-faq"> +<div><h3>Is this migrations with extra steps?</h3><p>Your migrations become <code>db/changes/</code>: forward and revert files, tracked and checksummed. <code>db/sql/</code> sits beside them as the current schema, so a fresh database builds from it without replaying history.</p></div> +<div><h3>Does it replace my ORM?</h3><p>It replaces the ORM's schema and migration layer. Query with the typed SDK (Kysely underneath), raw SQL, or the client you already use.</p></div> +<div><h3>Which databases?</h3><p>PostgreSQL, MySQL, SQLite, and SQL Server. The CLI is one binary with every driver built in.</p></div> +<div><h3>What about stored procedures?</h3><p>Call them from TypeScript with typed parameters and result rows, table-valued parameters included.</p></div> +<div><h3>Do I need both tools?</h3><p>No. ignatius reads markdown and never connects to a database, so it works with any stack. noorm runs your SQL whether or not a model exists.</p></div> +<div><h3>Does ignatius generate the SQL?</h3><p>No. It shows what you intend to build. Write the SQL yourself or hand the model to your agent; <a href="/modeling/model-to-sql">From model to SQL</a> shows how each field maps.</p></div> </div> + +</section> + +<section class="nh-section nh-start"> + +<h2 class="nh-h2">If you write SQL by hand and ship it to more than one environment, start here.</h2> + +```bash +# noorm: build and change the database +curl -fsSL https://noorm.dev/install.sh | sh + +# ignatius: model it first +curl -fsSL https://noorm.dev/ignatius.sh | sh +``` + +<div class="nh-actions nh-inline"> +<a class="nh-btn brand" href="/getting-started/installation">Get started →</a> +<a class="nh-btn alt" href="/download">Download</a> +</div> + +</section> diff --git a/docs/modeling/best-practices.md b/docs/modeling/best-practices.md index bdde8fca..ba5614cc 100644 --- a/docs/modeling/best-practices.md +++ b/docs/modeling/best-practices.md @@ -6,144 +6,100 @@ description: How to produce an ignatius model that survives a stakeholder review # Modeling best practices -A model earns its keep when someone who does not write SQL can read it, disagree with it, and be right. These are the habits that get you there. - - -## Model the decision, not the storage - - -The question a model answers is what the data is and how it is identified. Column types are logical for that reason: `text`, `integer`, `decimal`, `boolean`, `date`, `datetime`, `binary`. Whether a column becomes `VARCHAR(255)` or `TEXT`, which index it gets, and how it partitions are build-time decisions that belong in the SQL files noorm runs. - -Keeping those out of the model is what makes it reviewable by the people whose sign-off you need. - - -## Write the body - - -The frontmatter is the structure. The body is the reason the entity exists, and it is the part of the model that a schema dump can never give you. - -Say what the thing is in business terms, what it is not, and what rule made you draw the boundary where you did. The reference model's `Party` entity explains why customers, vendors, and employees are one entity rather than three: a company that is both a customer and a vendor should not have two identities, and a person who becomes a customer after being an employee keeps one record. That paragraph is what a stakeholder argues with. The column list is not. - -Link related entities in prose with `[[Entity]]`. A wiki link that points at a nonexistent entity renders muted and raises `body.unknown_link`, so the prose stays honest as the model changes. - - -## Always include examples - - -Add two or three realistic rows to every entity and input and output tables to every process. The modeling skill does this by default, and it is worth keeping when you author by hand. - -Structural validation cannot catch a wrong rule, only a broken reference. Concrete instances can. Write the rows for a case you believe is legal, and if one of them looks wrong when you read it back, the rule is wrong. This is the cheapest modeling test that exists, and it is the one that finds the errors that survive to production. - - -## Let derivation do its job - - -Do not declare `classification`, `identifying`, or cardinality. Describe the primary key, the columns, and the `on:` mapping, and let ignatius work out the rest. - -Two reasons. The derived answer stays correct when the key shape changes, and a hand-written label does not. And nothing reads those labels anyway: `identifying` and cardinality have no frontmatter field at all, and the only classification the parser accepts is `reference: true` for a lookup or code table. A `classification:` you set by hand is ignored without a finding, so it drifts out of agreement with the diagram in silence. - - -## Choose a key convention on purpose - - -Decide whether an entity inherits its parent's key or carries a surrogate `id`, and be able to say why. - -Inherited keys carry lineage in the identity, so a child cannot be attached to the wrong parent and the owning party is available without a join. They also make the lineage spotlight useful, since the feature works on shared-key families and has nothing to show on an all-surrogate schema. Surrogate keys are simpler to write and fit reference tables that nothing depends on for identity. - -Mixing conventions per entity is supported and often correct: a transactional spine that inherits keys alongside lookup tables that do not. Record the reason in the entity body so the next reader sees a decision instead of an inconsistency. - -When a team disagrees about this, serve the three reference variants that ship with ignatius (`key-inherited`, `orm-hybrid`, `orm-pure`) and compare them directly. They model the same business with the same entities, so the only variable is the keys. - - -## Name processes as verbs - - -A process transforms data. `Collect-Payment`, `Issue-Invoice`, and `Validate-Customer` read as work. A noun like `Payment-Handler` describes a component, and components belong in an architecture diagram rather than a DFD. - -The file name minus `.md` becomes the process id used in `proc:` tokens, wiki links, and the sub-diagram folder name, so pick it once and pick it well. - - -## Enumerate what a flow carries - - -On a `db:` endpoint the `data:` field is a column list, validated against the entity's keys and columns. On any other endpoint it is free text, and free text is where flows go vague. - -Write "gateway transaction reference, HTTP status, raw response" rather than "log data". The label is the contract for what crosses that arrow, and a reviewer can only catch a missing field if the field would have been named. Anything long is truncated on the canvas with an ellipsis and shown in full on hover, so a thorough label costs you nothing visually. - - -## Model the stores that are not tables - - -The caches, log files, queues, exported spreadsheets, signed forms, and PDFs are where the surprises live. They hold data nobody versioned, nobody backed up, and nobody remembered when the retention policy was written. +A model earns its keep when someone who does not write SQL can read it, disagree with it, and be right. + +The model is agreed before any SQL exists, so a wrong entity boundary costs a conversation instead of a migration. + +```mermaid +flowchart TD + A["write the model files"] --> B{"ignatius validate"} + B -->|"errors"| A + B -->|"clean"| C["ignatius export"] + C --> D{"stakeholder review"} + D -->|"disagrees"| A + D -->|"signs off"| E["agent writes db/sql/"] +``` -Give each one a `stores/<name>.md` with its kind and a body explaining why it exists, what is written to it, and what happens to it over time. "Append-only log of raw gateway responses, never read back during normal processing, retained for 7 years" is a compliance fact that exists in no schema and in no codebase. +## What belongs in the model? -## Decompose only where it earns it +- **Decisions, not storage.** Types are logical. Whether `text` becomes `varchar(255)`, which index it gets, and how it partitions belong in the SQL ([From model to SQL](/modeling/model-to-sql)). +- **The reason, in the body.** Frontmatter is the structure. The body is the argument a stakeholder reviews. The `party` body says every table in sales and billing inherits `party_id`, which is what stops one customer's payment from settling another customer's invoice. A schema dump cannot give you that sentence. +- **Two or three example rows per entity.** Validation catches a broken reference, not a wrong rule. A row that reads wrong when you read it back exposes the rule. +- **Stores that are not tables.** Caches, log files, queues, exported spreadsheets, and signed forms hold data nobody versioned. Give each a `stores/<name>.md` that says what is written to it, who reads it, and how long it is kept. -A sub-diagram is a folder named exactly after its parent process, and it must stay balanced with that parent: the data crossing its boundary has to match the parent's declared inputs and outputs, column for column on `db:` flows. `flow.unbalanced_decomposition` enforces that at every level. -That check is a feature, and it is also a cost. Decompose a process when the detail changes a decision, not to show completeness. Two levels that stay balanced beat four that drift. +## How do I choose keys and name relationships? -## Keep validate clean +- Describe `pk`, `columns`, and `on`. Leave `classification`, `identifying`, and cardinality out: ignatius derives them ([What gets derived](/modeling/entities#what-gets-derived)). +- Choose inherited or surrogate keys on purpose. Inherited keys stop a child attaching to the wrong parent and put the owning party on every row. Surrogate keys suit lookup tables and roots such as `party`. Record the reason in the entity body. +- When a team disagrees, serve the three variants of one model in the ignatius repository (`key-inherited`, `orm-hybrid`, `orm-pure` under `models/`). Only the keys differ. +- Name each relationship with a business act: `is placed by`, not `belongs to` ([What goes in a predicate?](/modeling/entities#what-goes-in-a-predicate)). -`ignatius validate` exits `1` on errors and `0` otherwise, and warnings alone do not fail it. Run it while authoring and wire it into CI next to your other checks. +## How do I write flows? -Treat warnings as a queue rather than noise. `entity.unknown_group`, `cluster.no_discriminator`, and `flow.unknown_attribute` each name a real gap: a group file you never wrote, an exclusive cluster with no way to tell its subtypes apart, a flow promising a column the entity does not have. The model that fails a review is usually the one that accumulated twenty warnings nobody read. -The `models/broken-demo` folder in the ignatius repo is deliberately broken to exercise every rule, which makes it a fast way to learn what each finding looks like before you meet it on your own model. +- **Processes are verbs.** `Place-Order`, `Issue-Invoice`, and `Collect-Payment` read as work. `Payment-Handler` names a component, which belongs in an architecture diagram. +- **Flow data lists fields.** Write "transaction reference, status, raw response", not "log data". A reviewer can only catch a missing field that was named. +- **Decompose only where the detail changes a decision.** A sub-diagram must balance with its parent, and `flow.unbalanced_decomposition` checks it at every level. Two levels that stay balanced beat four that drift. ## Review before you build -Export the model and put it in front of someone. - ```bash -ignatius export ./models -o model.html +ignatius validate model +ignatius export model -o model.html ``` -One self-contained HTML file with the graph, the dictionary, and the flows, working offline with nothing installed. Attach it to the ticket, mail it to the stakeholder, or commit it beside the model. - -Put your own name on it first. The default logo, title, and copyright line are placeholders, and a document going to a client or an approver should carry your organization rather than the tool's. See [Branding your model](/modeling/branding). - -This is the step the whole tool exists for. A wrong entity boundary costs a conversation now and a migration later. Get the disagreement out of people before the schema is real, then hand the agreed model to your agent as the specification for the SQL. +`validate` exits `1` on errors and `0` otherwise, so warnings alone do not fail it. Read them anyway: `entity.unknown_group` is a group file you never wrote, `cluster.no_discriminator` is an exclusive cluster that cannot tell its members apart, and `flow.unknown_attribute` is a flow promising a column the entity lacks. +`export` writes one HTML file holding the graph, the dictionary, and the flows. It works offline, so attach it to the ticket or mail it. Set `branding` in `ignatius.yml` first, because the default logo, title, and copyright are the tool's own ([Branding your model](/modeling/branding)). Then give the agreed model to your agent as the specification for the SQL. -## Keep the model next to the code +When an agent edits the model, review the edit as a diff. Record the baseline before it starts: +```bash +ignatius approve model +``` -Commit the model folder in the repository whose schema it describes. It diffs cleanly, so a schema change and its model change land in one reviewable commit, and the prose explaining why gets reviewed alongside the DDL. +``` +Approved 0 change(s), cleared 0 note(s). +``` -A model that lives in a wiki or a drawing tool drifts within a quarter. One that lives in the repo and gates CI does not. +Then open `ignatius serve model`. It marks every added, changed, and removed entity, column, relationship, and flow node since the last approval, and lists each beside the change note that explains it. Run `approve` again to accept the edit. The [modeling skill](/modeling/modeling-skill) writes the note. +Commit the model folder in the repository whose schema it describes. The `.gitignore` ignatius writes keeps `.ignatius/` out. A schema change and its model change land in one reviewable commit, and CI can run `validate` next to your other checks. -## A model that follows all of this +## Where can I see a finished model? -Reading a finished model beats reading rules about one. [`models/llm-memory-db-mssql`](https://github.com/noormdev/ignatius/tree/main/models/llm-memory-db-mssql) in the ignatius repo is the memory database an AI coding agent writes to, and it is the model in the [demo recording](/modeling/#information-modeling). ```bash git clone https://github.com/noormdev/ignatius.git ignatius serve ignatius/models/llm-memory-db-mssql -o ``` -It runs 38 entities across 8 groups, 27 processes in 6 diagrams, and one external actor, and `ignatius validate` reports no findings. Things worth looking at while it is open: +It validates clean and mixes keys on purpose: `Agent` and `Artifact` are surrogate roots, while `Milestone`, `Task`, and `Task_Artifact` inherit `milestone_id` and `task_no` down the chain. + + +## What bites -- **Every entity and every process carries examples.** All 38 and all 27, with no exceptions, which is what the practice above asks for and what most models skip first. -- **The key conventions are mixed on purpose.** `Agent` and `Artifact` are surrogate roots, while `Milestone` → `Task` (`milestone_id, task_no`) → `Task_Artifact` (`milestone_id, task_no, artifact_id`) inherits the whole way down. The transactional spine inherits; the things with independent identity do not. -- **Classification is visible, not declared.** Reference tables, associatives like `Memory_Tag`, and the `StateTransition` subtype cluster all get their shape from their keys. -- **The bodies carry business rules.** `Agent` explains why rows default `agent_id` to a sentinel: memories written before an agent existed, or after one is deleted, stay intact instead of being orphaned. No schema records that. -Two practices it does not exercise, because the domain has no occasion for them: it has no non-`db` stores and no sub-diagrams. For those, read [`models/key-inherited`](https://github.com/noormdev/ignatius/tree/main/models/key-inherited), which validates clean at 24 entities and has both a `file:` store with a retention policy in its body and a decomposed process with a balanced child diagram. +| Mistake or symptom | Cause | Fix | +|---|---|---| +| Review comments argue about `varchar` lengths | physical types sit in the model | keep types logical and decide them in the SQL | +| Warnings pile up unread | `validate` exits `0` on warnings | read each one, because each names a real gap | +| The export carries the tool's logo and title | `branding` is unset | set it before the file leaves your team | +| A reviewer cannot tell what crosses an arrow | `data:` on a non-table flow says "log data" | list the fields | -## Related +## Next - [Entities and key inheritance](/modeling/entities) - [Data flows](/modeling/data-flows) - [The modeling skill](/modeling/modeling-skill) +- [Branding your model](/modeling/branding) diff --git a/docs/modeling/branding.md b/docs/modeling/branding.md index 9400d61b..5250b81d 100644 --- a/docs/modeling/branding.md +++ b/docs/modeling/branding.md @@ -1,25 +1,15 @@ --- title: Branding your model -description: Put your own logo, title, and copyright on an ignatius model. The default chrome is a placeholder, not a claim on your work. +description: Put your own logo, title, and copyright on an ignatius model. The default chrome is noorm's placeholder, so replace it before you export a file for someone else. --- # Branding your model - -An ignatius model is yours. The entities, the prose, the flows, and the diagrams generated from them are your work product, and nothing about using the tool changes that. - -Out of the box the app wears noorm's chrome: a noorm logo, the title "Noorm Ignatius", the subtitle "Visualize your data model", and a footer reading "© 2026 Noorm Ignatius" above a "powered by Noorm" line, with the year taken from the current date. That is placeholder branding on a viewer you are expected to replace, not a claim on the model it displays. The copyright line exists so you have somewhere to put **your** organization, which is the point of it being configurable at all. - -Replace all of it in `ignatius.yml`. - - -## The branding block - - -Every field is optional and falls back to the default. +Put your organization's logo, title, and copyright on the model, so the file you export carries your name. ```yaml -name: My Schema +# models/acme/ignatius.yml +name: Acme Schema branding: logo: dark: ./assets/logo-dark.svg @@ -32,79 +22,72 @@ branding: poweredBy: false ``` -| Field | Default | Notes | -|---|---|---| -| `logo` | embedded noorm logo | One path applies to both modes, or use `{ dark, light }` for per-mode logos | -| `title` | `Noorm Ignatius` | Max 50 characters | -| `subtitle` | `Visualize your data model` | Max 50 characters | -| `copyright` | `{ holder: 'Noorm Ignatius', year: <current year> }` | Set `holder` to your organization | -| `poweredBy` | `true` | Set `false` to drop the footer attribution | +```bash +ignatius export models/acme -o acme-data-model.html +``` -Setting `copyright.holder` and `poweredBy: false` removes every mention of noorm from the rendered model. Nothing asks you to keep the attribution, and no feature is gated behind it. +``` +Wrote export to acme-data-model.html +``` -Logo paths resolve relative to the model root and are inlined as data URIs when you `export`, so a branded model stays a single self-contained HTML file with no external requests. +The model is your work product. Until you replace them, the app shows noorm's logo, the title "Noorm Ignatius", the subtitle "Visualize your data model", a footer `© <year> Noorm Ignatius`, and "powered by Noorm". The attribution is optional and gates no feature. -## Why this matters at export time +## What can I set? -`ignatius export` produces one HTML file you hand to somebody: a stakeholder approving the design, a client receiving a deliverable, an auditor reading a system description. That file is the artifact your organization's name belongs on. +Every field is optional and falls back to its default on its own. -```bash -ignatius export ./models -o acme-data-model.html -``` +| Field | Default | Notes | +|---|---|---| +| `logo` | embedded noorm logo | One path for both modes, or `{ dark, light }`. With one of the two, the other mode reuses it | +| `title` | `Noorm Ignatius` | At most 50 characters. The header cuts it with `…` past 200px, about 25 characters | +| `subtitle` | `Visualize your data model` | At most 50 characters. Cut past 240px, about 40 characters | +| `copyright` | `{ holder: 'Noorm Ignatius', year: <current year> }` | Set `holder` to your organization | +| `poweredBy` | `true` | `false` drops the footer attribution | + +A logo path is relative to the model root and can be `.svg`, `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp`. A `data:` or `http(s)` URL passes through. Logos are inlined, so a branded export stays one self-contained file. An export's tab title is the `name:` value. -Brand the model before that hand-off rather than after. It costs six lines of YAML and it is the difference between a document that reads as yours and one that reads as a tool's output. +To remove every noorm mark, set `logo`, `title`, `copyright.holder`, and `poweredBy: false`. -## Colors and spacing +## How do I change colors? -A `theme` block sets separate `dark` and `light` palettes plus layout spacing. It is deep-merged over the defaults, so set only what you want to change. +A `theme` block is deep-merged over the defaults, so set only what you change. The `dark` and `light` palettes take `background`, `surface`, `border`, `text`, `textMuted`, `edgeIdentifying`, `edgeReferential`, and `pastelMix`. `spacing.nodeSep` defaults to `60`. In a [data flow diagram](/modeling/data-flows), `theme.flowKinds` recolors a store kind (`cache`, `queue`, `file`, `doc`, `manual`, `other`, `db`) or `external` with `{ bg, fg, border }` per mode. ```yaml +# models/acme/ignatius.yml theme: dark: background: "#0e1116" - surface: "#161b22" - text: "#e6edf3" - light: - background: "#ffffff" - surface: "#f6f8fa" - text: "#1f2328" spacing: nodeSep: 60 -``` - -The viewer's light/dark toggle persists across reloads, and `export` takes `--theme light` or `--theme dark` to pick the starting palette. - - -### Data flow store colors - - -In a [data flow diagram](/modeling/data-flows) each store is colored by its kind. The defaults give every kind a distinct, mode-appropriate color: `cache` amber, `queue` violet, `file` lime, `doc` sky, `manual` rose, `other` slate, while `db` keeps the entity-store fill and `external` keeps the conventional green. - -Override any of them under `theme.flowKinds`. Each kind takes `dark` and `light` entries of `{ bg, fg, border }`, deep-merged like the rest of the theme. - -```yaml -theme: flowKinds: cache: dark: bg: "#3a2a00" light: bg: "#fff3c4" - file: - dark: - bg: "#16270a" ``` -Worth restraint here. The kind colors are a legend readers learn once, and a palette tuned for your brand rather than for distinguishability makes a busy diagram harder to read. Change them to fit your house style, not to make every store the same shade of corporate blue. +`ignatius export --theme light` starts the file in the light palette. + + +## What bites + + +| Symptom | Cause | Fix | +|---|---|---| +| A command exits `1` with `branding.title exceeds 50 characters (actual length: 68)` and a stack trace | `title` or `subtitle` is over 50 characters | Shorten it | +| The noorm logo still shows | The file is missing or has another extension, so `logo` falls back without a message | Check the path from the model root and the extension | +| The title ends in `…` | The header cuts it at 200px | Shorten it to about 25 characters | -## Related +## Next -- [Entities and key inheritance](/modeling/entities#the-folder-format) for the rest of `ignatius.yml` -- [Best practices](/modeling/best-practices#review-before-you-build) for the review hand-off this feeds -- The [themes and branding guide](https://github.com/noormdev/ignatius/blob/main/docs/guides/themes-and-branding.md) in the ignatius repo +- [Entities and key inheritance](/modeling/entities): the rest of `ignatius.yml` +- [ignatius commands](/modeling/installation): `export` and its flags +- [Best practices](/modeling/best-practices): review the model before you hand it over +- [Themes and branding guide](https://github.com/noormdev/ignatius/blob/main/docs/guides/themes-and-branding.md): the reference in the ignatius repo diff --git a/docs/modeling/data-flows.md b/docs/modeling/data-flows.md index 6a6e57a0..6ec22090 100644 --- a/docs/modeling/data-flows.md +++ b/docs/modeling/data-flows.md @@ -1,181 +1,151 @@ --- title: Data flows -description: SSADM data flow diagrams in ignatius. Show how processes move data between people, the database, caches, files, and paper. +description: SSADM data flow diagrams in ignatius. Show where data comes from, including people outside the system, which process moves it, and which store keeps it, including stores that are not tables. --- # Data flows -An entity diagram tells you what the data is. It does not tell you who touches it, or where it rests when it is not in a table. +An entity diagram shows what the data is. A data flow diagram shows where it comes from, which process moves it, and where it rests, including the people and files that are not tables. -ignatius answers that with data flow diagrams in the SSADM style, using Gane-Sarson notation: numbered process hubs, green external boxes for actors outside the system, and open-ended `D#` boxes for stores. Flows live in the same folder as the entities, in the same markdown format, and a `db:` store in a diagram is the same entity you modeled in the graph. Click it and the full entity dialog opens, with columns, relationships, and examples. +| Part | Answers | Endpoint token | Example | +|---|---|---|---| +| External | who sends data in or receives it, outside the system | `ext:<Name>` | `ext:Customer` | +| Process | what moves or transforms it | a file in the diagram folder, or `proc:<Name>` at a sub-diagram boundary | `Collect-Payment.md` | +| Store, entity | where it rests as a table | `db:<entity>` | `db:payment` | +| Store, other | where it rests when it is not a table | `cache:` `queue:` `file:` `doc:` `manual:` `other:` | `file:gateway-log` | +Three process files, one external, and a log file that no table holds draw this diagram: -## Why the diagram includes more than tables +<ModelViewer class="tall" model="order-to-cash" hash="view=flow&dfd=order-to-cash" :tabs="false" label="order-to-cash data flows" title="Order-to-cash data flow diagram rendered by ignatius" /> +One file declares the arrows into and out of Collect payment: -The useful part of a DFD is that a data store is not required to be a database table. - -A cache holds derived state. A log file holds gateway responses nobody reads until a dispute. A queue holds work in flight. A signed paper form sits in a drawer, and a PDF sits in someone's mail. All of those are places your business keeps data, and all of them are places it can go missing, go stale, or leak. Model them and the diagram shows where information lives, including the parts that never reached the database. - -The same applies to people. An external entity is a source or a sink that sits outside the system boundary: a customer, a supplier, a regulator, a support agent. Drawing them makes the handoffs explicit, which is where most process defects are. - -Two distinctions are worth holding onto: - -- **Stores are inside the system, externals are outside it.** Both can be the source or the sink of a flow, but only a store represents state the system owns. -- **Every entity is a store, not every store is an entity.** A `db:` store resolves to a real entity in the graph. A cache, file, queue, document, or manual record is a store and nothing more. - - -## Folder layout - - -Flows live in a `flows/` folder at the model root. Each diagram is a folder, and each process is a file inside it. - -``` -models/ - ignatius.yml - data/ - identity/ ... # entities live under data/ - externals/ - Customer.md # shared external, usable by every diagram - stores/ - gateway-log.md # optional description of a non-entity store - flows/ - order-to-cash/ - Create-Sales-Order.md # process 1 - Create-Sales-Order/ # same-named folder is the sub-diagram - Validate-Customer.md - Record-Order.md - Issue-Invoice.md # process 2 - Collect-Payment.md # process 3 - refund/ - Process-Return.md -``` - -Files under `flows/` are never scanned as entities. The file name minus `.md` is the process id used everywhere: in `proc:` tokens, in wiki links, and as the sub-diagram folder name. Name it as an imperative phrase with hyphens for spaces, so `Collect Payment` becomes `Collect-Payment.md`. - - -## A process file - - -Frontmatter declares the data contract. The body explains the business. The arrows on the diagram are generated from `inputs:` and `outputs:`. +<div class="nh-file"> +<div class="nh-panel-head"><span>model/flows/order-to-cash/Collect-Payment.md (excerpt)</span><span>you write this</span></div> ```markdown --- -process: Collect Payment +process: Collect payment number: 3 inputs: - - from: ext:Customer - data: payment details - - from: db:PaymentMethod - data: [party_id, payment_method_id, type, label] + - from: ext:Customer + label: payment + data: card, amount + - from: db:invoice_line + label: open lines + data: [party_id, invoice_no, line_no, amount] outputs: - - to: db:Payment - data: [party_id, payment_method_id, payment_id, amount] - - to: file:gateway-log - data: gateway transaction reference, HTTP status, raw response - - to: ext:Customer - data: receipt -examples: - in: - - from: ext:Customer - label: payment details - rows: - - { card: "****4242", amount: 49.99, currency: GBP } - out: - - to: db:Payment - label: settled payment record - rows: - - { party_id: 1001, payment_method_id: 42, payment_id: 9001, amount: 49.99 } + - to: db:payment + label: payment + data: [party_id, payment_no, amount] + - to: db:payment_allocation + label: allocations + data: [party_id, payment_no, invoice_no, line_no, amount] + - to: file:gateway-log + label: gateway response + data: transaction reference, status, raw response + - to: ext:Customer + label: receipt + data: payment number, status --- -Settles an invoice by recording a [[Payment]] and allocating it against the -invoice line it pays. A receipt is returned to the [[Customer]]. +Writes the [[payment]] and its [[payment_allocation]] rows in one transaction. ``` -| Field | Required | Meaning | -|---|---|---| -| `process` | yes | The human label shown on the node | -| `number` | no | Local rank among sibling processes, falling back to file order. Full SSADM numbers like `1.2.1` are composed from the folder nesting | -| `inputs` / `outputs` | yes | The flows. Each names an endpoint and the `data:` it carries | -| `examples` | no | Sample in and out rows, rendered as tables in the process dialog, one entry per flow | - -The `data:` field is both the arrow's label and its contract. On a `db:` endpoint it is always column names, a string for one or a list for several, and every name is checked against the entity's `pk` and `columns` under the `flow.unknown_attribute` rule. On any other endpoint it is an opaque label, so make it enumerate what the flow carries rather than summarize it in one word. - -Bodies support the same `[[Entity]]` wiki links as entity files, and can also link to processes, externals, and stores by name. Links open the target's dialog in place. +</div> -## Endpoints +## What does a process file declare? -Every flow connects a process to something. The token's prefix says what. +The file name minus `.md` is the process id, an imperative phrase like `Collect-Payment`. The frontmatter is the data contract and the body explains the business. -| Token | Endpoint | +| Field | Meaning | |---|---| -| `db:<Entity>` | A store backed by a modeled entity. Must match an entity id exactly | -| `ext:<Name>` | An external entity, an actor outside the system boundary | -| `proc:<Name>` | Another process, used for sub-diagram boundary flows | -| `cache:` `queue:` `file:` `doc:` `manual:` `other:` | A non-entity store of that kind | +| `process` | The label on the node. Required | +| `number` | Rank among sibling processes. Without it, file order decides | +| `inputs`, `outputs` | The flows: an endpoint, a `label` for the arrow, and the `data` it carries | +| `examples` | Sample `in` and `out` rows, shown in the process dialog | -The prefix set is closed. A store that fits none of the named kinds is authored as `other:<name>`; there is no way to invent a prefix. A bare, unprefixed name resolves only when it is unambiguous across all namespaces, and otherwise raises `flow.ambiguous_endpoint` asking you to qualify it. +`data` is the contract. On a `db:` endpoint it is column names, and each is checked against the entity's `pk` and `columns`. On any other endpoint it is free text, so list what crosses: "transaction reference, status, raw response", not "log data". -### Externals +## What do externals and non-table stores need? -Describe an external once in `externals/<Name>.md` at the model root, with an `external:` label in frontmatter and a body covering its role, what it does, and what it expects back. Every diagram at any nesting depth can then reference `ext:<Name>`. There is no per-diagram override. +An `ext:` token needs `externals/<Name>.md`, a body saying who the actor is and what it expects back. A `db:` store needs no file, because the entity is the store. - -### Stores - - -A `db:` store needs no file. It is the entity, documented in the entity's own markdown. A non-`db` store exists by being referenced, and an optional `stores/<name>.md` adds a kind, a display title, and the reason it exists. +A store that is not a table exists by being referenced. An optional `stores/<name>.md` adds a `kind`, a `title`, and a body for facts no schema records: ```markdown --- kind: file -title: Payment Gateway Log +title: Payment gateway log --- -Append-only log of raw gateway responses. Used for reconciliation and dispute -resolution; never read back during normal processing. Retained for 7 years. +Raw responses from the payment gateway, appended by Collect payment. Read only for reconciliation. ``` -A store file's `kind:` is one of `cache`, `queue`, `file`, `doc`, `manual`, or `other`. An absent or unrecognized value falls back to `other`. `db` is not in that list, because a `db:` store is the entity and needs no file. Kind drives the node's color in the diagram, theme-aware and overridable under `theme.flowKinds` in `ignatius.yml`. +`kind` is one of `cache`, `queue`, `file`, `doc`, `manual`, or `other`. The viewer numbers stores by kind: `D1` for an entity, `F1` for a file, `C1` for a cache, and likewise `Q`, `Do`, `M`, and `O`. -Writing that body is where the value is. "Retained for 7 years" and "never read back during normal processing" are the facts that decide whether the store is a compliance asset or a liability, and they exist nowhere in a schema. +## How do I break a process into smaller ones? -## Sub-diagrams +Put a folder named exactly like the process file, minus `.md`, next to it. The folder holds the child diagram. -A process decomposes by placing a folder with the process's exact file name next to its file. The folder holds the child diagram, with its own process files numbered locally, and full dotted numbers composed from the nesting. Decomposition recurses as deep as it needs to. There is no depth cap, and the ancestor chain is preserved at every level, so a process four layers down reads as something like `1.4.2.1`. +``` +flows/order-to-cash/ +├── Place-Order.md +├── Place-Order/ the sub-diagram of Place-Order +│ ├── Validate-Customer.md +│ └── Record-Order.md +├── Issue-Invoice.md +└── Collect-Payment.md +``` -The child diagram must be balanced with its parent: the data crossing the sub-diagram's boundary has to match the parent process's declared inputs and outputs, column for column on `db:` flows. The `flow.unbalanced_decomposition` rule checks that at every level, which is the thing hand-drawn DFDs always get wrong once the diagram is more than one page. +Numbers compose from the nesting, with no depth limit. The child must balance with its parent: the data crossing its boundary matches the parent's inputs and outputs, column for column on `db:` flows. `flow.unbalanced_decomposition` checks every level. -In the viewer, a process with a sub-diagram renders with a stacked-shadow affordance. Clicking drills down, and a breadcrumb leads back up. +ignatius also derives two diagrams you do not write: the Context diagram, with every external around one `System` box, and the overview the Flows view opens on, with one box per diagram folder. `Place order` is numbered `1.1` in the viewer because it sits in the first folder. +A folder describes itself with `description:` frontmatter in `flows/<diagram>/index.md`, which `ignatius index` leaves intact. A sub-diagram uses its process's `description`. -## Viewing and validating +## How do I check it? -`ignatius serve` shows flows in the Flows view at `#view=flow`. The active diagram is deep-linkable through the `dfd=` hash parameter and survives a refresh. `ignatius export` writes the Flows view into the same single HTML file as the other two views. -Every node carries an ⓘ badge. A `db:` store opens the rich entity dialog; a process, external, or non-`db` store opens its markdown. The process dictionary, meaning every process, external, and store with its body and its input and output tables, is fused into the Dictionary view and searchable alongside the entities. +Press `f` in `ignatius serve` for the Flows view, and `i` for the flow index. Saving a file redraws it, and `#view=flow&dfd=order-to-cash` deep-links to a diagram. Hover an arrow to see everything crossing it. -Hovering an edge that carries data reveals a tooltip listing everything crossing it, under a `source → target` header. That includes the full contents of `db:` column lists which are abbreviated on the canvas when they run long. Labels over 22 characters show a truncated preview with an ellipsis, so you can always see at a glance which edges are hiding detail, and the tooltip stays legible at any zoom level. +`ignatius validate` runs the `flow.*` rules whenever `flows/` exists. Rename `payment_no` to `payment_id` in the `db:payment` output above and it warns: -`ignatius validate` runs twelve `flow.*` rules whenever a `flows/` directory exists, covering unknown references, column contracts, connection shape, numbering, store-token conflicts, and decomposition balance. One rule is configurable: direct process-to-process flows warn by default and can be silenced with `flow_rules: { process_to_process: false }` in `ignatius.yml`. +```bash +ignatius validate model +``` +``` +warn flow.unknown_attribute order-to-cash/Collect-Payment Column 'payment_id' on 'payment' not found in entity pk or columns. +✓ /home/you/shop/model: valid with 1 warning(s) across 11 entities. +``` -## Authoring with the skill +Warnings exit `0` and errors exit `1`. -The [`ignatius-modeling` skill](/modeling/modeling-skill) has two doors into flows. `/ignatius-modeling flow` walks you through a diagram step by step when you can already name your processes. `/ignatius-modeling discover` interviews you about how the business runs and generates both the entities and the flows. Both verify their output with `ignatius validate`. +## What bites + + +| Mistake or symptom | Cause | Fix | +|---|---|---| +| `flow.unknown_external`, the arrows vanish | every `ext:` token needs `externals/<Name>.md` | add the file or fix the name | +| `flow.illegal_connection`, the arrow vanishes | neither end is a process, as in a store sending to an external | route it through a process | +| A process is missing, with no finding | a file without a `process:` field is skipped | add `process:` | +| A sub-diagram is ignored | the folder name differs from the process file name, or it holds no `.md` | rename it to match | -## Related +## Next -- [Entities and key inheritance](/modeling/entities) covers the entities these diagrams reference. -- The [flows guide](https://github.com/noormdev/ignatius/blob/main/docs/guides/flows.md) and the [validation catalog](https://github.com/noormdev/ignatius/blob/main/docs/guides/validation.md) carry the exhaustive reference. +- [Entities and key inheritance](/modeling/entities): the entities these flows name +- [The modeling skill](/modeling/modeling-skill): `/ignatius-modeling flow` walks you through a diagram +- The [flows guide](https://github.com/noormdev/ignatius/blob/main/docs/guides/flows.md) covers clusters, stacks, and arrow labels +- The [validation catalog](https://github.com/noormdev/ignatius/blob/main/docs/guides/validation.md) lists every `flow.*` rule diff --git a/docs/modeling/entities.md b/docs/modeling/entities.md index c89a93d1..b5a3fec5 100644 --- a/docs/modeling/entities.md +++ b/docs/modeling/entities.md @@ -1,357 +1,246 @@ --- title: Entities and key inheritance -description: The ignatius entity format, what the tool derives from your key shape, and why IDEF1X key inheritance is the premise the whole tool is built on. +description: The ignatius entity file, how a child's key carries its parents' keys, and what ignatius derives from those keys so you never label an edge by hand. --- # Entities and key inheritance -IDEF1X is the premise ignatius is built on. In IDEF1X an entity is defined by how it is identified, and identity flows down through relationships: a child that depends on a parent carries the parent's key inside its own. That single idea decides the cardinality of an edge, whether the edge is identifying, and how the entity gets classified. ignatius implements that premise directly. You describe the key shape, and the notation follows. - -You can still model an ORM-style schema where every table has one surrogate `id`. ignatius draws that correctly too. It is not what the tool optimizes for. - -::: tip Modeling IDEF1X with an agent -IDEF1X is a methodology, not a file format. The calls it asks you to make (what counts as an entity, where identity comes from, which relationships are identifying) are the ones no tool can make for you. - -If you want an agent reasoning in those terms alongside you, there is a separate [`idef1x` skill](https://www.skills.sh/damusix/skills/idef1x) that installs with the same `skills` CLI as the modeling skill: - -```bash -npx skills add https://github.com/damusix/skills --skill idef1x +An entity file describes one table: its key, its columns, and the parents it points at. ignatius derives the rest of the diagram from the keys. + +A child's key contains its parent's key. That is the IDEF1X premise ignatius implements, and it is why every `order_line` names its party without a join. + +Each child carries its parents' key columns inside its own key. + +```mermaid +erDiagram + party ||--o{ sales_order : places + sales_order ||--o{ order_line : contains + product ||..o{ order_line : "is sold via" + party { + integer party_id PK + } + sales_order { + integer party_id PK, FK + integer order_no PK + } + order_line { + integer party_id PK, FK + integer order_no PK, FK + integer line_no PK + integer product_id FK + } + product { + integer product_id PK + } ``` -It advises on method. The [`ignatius-modeling` skill](/modeling/modeling-skill) writes the files. Neither is required to use ignatius, and the rest of this page assumes neither. -::: - ## The folder format -A folder is a model root when it contains an `ignatius.yml` file. Five top-level folders are recognized, and everything else at the root is left alone. +A folder with an `ignatius.yml` is a model. Entities go under `data/`. The subfolder is for your own organization, because an entity's group comes from its `group:` field. ``` -models/ - ignatius.yml - data/ - identity/ - Party.md - Person.md - Business.md - transactional/ - SalesInvoice.md - SI_Line.md - groups/ - identity.md - transactional.md - externals/ # optional, shared DFD externals - stores/ # optional, shared non-db DFD stores - flows/ # optional, data flow diagrams - notes/ # free-form, never scanned +model/ +├── ignatius.yml +├── data/ +│ ├── parties/ party.md, person.md, business.md, party_type.md +│ ├── sales/ sales_order.md, order_line.md, product.md +│ └── billing/ invoice.md, invoice_line.md, payment.md, ... +├── groups/ one file per group +├── externals/ Customer.md +├── stores/ gateway-log.md +├── clusters/ optional: named sets of stores for flows +└── flows/ + └── order-to-cash/ Place-Order.md, Issue-Invoice.md, ... ``` -Entities live under `data/`. The subdirectory is organizational convenience only: an entity's group comes from its `group:` frontmatter field, not from the folder it sits in. `groups/` is optional, and a model with no groups parses without error. - -`ignatius.yml` marks the root and carries display config. The minimum is one line. +`ignatius.yml` can be one line, `name: Order to cash`. `theme` and `branding` change how the model looks: see [Branding your model](/modeling/branding). `ignatius serve` and `ignatius approve` also write `.ignatius/`, the local change-review state, and a `.gitignore` entry that keeps it out of version control. -```yaml -name: My Schema -``` +A group file in `groups/` holds a `label` and a `color`. It tints the entities that name it and does not affect layout. -Add `version`, `description`, and `updated` for model metadata, a `theme` block to override colors and spacing, and a `branding` block for a logo, title, or copyright line. The model is yours, and the default chrome is a placeholder you are meant to replace. See [Branding your model](/modeling/branding). +## What does an entity file contain? -## An entity file +Frontmatter carries the structure. The body says why the entity exists. The viewer builds the attribute table from the frontmatter, so do not write one. -Frontmatter carries the structure. The body is free-form documentation. The attribute table you see in the viewer is generated from the frontmatter, so do not write one in the body. +<div class="nh-file"> +<div class="nh-panel-head"><span>model/data/sales/order_line.md (excerpt)</span><span>you write this</span></div> ```markdown --- -entity: SalesInvoice -group: transactional +entity: order_line +group: sales pk: - - party_id - - sales_invoice_id + - party_id + - order_no + - line_no columns: - party_id: - type: integer - desc: "Billed party, foreign key to Party." - sales_invoice_id: - type: integer - desc: "Identifier of the invoice within the party." - issued_at: - type: datetime - desc: "Timestamp the invoice was issued." - default: now - total: - type: decimal - desc: "Invoice total; reconciles to the sum of its line items." + party_id: + type: integer + order_no: + type: integer + line_no: + type: integer + product_id: + type: integer examples: - - party_id: 2 - sales_invoice_id: 5001 - issued_at: "2024-03-05T14:25:00Z" - total: 138.00 + - party_id: 2 + order_no: 1 + line_no: 1 + product_id: 10 relationships: - - target: Party - on: - party_id: party_id - predicate: { fwd: owes on, rev: is owed by } + - target: sales_order + on: + party_id: party_id + order_no: order_no + predicate: { fwd: contains, rev: is part of } + - target: product + on: + product_id: product_id + predicate: { fwd: is sold via, rev: sells } --- -# SalesInvoice - -A **SalesInvoice** is a bill issued to a [[Party]] for amounts owed. It is the -demand for payment, the document a `Payment` is ultimately applied against, -line by line. -``` - -Note what is absent: no `classification`, no per-edge `identifying`, no `cardinality`. ignatius derives all three. - +# order_line -### Columns - - -Each column takes a logical `type` and three optional fields. - -| Field | Default | Meaning | -|---|---|---| -| `type` | required | One of `text`, `integer`, `decimal`, `boolean`, `date`, `datetime`, `binary` | -| `nullable` | `false` | Whether the column accepts null | -| `default` | none | A default value note | -| `desc` | none | A short note on what the column is for | - -The types are logical, not physical. ignatius models information, so it does not care whether your `text` becomes `VARCHAR(255)` or `TEXT`. That decision belongs in the SQL files noorm runs. - - -### Relationships - - -A relationship names a `target` entity and maps foreign-key columns with `on: { child_column: parent_column }`. The `predicate` labels the edge in the graph and the dictionary. - -```yaml -relationships: - - target: Party - on: - party_id: party_id - predicate: is a +One product on a [[sales_order]]. The key carries the whole lineage, so the owning party is on every row without a join. ``` -A predicate can carry both reading directions with `{ fwd, rev }`, so the edge reads correctly whichever way you traverse it. - - -#### Lazy predicates and real ones +</div> +Only `entity` is required. A missing `pk` or `columns` raises a warning. The other fields: -In IDEF1X, entities are nouns and relationships are verbs. The predicate is where that verb goes, and it is the part of the model a tool cannot derive for you. Everything else on the edge comes from the keys. +| Field | Meaning | +|---|---| +| `group` | A file name in `groups/` | +| `columns` | Each takes a `type`, plus optional `nullable` (default `false`), `default`, `desc` | +| `ak` | Alternate keys: a `rule` naming the business rule, and its `columns` | +| `relationships` | The parents: `target`, `on: { child_column: parent_column }`, `predicate` | +| `subtypes` | A subtype cluster, declared on the basetype | +| `reference: true` | Marks a lookup table | +| `examples` | Sample rows keyed by column name | -`has many`, `belongs to`, and `has one` are the reflex answers, and they are all restatements of something already on the diagram. "Party has many SalesOrders" tells you the cardinality the crow's-foot marker is already showing. It says nothing about the business. The relationship exists because a party **places** orders, and that is the fact the schema is there to record. +`type` is logical: `text`, `integer`, `decimal`, `boolean`, `date`, `datetime`, `binary`, or `json` for a document with an open-ended shape. The SQL picks the physical type ([From model to SQL](/modeling/model-to-sql)). -| Lazy | What it restates | What the business actually says | -|---|---|---| -| `belongs to` | the foreign key | `is placed by`, `is owed by`, `is part of` | -| `has many` | the cardinality marker | `contains`, `journals`, `is broken down into` | -| `has one` | the key shape | `is realized as`, `is settled via` | +Write two or three realistic `examples` per entity. A row that looks wrong when you read it back means the rule is wrong, and no validation rule catches that. -The replacements are not synonyms for the lazy versions. `contains` and `is part of` assert composition, that the child is a constituent of the parent rather than merely pointing at it. `is sold via` and `is realized as` assert that one thing is the concrete form another takes. Those are different claims about the world, and picking between them is modeling work. +In a body, `[[party]]` links to that entity and `[[party|the buyer]]` changes the label. -#### The sentence has to ring true +## What goes in a predicate? -Read the relationship aloud as a sentence, in both directions. If it is not something a person who runs the business would say, and nod at, then either the predicate is wrong or the relationship is. +The business verb for the relationship. It is the one part of an edge ignatius cannot derive. -This is the cheapest correctness check in the model, and it catches things no structural rule can. A predicate you cannot phrase truthfully is usually a relationship that does not belong, or an entity whose purpose you have not settled. When the sentence does ring true, it *is* the business case for the entity carrying it: the reason that table exists, stated in one line, in the language of the people who asked for it. +Write both readings. `fwd` reads parent to child and `rev` reads child to parent. Neither is derived from the other, because the inverse of "is realized as" is "is a". A plain string sets both to the same phrase. -The reference model reads like this in both directions: +`belongs to`, `has many`, and `has one` restate the foreign key, the cardinality marker, and the key shape. Read the verb aloud in both directions. If someone who runs the business would not say it, the predicate or the relationship is wrong. -| Relationship | Forward, parent to child | Reverse, child to parent | +| Relationship | Parent reads | Child reads | |---|---|---| -| `SI_Line → SalesInvoice` | SalesInvoice **contains** SI_Line | SI_Line **is part of** SalesInvoice | -| `SalesOrder → Party` | Party **places** SalesOrder | SalesOrder **is placed by** Party | -| `SalesInvoice → Party` | Party **owes on** SalesInvoice | SalesInvoice **is owed by** Party | -| `Business → Party` | Party **is realized as** Business | Business **is a** Party | -| `SOL_Subscription → Subscription` | Subscription **is sold via** SOL_Subscription | SOL_Subscription **sells** Subscription | - -Read the whole diagram with predicates like those and it narrates the business. Read it with `has many` everywhere and it narrates the foreign keys, which you could have got from the DDL. - +| `sales_order → party` | party places sales_order | sales_order is placed by party | +| `invoice → party` | party owes on invoice | invoice is owed by party | -#### Both directions are authored, neither is derived - - -`fwd` reads parent to child. `rev` reads child to parent. ignatius does not invent one from the other, because English does not work that way: the inverse of "is realized as" is "is a", not "is realized as by". - -```yaml -relationships: - - target: Identity - on: - party_id: party_id - predicate: { fwd: is realized as, rev: is a } -``` - -A plain string sets both directions to the same phrase, which is fine when one reading works both ways (`is a`) and is the right first move when you have not settled the reverse yet. - -```yaml -predicate: is a -``` - -Supply only one key of the object form and the other defaults to empty. - -The graph draws edges parent to child and labels them with the forward predicate, so the label matches the direction of the line. Hover an entity and every edge touching it re-reads outward from that entity: edges where it is the child flip to the reverse predicate, and edges where it is the parent stay forward. The dictionary, which has no hover, shows the forward predicate and appends the reverse in a muted style when the two differ. - -One thing worth carrying past the diagram: once a relationship has a real verb, the foreign key can take its name. A column named for the sentence documents itself, and the schema, the dictionary, and the application code end up speaking the same language as the people who asked for the feature. - - -### Example rows - - -An entity can carry sample instances in `examples:`, a list of row objects keyed by column name. - -```yaml -examples: - - party_id: 1 - type: BUSINESS - - party_id: 2 - type: PERSON -``` - -They render as a collapsible table in the entity dialog and the dictionary. Two or three realistic rows are enough. Their job is to make the rules concrete: a sample row that violates a constraint you believe in reveals a modeling error that no structural check can find. Every key must be a real column or PK column, and the live server flags unknown keys with an `entity.example_unknown_column` warning. - - -### Linking entities in prose - - -Double brackets in the body become links. - -```markdown -A **Person** is the specialization of a [[Party]] that is a natural human. -A customer settles invoices with a [[PaymentMethod|payment method]] on file. -``` - -`[[Party]]` links to Party and displays "Party". `[[PaymentMethod|payment method]]` links to PaymentMethod and displays "payment method". In the graph the link opens that entity's dialog; in the dictionary it jumps to its section. The target must match an entity id exactly, case-sensitive. A link to an entity that does not exist renders as muted, non-clickable text and is reported as a `body.unknown_link` finding, so a typo never passes silently. - - -## Key inheritance in practice - - -Here is the pattern the tool is shaped around, taken from the reference model that ships with ignatius. - -``` -Party pk: party_id -SalesInvoice pk: party_id, sales_invoice_id -SI_Line pk: party_id, sales_invoice_id, line_seq -``` - -`SalesInvoice` does not get an independent `invoice_id`. It inherits `party_id` from `Party` and adds its own discriminator. `SI_Line` inherits both and adds a line sequence. Every line item carries its full lineage in its own identity, so the owning party is available without a join, and a line item that belongs to the wrong invoice is not expressible. - -Three things fall out of that automatically: - -- The edge from `SalesInvoice` to `Party` is **identifying**, because the foreign-key column sits inside the child's primary key. -- The cardinality is **one-to-many**, because the child has primary-key columns beyond the foreign key. -- `SalesInvoice` classifies as **dependent**, because it has exactly one identifying parent. - -None of that was declared. Changing the key shape changes all three on the next reload. +The graph labels each edge with `fwd`. Hover an entity and its edges read outward from it, so edges where it is the child show `rev`. In the SQL, the reverse predicate names the foreign key: `sales_order_isplacedby_party`. ## What gets derived -### Cardinality - - -Cardinality comes from the primary-key layout, the relationship type, and foreign-key nullability. There is no `cardinality` field. +A changed key changes the diagram on the next reload. An edge is identifying when every foreign-key column is part of the child's primary key. -- An identifying edge where the child's primary key equals the foreign key exactly is one-to-one. -- A child with primary-key columns beyond the foreign key is one-to-many. -- A nullable foreign key makes the parent side optional. +Classification takes the first rule that matches, top to bottom. -An edge is identifying when its foreign-key columns are part of the child's primary key. That too comes from the key shape, never from a per-edge flag. +```mermaid +flowchart TD + A["entity"] --> B{"reference: true?"} + B -->|yes| C["Classifier"] + B -->|no| D{"in a subtypes:<br/>cluster?"} + D -->|yes| E["Subtype"] + D -->|no| F{"identifying<br/>parents"} + F -->|"2 or more"| G["Associative"] + F -->|"1"| H["Dependent"] + F -->|"0"| I["Independent"] +``` +| Entity | Primary key | Derived | Why | +|---|---|---|---| +| `party_type` | `code` | Classifier | `reference: true` | +| `party` | `party_id` | Independent | its only parent, `party_type`, is outside the key | +| `person` | `party_id` | Subtype | listed in the `subtypes:` of `party` | +| `order_line` | `party_id`, `order_no`, `line_no` | Dependent | `sales_order` is in the key, `product` is not | +| `payment_allocation` | `party_id`, `payment_no`, `invoice_no`, `line_no` | Associative | `payment` and `invoice_line` are both in the key | -### Classification +The dialog for `order_line` shows the result: the DEPENDENT badge and the PK, FK marks on `party_id` and `order_no`, none of which the file declares. +<ModelViewer class="tall" model="order-to-cash" hash="view=graph&entity=order_line" :tabs="false" label="order_line in the model" title="Order-to-cash model with order_line selected" /> -Classification comes from how an entity connects to others, applied in this order. +Cardinality comes from the same keys: -1. **Classifier**, flagged as a reference or classifier entity. -2. **Subtype**, a member of another entity's subtype cluster. -3. **Associative**, two or more identifying parents. -4. **Dependent**, exactly one identifying parent. -5. **Independent**, none of the above. +| Edge | Parent : child | Example | +|---|---|---| +| identifying, FK is part of the child key | 1 : many | `sales_order → party` | +| identifying, FK is the whole child key | 1 : 1 | none in this model | +| identifying, child is a subtype | 1 : 0..1 | `person → party` | +| not identifying | 1 : many | `order_line → product` | -Classification sets the node shape, so the diagram reflects structure rather than a label you maintain. The only declaration the parser reads is `reference: true`, which marks a lookup or code table and satisfies rule 1. A legacy `classification: Classifier` does the same thing. Any other `classification:` value is ignored, and no finding reports the mismatch, so a label written by hand drifts silently. +On a not-identifying edge, a nullable FK makes the parent side optional (`0..1`), and an `ak` on the FK columns makes the child side `1`. -### Subtype clusters +## How do subtypes work? -A basetype declares its members in `subtypes:`, with a discriminator mapping per member. +The basetype lists its members and the column that tells them apart: ```yaml +# model/data/parties/party.md (excerpt) subtypes: - - exclusive: true - desc: Every Party is exactly one of Business or Person - members: - Business: - type: PartyType.code.BUSINESS - Person: - type: PartyType.code.PERSON + - exclusive: true + desc: Every party is exactly one of person or business + members: + person: + party_type: party_type.code.PERSON + business: + party_type: party_type.code.BUSINESS ``` -Clusters render with diamond joiners between the basetype and its members. An exclusive cluster shows an X in the diamond and needs a discriminator column saying which subtype a basetype row is. An inclusive cluster leaves the diamond empty and needs no discriminator, because several subtypes can coexist for the same row. An exclusive cluster with no discriminator raises `cluster.no_discriminator`. - - -## Lineage: how key inheritance pays off in the viewer - - -A key-inherited entity shares its primary-key ancestry with a whole family. Selecting one should surface the family, and ignatius does that with a lineage spotlight: dotted lines in the dictionary and dotted inferred-upstream rays in the graph. - -The rule is precise. A **key edge** is one whose child-side foreign-key columns are all contained in the child's primary key. That is a subset test, not an equality test, which is what makes it correct for both cases: - -- A subtype member pointing at its basetype, where the foreign key equals the full primary key. -- `SalesInvoice → Party`, where `party_id` is a proper subset of `{party_id, sales_invoice_id}`. This is the identifying one-to-many case, and an equality test would miss it. - -An entity's lineage is the transitive connected component of that entity in the graph of key edges only, traversed in both directions. Secondary foreign keys are excluded. A classifier reference like `Party → PartyType` is a real relationship but not a shared-key ancestry, so it never pulls an unrelated entity into the spotlight. - -Subtype clusters fall out for free, since every member-to-basetype relationship is itself a key edge. - -The practical effect: select `SSN` in the reference model and the spotlight reaches `SalesInvoice`, `SI_Line`, `SalesOrder`, `SO_Line`, and `PaymentAllocation`, because all of them carry `party_id` inside a larger key. That is the family that moves together when the identity spine changes. On an all-surrogate schema the same feature has nothing to show, because nothing shares a key. +Each member is also its own entity file, keyed by the basetype's key, with a relationship back to it. An exclusive cluster draws an X in the diamond and needs the mapping form above. An inclusive cluster (`exclusive: false`) leaves the diamond empty and lists members by name, because several can coexist for one row. ## The two key conventions -A model can use either convention, and can mix them per entity. The modeling skill detects which one a model uses from the shape of its existing entities rather than asking you to pick a mode. - -| Convention | Primary key shape | Foreign key placement | +| Convention | Primary key | Foreign key | |---|---|---| -| `key-inherited` | Composite: the parent's primary-key columns plus a local discriminator | Foreign-key columns live inside the child's primary key | -| `orm-oriented` | A single surrogate `id`, typically integer or uuid | Foreign-key columns sit outside the primary key as plain columns | +| `key-inherited` | the parent's key columns plus a local discriminator | inside the child's key | +| `orm-oriented` | a single surrogate `id` | a plain column outside the key | -ignatius ships three variants of the same data model to compare directly: `key-inherited`, `orm-hybrid`, and `orm-pure`. Serving each in turn is the fastest way to see what the convention costs and buys, because the entities and the business meaning are held constant while only the keys change. +A model can mix them per entity. `party` is the root of the order-to-cash model, nothing above it supplies a key, and a surrogate is right there. Record the reason in the entity body. The [modeling skill](/modeling/modeling-skill) detects the prevailing convention from existing entities. -Mixing is legitimate. A transactional spine can inherit keys while a lookup table keeps a surrogate code. Do it deliberately rather than by accident, and record the reason in the entity body. +Key inheritance pays off in the graph. Hold Shift and hover `party`: dotted lines reach `order_line`, `invoice_line`, and `payment_allocation`, which are not direct neighbors but carry `party_id` inside their keys. On a schema where every table has its own `id`, there is nothing to show. +The ignatius repository ships one data model three ways under `models/`: `key-inherited`, `orm-hybrid`, and `orm-pure`. Only the keys differ. -## Groups +## What bites -A group is a markdown file in `groups/` with a label and a color in frontmatter and a description in the body. -```markdown ---- -label: Identity & Accounts -color: "#2ea043" ---- - -Party identity, classifications, and ID documents. -``` - -Groups set a border color and a pastel fill for their entities. They do not affect layout. An entity whose `group` names a file that does not exist renders without a color band and raises an `entity.unknown_group` warning. +| Mistake or symptom | Cause | Fix | +|---|---|---| +| `classification: Dependent` or `identifying: true` does nothing | ignatius reads only `reference: true`, and no finding reports the rest | delete the line and change the key shape | +| `cluster.no_discriminator` warning | an exclusive cluster lists `members` as plain names | use the mapping form: `person: { party_type: party_type.code.PERSON }` | +| `[[Party]]` renders as muted text, `body.unknown_link` | link targets match the entity id exactly, and the id is `party` | write `[[party]]` | +| `entity.example_unknown_column` never shows in `validate` | the rule is live-only, reported by `ignatius serve` | check the rows in the viewer | +| An entity file named `index.md` fails with `config.index_file_entity` | `index.md` is the reserved router name | rename the file | -## Related +## Next -- [Data flows](/modeling/data-flows) puts these entities to work as data stores in process diagrams. -- [The case for proper relational design](/guide/relational-design) makes the argument for inherited keys in the database itself. -- The [`idef1x` skill](https://www.skills.sh/damusix/skills/idef1x) coaches an agent through the methodology itself, separately from ignatius. -- The [derivation guide](https://github.com/noormdev/ignatius/blob/main/docs/guides/derivation.md) and the [folder format reference](https://github.com/noormdev/ignatius/blob/main/docs/guides/folder-format.md) carry the exhaustive rules. +- [Data flows](/modeling/data-flows): the processes, externals, and stores that use these entities +- [From model to SQL](/modeling/model-to-sql): how each field becomes DDL +- [The case for proper relational design](/guide/relational-design): the argument for inherited keys in the database +- The [`idef1x` skill](https://www.skills.sh/damusix/skills/idef1x) coaches an agent through the methodology, separately from ignatius +- The [folder format reference](https://github.com/noormdev/ignatius/blob/main/docs/guides/folder-format.md) lists every rule diff --git a/docs/modeling/index.md b/docs/modeling/index.md index 18f07ec5..f118e35f 100644 --- a/docs/modeling/index.md +++ b/docs/modeling/index.md @@ -1,127 +1,184 @@ --- -title: Information modeling -description: Plan the data model before you build it. ignatius turns a folder of markdown into an IDEF1X entity diagram, a searchable data dictionary, and SSADM data flow diagrams. +layout: home +title: ignatius, data modeling +description: ignatius turns a folder of markdown into data flow diagrams, a relationship graph, and a data dictionary, so a team agrees on the data model before anyone writes the SQL. --- -# Information modeling +<section class="nh-hero"> +<p class="nh-eyebrow">ignatius · data modeling · works with or without noorm</p> -Most schemas get designed while they are being built. A table appears because a feature needed it, a column gets added because a bug needed it, and the shape of the whole thing only exists in the migration history and in the head of whoever wrote it. By the time anyone can see the model, it is already in production and expensive to change. +<h1 class="nh-title">Agree on the data model<br><span>before anyone writes the SQL</span></h1> -[ignatius](https://github.com/noormdev/ignatius) moves that work earlier. It is a planning tool for information modeling: you describe entities, keys, and relationships in markdown files, and it renders them as an interactive diagram you can hand to another person. The binary never connects to a database and never generates DDL. Its output is a picture of what you intend to build, precise enough to argue about before anyone writes SQL. - -<!-- Wrapped in a div because `video` is not a CommonMark block-level tag: - left bare, markdown-it splits the multi-line element across paragraphs and - Vue's SFC parser then rejects the orphaned closing tag. --> -<div class="demo-video"> -<video src="/video/ignatius.mp4" poster="/video/ignatius-poster.jpg" width="1316" height="858" controls muted playsinline preload="none"></video> +<div class="nh-row"> +<p class="nh-lead">Analysis costs less than rebuilding a system. ignatius turns a folder of markdown into a picture of the plan: what captures data, the shape it takes, why each table exists, the rules it follows, and who outside the system sends it and receives it.</p> +<div class="nh-actions"> +<a class="nh-btn brand" href="/download">Download</a> +<a class="nh-btn alt" href="#try-it">Try it in two minutes</a> +<p class="nh-caption">one binary · macOS, Linux, Windows</p> +</div> </div> -Above: a model of an LLM memory database, walked through all three views. The entity dialog, the searchable dictionary with its key-inheritance lineage, and a decomposed process flow where an external LLM agent creates and files artifacts. - - -## What it produces - - -One folder of markdown files drives three views of the same model. +</section> -| View | What it shows | -|---|---| -| Data Graph | An IDEF1X entity-relationship diagram in crow's-foot notation. Pan, zoom, click an entity to open its full definition. | -| Data Dictionary | Every entity, process, external, and store in one searchable reference page. | -| Data Flows | SSADM data flow diagrams showing how processes move data between people, the database, and everything else. | +<section class="nh-section"> -The diagrams are generated, never drawn. You write the structure as text, and ignatius derives the cardinality, the entity classification, and the subtype clusters from the key shape you described. Change a foreign key and the notation updates on the next reload. A model that lives in markdown diffs like code, reviews like code, and cannot drift from a picture somebody drew once in a diagramming app. +<p class="nh-eyebrow">01 · Movement</p> +<h2 class="nh-h2">Do you know how data moves through your system, and who produces it?</h2> -## Why a web instead of a list +<p class="nh-chapter">A data flow diagram answers both. It shows who produces data, including people outside the technical system, which process moves or transforms it, and which store keeps it: a table, a cache, a log file, a paper form. Most teams skip this step and draw logic flowcharts, which show decisions, not data.</p> +<!--@include: ../parts/ignatius-dfd.md--> -A table list tells you what exists. It does not tell you how anything connects, or why any of it is there. +<p class="nh-note">Read each process as an operation your database must support. <strong>Place order</strong> takes the customer's order request and product prices and writes <code>sales_order</code> and <code>order_line</code>. That is <code>place_order()</code>, specified before anyone writes the SQL or the code that calls it.</p> -The graph gives you the web: which entities hang off which, where identity comes from, which clusters are subtypes of a common basetype. Each entity carries a prose body explaining what it is and what business function it serves, so the answer to "why does this table exist" lives next to the table. Each entity also carries example rows. Two or three realistic instances make the rules concrete, and a sample row that violates a constraint you believed in reveals a modeling error no structural check will catch. +</section> -You make better decisions with a full view of the thing you are about to build. That is the whole argument for doing this before the schema exists rather than after. +<section class="nh-section"> +<p class="nh-eyebrow">02 · Relationships</p> -## Written for people and for models +<h2 class="nh-h2">Have you ever visualized your entity relationships clearly?</h2> +<p class="nh-chapter">An ER diagram lists every column and buries the relationships under them. ignatius draws the relationships alone: entities by name, joined by the business verbs that relate them. Click an entity to open its columns, keys, example rows, and the reason it exists, then walk the graph to tell the story in a meeting.</p> -The source is markdown with YAML frontmatter, which suits both readers. +<!--@include: ../parts/ignatius-viewer.md--> -A stakeholder gets the rendered app: a diagram they can explore, a dictionary they can search, and prose that explains the business meaning rather than the storage layout. `ignatius export` writes all three views into one self-contained HTML file with no external dependencies, so a reviewer opens it from `file://` and needs nothing installed. +</section> -A coding agent gets plain text it can read and write directly. ignatius comes with the [`ignatius-modeling` skill](/modeling/modeling-skill) for Claude Code, which authors entity and flow files through guided Q&A and verifies each one with `ignatius validate` before moving on. +<section class="nh-section"> +<p class="nh-eyebrow">03 · Agents</p> -## The workflow +<h2 class="nh-h2">In the age of AI agents, review the plan first</h2> +<p class="nh-chapter">Agents write schemas faster than anyone can read them, so the review has to happen before the code exists. The model is markdown, so it is the thing you review before you sign off on what the agent builds. The agent reads the same files, and the ignatius-modeling skill writes them one question at a time and validates each one.</p> -Model first, get agreement, then build. +<!--@include: ../parts/ignatius-cmds.md--> -1. **Model.** Describe the entities and the processes in markdown, either by hand or through the modeling skill. -2. **Review.** Serve the model or export it to HTML and walk stakeholders through the graph and the flows. Arguments about the shape of the data are cheap here and expensive later. -3. **Build.** Once the model is agreed, hand it to your agent as the specification. The files that describe the model are the same files the LLM reads to write the schema. +</section> -noorm is the other half of that last step. ignatius describes what you intend to build. noorm builds it, versions it, and keeps every environment in sync. The two share a bias: your schema belongs in files you own, not in an object model a tool derived for you. +<section class="nh-section"> +<p class="nh-eyebrow">04 · Documentation</p> -## Why a separate tool +<h2 class="nh-h2">The plan becomes the visualization</h2> +<p class="nh-chapter">Everything above comes from one folder of markdown. The same files are your database's documentation and its data dictionary, the place the team plans and shares plans, and the place an agent looks to see what it should build.</p> -noorm already explores schemas and runs SQL, so it is fair to ask why modeling is a second binary instead of a `noorm model` subcommand. Three reasons, and they all come back to ignatius being useful at a moment when noorm is not. +<!--@include: ../parts/ignatius-problems.md--> -**It never touches a database.** ignatius carries no driver and emits no DDL. Its value is being usable before the database exists, which is exactly the moment a tool that requires a connection has nothing to offer. Putting the planning step inside noorm would gate it behind the thing you have not built yet. +</section> -**It does not need noorm.** Nothing in the format is noorm-specific: markdown in, IDEF1X and SSADM diagrams out, with no dialect anywhere in the pipeline. A team on Prisma, ActiveRecord, or hand-written migrations gets the same thing from it. Making it a subcommand would shrink its audience to noorm users and buy nothing. +<section class="nh-section"> -**The audiences differ.** `ignatius export` produces a file for the person approving a design, not the person deploying a schema. That reader is not going to install a schema and change manager to look at a diagram. +<p class="nh-eyebrow">05 · Lineage</p> -There is a practical consequence worth knowing as you read: the two ship on their own cadences from their own repositories. That is why these pages carry the narrative and link out to the ignatius repo for the exhaustive rule catalogs rather than copying them here, where they would drift out of date. +<h2 class="nh-h2">Built on the shoulders of giants</h2> +<p class="nh-chapter">ignatius is not a new invention. It is a looking glass into excellent standards: IDEF1X for entities, grown out of E. F. Codd's relational model, and SSADM-style data flow diagrams for movement.</p> -## Two notations, two questions +| Year | Step | +|---|---| +| 1970 | E. F. Codd publishes the relational model of data. | +| 1977–78 | Chris Gane and Trish Sarson, then Tom DeMarco, publish structured analysis built on data flow diagrams. | +| 1981 | The US Air Force's ICAM program publishes IDEF1, built on Codd's relational theory and Peter Chen's entity-relationship model. | +| 1982 | Robert Brown develops the logical database design technique for Bank of America. | +| 1983 on | The Air Force extends IDEF1 into IDEF1X, using Brown's technique. | +| 1993 | NIST publishes IDEF1X as federal standard FIPS 184. | +| 2012 | IDEF1X becomes part of the international standard ISO/IEC/IEEE 31320-2. | +</section> -ignatius answers a different question in each of its diagram views. +<section class="nh-section" id="try-it"> -**IDEF1X** answers "what is the data, and how is it identified". It is the premise the entity side is built on, which is why key inheritance gets first-class treatment: identifying relationships, compound keys that carry their lineage, and subtype clusters with exclusive or inclusive discriminators. See [Entities and key inheritance](/modeling/entities). +<p class="nh-eyebrow">Try it</p> -**SSADM data flow diagrams** answer "who touches the data, and where does it live". Processes, external actors, and data stores. Stores are not only tables. A cache, a log file, a queue, a paper form, and a document all appear as first-class stores, so the diagram shows where data rests rather than only the part of it that reached a database. See [Data flows](/modeling/data-flows). +<h2 class="nh-h2">A model in two minutes</h2> +<div class="nh-steps"> +<div> +<div> +<p class="nh-step">01 · Install</p> +<p class="nh-step-note">Windows and other options: <a href="/download">Download</a>.</p> +</div> +<div> -## Where the name comes from +```bash +curl -fsSL https://noorm.dev/ignatius.sh | sh +``` +</div> +</div> +<div> +<div> +<p class="nh-step">02 · Write one entity</p> +<p class="nh-step-note">A folder with an <code>ignatius.yml</code> is a model. Each file under <code>data/</code> is one entity: keys and columns in the frontmatter, what it is for in the body.</p> +</div> +<div> -ignatius is named for Derek Ignatius Asirvadem, who mentored me for a couple of years and introduced me to IDEF1X. +<div class="nh-file"> +<div class="nh-panel-head"><span>model/ignatius.yml</span><span>you write this</span></div> -Studying it and then practising it convinced me there is no higher standard for modeling data. The industry mostly skips modeling and goes straight to building. Modeling first solves something like 80% of the problems up front, and it produces months of valid programming work before anyone opens an editor. +```yaml +name: Shop +``` -The other 20% is real and worth naming: once you start building, you **do** find problems in the model. That is the part that drifts. What matters is when you find it. While nothing is physically committed to the system yet, changing the model is cheap. Once it is code, it is not. Modeling holds that cheap window open longer, and an agent working from a written model holds it open longer still. +</div> +<div class="nh-file"> +<div class="nh-panel-head"><span>model/data/party.md</span><span>you write this</span></div> -Before ignatius I drew models in OmniGraffle, on Derek's recommendation. Two things broke that. My own obsession with perfect lines and perfect names turned drawing into the bottleneck, and there was no way to tell an LLM "this is the relationship between these two entities" from a picture. So the model moved into markdown, where both problems go away. The layout is generated rather than drawn, and the source is text an agent can read and write. +```markdown +--- +entity: party +pk: [party_id] +columns: + party_id: { type: integer } + name: { type: text } +--- -None of these tools would exist without his guidance. ignatius is a tribute to him. +# party +Anyone who places orders. +``` -## Next steps +</div> +</div> +</div> +<div> +<div> +<p class="nh-step">03 · Serve it</p> +<p class="nh-step-note">The browser opens on the diagram. Save <code>party.md</code> and the open tab redraws. Add the next entity with a <code>relationships:</code> block pointing at <code>party</code>, and the edge appears.</p> +</div> +<div> -<div class="next-steps"> +```bash +ignatius serve model -o +``` -[**Installation**](/modeling/installation) -Install the binary and the modeling skill. +</div> +</div> +</div> -[**Entities and key inheritance**](/modeling/entities) -The entity format, what gets derived, and the two key conventions. +</section> -[**Data flows**](/modeling/data-flows) -Processes, externals, stores, and sub-diagrams. +<section class="nh-section"> -[**Reverse-engineering**](/modeling/reverse-engineering) -Extract a model from a database you already run. +<p class="nh-eyebrow">Go deeper</p> -[**Best practices**](/modeling/best-practices) -How to get a model worth showing to a stakeholder. +<h2 class="nh-h2">Where to go next</h2> +<div class="nh-cards"> +<a href="/modeling/entities"><strong>Entity files</strong><span>The entity format, and what ignatius derives from your keys.</span></a> +<a href="/modeling/data-flows"><strong>Data flows</strong><span>Processes, externals, and stores, including the ones that are not tables.</span></a> +<a href="/modeling/model-to-sql"><strong>From model to SQL</strong><span>How each field of an entity file becomes DDL.</span></a> +<a href="/modeling/reverse-engineering"><strong>Reverse-engineering</strong><span>Extract a model from a database you already run.</span></a> +<a href="/modeling/modeling-skill"><strong>The modeling skill</strong><span>Author entities and flows from Claude Code, one question at a time.</span></a> +<a href="/modeling/best-practices"><strong>Best practices</strong><span>What makes a model worth showing to a stakeholder.</span></a> +<a href="/modeling/installation"><strong>Commands</strong><span>Install, serve, validate, export, index, approve, and update.</span></a> +<a href="/modeling/why"><strong>Why ignatius</strong><span>The case for modeling first, and where the name comes from.</span></a> </div> + +</section> diff --git a/docs/modeling/installation.md b/docs/modeling/installation.md index aaa88824..c6fd28af 100644 --- a/docs/modeling/installation.md +++ b/docs/modeling/installation.md @@ -1,163 +1,120 @@ --- -title: Installing ignatius -description: Install the ignatius binary and the ignatius-modeling skill, then serve, validate, and export your first model. +title: ignatius commands +description: Install ignatius, then serve, validate, export, index, approve, and update a model. Flags, exit codes, and how a command finds your model folder. --- -# Installing ignatius +# ignatius commands - -ignatius is a single self-contained binary with no runtime dependency. It runs on machines without Bun or Node installed. - - -## Install script - - -The script detects your operating system and architecture, downloads the matching binary from the [latest release](https://github.com/noormdev/ignatius/releases/latest), verifies its checksum, and puts it on your `$PATH`. +Install the ignatius binary, then serve, check, or export a model from the command line. ```bash -curl -fsSL https://raw.githubusercontent.com/noormdev/ignatius/main/install.sh | sh +curl -fsSL https://noorm.dev/ignatius.sh | sh +ignatius serve path/to/model -o ``` -It installs to `/usr/local/bin` when that directory is writable and `$HOME/.local/bin` otherwise. Two environment variables override the defaults: - -- `IGNATIUS_INSTALL_DIR=/some/dir` installs somewhere else. -- `IGNATIUS_VERSION=v0.2.0` pins a release tag instead of taking the latest. - -Verify with `ignatius --help`. - - -## Download a release - - -To install by hand, or on Windows, take the binary for your platform from the [latest release](https://github.com/noormdev/ignatius/releases/latest). - -```bash -# macOS arm64 -curl -L -o ignatius https://github.com/noormdev/ignatius/releases/latest/download/ignatius-darwin-arm64 -chmod +x ignatius -sudo mv ignatius /usr/local/bin/ - -# Linux x64 -curl -L -o ignatius https://github.com/noormdev/ignatius/releases/latest/download/ignatius-linux-x64 -chmod +x ignatius -sudo mv ignatius /usr/local/bin/ ``` - -Windows users take `ignatius-windows-x64.exe` from the same page. Each release includes `checksums.txt` for verifying a manual download with `shasum -a 256 -c`. - - -## From source - - -ignatius builds with [Bun](https://bun.com). - -```bash -git clone https://github.com/noormdev/ignatius.git -cd ignatius -bun install -bun run build:cli +ignatius serving at http://localhost:3000 ``` -That produces `./dist/ignatius`. +The browser opens on the diagram. Save a `.md`, `.yaml`, or `.yml` file in the model folder and the open tab redraws. Windows, a pinned version, and building from source are on [Download](/download). -## Staying current +## What does each command do? -```bash -ignatius version # print the installed version -ignatius update # check for a newer release, then prompt -ignatius update --check # report only, never install -ignatius update --yes # install without prompting, for scripts -``` +| Command | Flags | Effect | +|---|---|---| +| `ignatius serve [path]` | `-p`, `--port <n>` (3000); `--model <key>`; `-o`, `--open` | Serves the model with live reload. `server` is an alias | +| `ignatius validate [path]` | `--model <key>`; `--index` | Prints findings to stderr and a summary to stdout. Writes nothing | +| `ignatius export [path]` | `-o`, `--out <file>` (required); `--theme <mode>` (`dark`); `--model <key>` | Writes the Graph, Dictionary, and Flows views as one offline HTML file. `--theme` takes `light` or `dark` | +| `ignatius index [path]` | `--model <key>`; `--agents` | Writes a router file (`index_file:` in `ignatius.yml`, default `index.md`) into every organizing folder. `--agents` adds `AGENTS.md` and `SKILL.md`, plus `CLAUDE.md` when the harness resolves to Claude | +| `ignatius approve [path]` | `--model <key>` | Accepts the current files as the change-review baseline and deletes the notes in `.ignatius/changes/` | +| `ignatius version` | none | Prints the version. `ignatius --version` prints the same | +| `ignatius update` | `--check`; `-y`, `--yes` | Replaces the running binary with the latest release | -`update` downloads the binary for your platform, verifies its checksum against the release `checksums.txt`, and replaces the running executable in place. That needs write access to the installed binary, so a binary in `/usr/local/bin` wants `sudo ignatius update --yes`. Outside a terminal it reports the available version and exits without installing unless you pass `--yes`. Windows binaries cannot replace themselves while running, so on Windows the command points you at the release download instead. +`serve` marks what changed since the last `approve`; see [Change review](https://github.com/noormdev/ignatius/blob/main/docs/guides/change-review.md). Press `?` in the app for the keys that work in the current view, or read the [keyboard shortcuts](https://github.com/noormdev/ignatius/blob/main/docs/guides/commands.md#keyboard-shortcuts). -## The modeling skill +## How does a command find the model? -To author models from Claude Code, install the `ignatius-modeling` skill with the [`skills`](https://www.npmjs.com/package/skills) CLI. +Every model command takes an optional `path`. A model root is a folder with an `ignatius.yml`: -```bash -npx skills add https://github.com/noormdev/ignatius --skill ignatius-modeling +```mermaid +flowchart TD + A{"ignatius.yml<br/>in path?"} -->|yes| R["run on that folder"] + A -->|no| S["search folders<br/>below path"] + S --> C{"roots found"} + C -->|one| R + C -->|none| W["walk up for a root,<br/>else exit 1"] + C -->|several| K{"--model given?"} + W --> R + K -->|matches| R + K -->|no match| X["exit 1"] + K -->|omitted| P["terminal: prompt<br/>otherwise exit 2"] ``` -That adds the skill to the current project's `.claude/skills/`. Add `-g` to install it globally for every project on the machine. Reload skills in Claude Code and `/ignatius-modeling` becomes available. See [The modeling skill](/modeling/modeling-skill) for what each mode does. +The search skips `node_modules`, `.git`, `dist`, `tmp`, `trash`, `.worktrees`, `.claude`, and any folder starting with `_`. The `--model` key is the root's path relative to the search path, such as `sales`. -## Serve a model +## What do the exit codes mean? -Point ignatius at a folder that contains an `ignatius.yml` file. +| Code | When | +|---|---| +| `0` | No errors. Warnings alone do not fail `validate`, `export`, or `index` | +| `1` | An error finding, no model found, `--model` matches nothing, or `export` without `-o` | +| `2` | Several models, no `--model`, and no terminal to prompt in | +| `130` | You cancelled a prompt | + +An error finding is an omitted edge, an unknown target, or an unparseable file. It prints as `<severity> <rule> <location> <message>`: ```bash -ignatius serve path/to/your/models --port 3000 +ignatius validate model ``` -Editing any `.md` or `.yaml` file under that folder pushes an update to the open browser tab over server-sent events, so the diagram reloads without a refresh. - -The path argument is optional. With no path, ignatius searches up and down from the current directory for a model root. When it finds more than one it prompts you with a list in a terminal, or takes `--model <key>` to skip the prompt. In a non-interactive shell an ambiguous run exits with an error and prints the available keys rather than hanging. The search skips `node_modules`, `.git`, `dist`, `tmp`, `trash`, `.worktrees`, `.claude`, and any path segment starting with `_`. - -The default port is 3000. When that port is taken, ignatius counts up to the next free one and asks which to use, with the free port as the default. Run non-interactively it advances automatically and prints the port it settled on. Pass `-o` to open the app in your browser once the server binds, on whichever port it ended up with. - -The app has three views, switched without a page reload and reflected in the URL hash: Graph (`#view=graph`), Dictionary (`#view=dict`), and Flows (`#view=flow`). Back, forward, and deep links all work. - - -## Validate a model - - -```bash -ignatius validate [path] [--model <key>] +``` +error edge.unknown_target sales_order→partyy Edge target "partyy" not present in model. +✗ /home/you/shop/model: 1 error(s), 0 warning(s) across 11 entities. ``` -`validate` checks the model and reports findings without writing anything. It prints each finding to stderr and a one-line summary to stdout, then exits `1` when the model has errors and `0` otherwise. Warnings alone do not fail the command. When the model has a `flows/` directory the flow rules run too. +A clean run prints `✓ <path>: valid — 11 entities, no findings.` The [rule catalog](https://github.com/noormdev/ignatius/blob/main/docs/guides/validation.md) lists every rule. -That exit-code contract makes it usable as a CI gate. Findings come in two classes: a warning still renders the entity with a badge, while an error omits the broken reference (one edge, one cluster, one unparseable file) and names the omission in a banner. A typo in one of three foreign keys strips that edge and keeps the entity. The [full rule catalog](https://github.com/noormdev/ignatius/blob/main/docs/guides/validation.md) lists every rule and its class. +## How does serve pick a port? -## Export a model +When 3000 is taken, a terminal asks which port to use, with the next free one as the default. A pipe or CI counts up until a bind succeeds and prints `Port 3000 is in use — trying 3001.` to stderr. `-o` opens the port it settled on. -```bash -ignatius export [path] -o model.html [--theme light|dark] [--model <key>] -``` -`export` writes all three views into one HTML file with no external dependencies. View switching, dictionary search, the entity dialog, the theme toggle, and node-position persistence all work offline from `file://`. `-o` is required. +## How do I stay current? -This is the artifact you send to someone who will not install anything. Commit it next to the model, attach it to a ticket, or mail it to the stakeholder whose sign-off you need. +| Situation | `ignatius update` does | +|---|---| +| Already latest | Prints `ignatius 0.23.0 is up to date.` | +| Newer release | Prints `A new version is available: 0.23.0 → <latest>`. With `--check`, or with no terminal and no `--yes`, it stops there. In a terminal it asks `Update to <latest> now?`, downloads, checks the checksum, and swaps the binary | +| Windows | Prints the release URL. A running `.exe` cannot replace itself | +| Binary in a folder you cannot write | Exits `1` with `cannot write to …`. Run `sudo ignatius update --yes` | -## Keyboard shortcuts +## What bites -Single-key shortcuts work while no text field is focused and no modifier is held. -| Key | Action | -|---|---| -| `g` | Switch to the Data Graph | -| `d` | Switch to the Dictionary | -| `f` | Switch to the Data Flows | -| `l` | Toggle graph layout, organic or hierarchical (Graph view) | -| `b` | Toggle dictionary lens, read or browse (Dictionary view) | -| `/` | Focus the search bar | -| `?` | Open the help overlay for the current view | -| `↑` `↓` `←` `→` | Pan the canvas (Graph and Flows); hold `Shift` for a longer step | - -Zoom and search also answer to modifier chords. Those work even while a text field is focused, because the chords are not typed characters, and zoom acts on the diagram canvas rather than the browser page. - -| Input | Action | -|---|---| -| `Cmd`/`Ctrl` + `=` | Zoom in | -| `Cmd`/`Ctrl` + `-` | Zoom out | -| `Cmd`/`Ctrl` + `0` | Fit the diagram to the screen | -| `Cmd`/`Ctrl` + `k` | Focus the search bar | -| Trackpad pinch | Zoom toward the pointer | +| Symptom | Cause | Fix | +|---|---|---| +| `-o <output.html> is required` | `export` has no default output | `ignatius export model -o model.html` | +| CI exits `2`: `multiple models found: a, b` | Two roots under the path and no terminal to choose in | Add `--model a` | +| `dict was removed`, or `graph`, `flow` | Those subcommands no longer exist | `ignatius export -o model.html` | +| `export` exits `1` and the HTML file exists | `export` writes the file, then fails on error findings | Fix the findings before you share the file | +| `validate` passes with a stale router | Plain `validate` never hashes routers | `validate --index`, then `ignatius index` | +| `update` ran with no checksum check | A mismatch aborts, but an unreachable `checksums.txt` skips the check | Rerun `ignatius update` once the release page is reachable | -## Next steps +## Next -- [Entities and key inheritance](/modeling/entities) covers the folder format and the entity file. -- [Data flows](/modeling/data-flows) covers processes, externals, and stores. -- [The modeling skill](/modeling/modeling-skill) covers authoring from Claude Code. +- [Download](/download): Windows, a pinned version, and the agent skill +- [Entities and key inheritance](/modeling/entities): the folder format a model root holds +- [The modeling skill](/modeling/modeling-skill): author a model from Claude Code +- [Branding your model](/modeling/branding): put your name on the export diff --git a/docs/modeling/model-to-sql.md b/docs/modeling/model-to-sql.md new file mode 100644 index 00000000..03af144a --- /dev/null +++ b/docs/modeling/model-to-sql.md @@ -0,0 +1,119 @@ +--- +title: From model to SQL +description: How each field of an ignatius entity file maps to a CREATE TABLE statement, from keys and foreign keys to constraint names, types, alternate keys, and subtypes. +--- + +# From model to SQL + +Each entity file becomes one `CREATE TABLE`. Keys, foreign keys, and constraint names come from the frontmatter. The physical type is the one decision you make per dialect. + +<div class="pair"> + +<div class="nh-file"> +<div class="nh-panel-head"><span>model/data/sales/sales_order.md</span><span>you write this</span></div> + +```yaml +--- +entity: sales_order +group: sales +pk: [party_id, order_no] +columns: + party_id: { type: integer } + order_no: { type: integer } + placed_at: { type: datetime, default: now } + status: { type: text, default: open } +examples: + - { party_id: 2, order_no: 1, status: invoiced } + - { party_id: 2, order_no: 2, status: open } + - { party_id: 1, order_no: 1, status: paid } +relationships: + - target: party + on: { party_id: party_id } + predicate: + fwd: places + rev: is placed by +--- +``` + +</div> + +<p class="pair-joint">becomes</p> + +<div class="nh-file"> +<div class="nh-panel-head"><span>db/sql/01_tables/02_sales_order.sql</span><span>what noorm runs</span></div> + +```sql +CREATE TABLE sales_order ( + party_id integer NOT NULL, + order_no integer NOT NULL, + placed_at timestamptz NOT NULL DEFAULT now(), + status text NOT NULL DEFAULT 'open', + CONSTRAINT sales_order_pk + PRIMARY KEY (party_id, order_no), + CONSTRAINT sales_order_isplacedby_party + FOREIGN KEY (party_id) REFERENCES party (party_id) +); +``` + +</div> + +</div> + + +## How each field maps + + +| Entity file | SQL | +|---|---| +| `pk:` | `PRIMARY KEY (…)` | +| `relationships[].on` | `FOREIGN KEY (…) REFERENCES …` | +| `predicate.rev` and `target` | the constraint name, `<table>_<reverse predicate>_<target>`: `sales_order_isplacedby_party` | +| `type: datetime` | the physical type for your dialect: `timestamptz` on Postgres, `datetime2` on SQL Server, `datetime` on MySQL | +| `ak:` with a `rule` | a `UNIQUE` constraint named for the rule: rule `is identified by sku` gives `product_isidentifiedby_sku` | +| `subtypes:` | one table per member, whose primary key is the foreign key to the basetype | + +The predicate is the part a tool cannot derive, so it is also the part worth reading in review. A constraint named `sales_order_isplacedby_party` states the business rule the foreign key enforces. A name like `sales_order_party_id_fkey` states only which column it sits on. + + +## What inherited keys buy + + +`payment_allocation` has two identifying parents, `payment` and `invoice_line`. Both carry `party_id`, and the table stores it once: + +```sql +CREATE TABLE payment_allocation ( + party_id integer NOT NULL, + payment_no integer NOT NULL, + invoice_no integer NOT NULL, + line_no integer NOT NULL, + amount numeric(12,2) NOT NULL, + CONSTRAINT payment_allocation_pk + PRIMARY KEY (party_id, payment_no, invoice_no, line_no), + CONSTRAINT payment_allocation_isdrawnfrom_payment + FOREIGN KEY (party_id, payment_no) + REFERENCES payment (party_id, payment_no), + CONSTRAINT payment_allocation_settles_invoice_line + FOREIGN KEY (party_id, invoice_no, line_no) + REFERENCES invoice_line (party_id, invoice_no, line_no) +); +``` + +A payment from party 1 cannot settle an invoice line of party 2: the single `party_id` column would have to hold both values. Two foreign keys enforce it, with no trigger and no application check. The model's entry for `payment_allocation` shows both parents in its key, and the classification ignatius derived from that shape: + +<ModelViewer class="tall" model="order-to-cash" hash="view=graph&entity=payment_allocation" :tabs="false" label="payment_allocation in the model" title="Order-to-cash model with payment_allocation selected" /> + + +## Check the model before you write the SQL + + +```bash +ignatius validate model/ +``` + +``` +✓ /home/you/shop/model: valid — 11 entities, no findings. +``` + +`validate` exits `1` when the model has errors and `0` otherwise, so it gates a pull request as it stands. The [full rule catalog](https://github.com/noormdev/ignatius/blob/main/docs/guides/validation.md) lists every check. + +Next: [Reverse-engineering an existing database](/modeling/reverse-engineering) diff --git a/docs/modeling/modeling-skill.md b/docs/modeling/modeling-skill.md index 6059da12..5310fa31 100644 --- a/docs/modeling/modeling-skill.md +++ b/docs/modeling/modeling-skill.md @@ -1,86 +1,97 @@ --- title: The modeling skill -description: /ignatius-modeling authors ignatius entities and data flow diagrams through guided Q&A in Claude Code, and verifies each file with ignatius validate. +description: /ignatius-modeling writes ignatius entities and data flow diagrams by asking one question at a time, runs ignatius validate on each file, and ends every session with a change note you review and approve. --- # The modeling skill +`/ignatius-modeling` is a Claude Code skill that writes ignatius entity and flow files by asking you one question at a time. + +One session asks, writes, validates, and then waits for you to approve: + +```mermaid +sequenceDiagram + participant You + participant Skill + participant Files as model files + participant Validate as ignatius validate + You->>Skill: /ignatius-modeling entity + loop each step + Skill->>You: one question + You->>Skill: answer + end + Skill->>Files: write the entity file + Skill->>Validate: check the model folder + Validate-->>Skill: findings, up to 5 attempts + Skill->>Files: write .ignatius/changes/slug.md + You->>Files: ignatius serve, read the changes + You->>Files: ignatius approve +``` -`/ignatius-modeling` is a Claude Code skill that authors a model through question and answer, then checks its own work. It writes one entity, a data flow diagram, a fresh model skeleton, or a full discovery session that works the model out from how your business runs. - -The skill exists because the hard part of modeling is not the YAML. It is knowing which questions to ask, in which order, and noticing when an answer contradicts an earlier one. The skill carries that interview structure, and you supply the domain knowledge. - -**Prerequisites:** Claude Code with skill support, and the `ignatius` binary on your `$PATH` or built locally with `bun run build:cli`. - - -## Install - +The skill needs Claude Code and the `ignatius` binary on your `$PATH` ([Download](/download)). Install the skill into the current project, or add `-g` for every project: ```bash npx skills add https://github.com/noormdev/ignatius --skill ignatius-modeling ``` -That adds `ignatius-modeling` to the current project's `.claude/skills/`. Add `-g` to install it globally for every project on the machine. Reload skills in Claude Code and `/ignatius-modeling` becomes available. +## Which mode do I run? -## Modes - -| Invocation | What it does | +| Invocation | Use it to | |---|---| -| `/ignatius-modeling entity` | Interactive Q&A to author one entity file | -| `/ignatius-modeling model` | Bootstrap a new model skeleton: `ignatius.yml`, group files, directories | -| `/ignatius-modeling flow` | Interactive Q&A to author a [data flow diagram](/modeling/data-flows) | -| `/ignatius-modeling discover` | Socratic interview that works the model out from how your business runs, generating entities and flows | -| `/ignatius-modeling` | Prompts you to choose a mode | +| `/ignatius-modeling entity` | Add one entity | +| `/ignatius-modeling model` | Bootstrap `ignatius.yml`, group files, and folders | +| `/ignatius-modeling flow` | Author a [data flow diagram](/modeling/data-flows) when you can already name your processes | +| `/ignatius-modeling discover` | Work out entities and flows from how the business runs. With a real database, schema dump, codebase, or API, it reads that instead of interviewing you | +| `/ignatius-modeling` | Asks which mode | +Every entity gets example rows and every process gets input and output examples, because concrete rows expose wrong rules that pass every structural check. The skill picks `key-inherited` or `orm-oriented` for each new entity from the keys you already have; see [the two key conventions](/modeling/entities). To point `discover` at a live database, see [Reverse-engineering](/modeling/reverse-engineering). -## Choosing between flow and discover +## What does validation catch? -`flow` and `discover` are two doors into the same artifacts. -Pick `flow` when you can already name your processes. It walks the structure step by step: processes as verbs, externals, the decision between a `db:` store and a `kind:` store, the data each flow carries, sample rows, and the business narrative. +After each write the skill runs `ignatius validate <model-root>`, reports each finding with a fix hint, and asks whether to revise. It stops after five attempts. -Pick `discover` when you know what the business does but have not decomposed it yet. The skill interviews you in plain language, derives the entities your processes require, writes those first, then writes the flows that reference them. When a real system already exists, whether a database, a schema dump, a codebase, or an API, `discover` reads it instead of interviewing you and then walks you through the judgment calls it cannot infer. To point it at a live database, see [Reverse-engineering an existing database](/modeling/reverse-engineering). +| Check | Done by | +|---|---| +| Unknown targets, dangling foreign-key columns, missing discriminators, unknown columns in flow contracts, unbalanced decompositions | `ignatius validate` | +| Example rows with a key that is not a column | Only `ignatius serve` shows it. `validate` never prints that rule, so the skill reads the rows back | +| Each predicate reads as a true sentence both ways, and the body states the purpose and rules | The skill's self-check, then you | -Both modes always produce example data. Every entity gets sample rows and every process gets input and output tables, because concrete instances expose wrong rules that pass every structural check. +## How do I review what it changed? -## Convention detection +Every session that changes model files ends with one change note, `.ignatius/changes/<slug>.md`. It names the entities, columns, diagrams, and nodes the session touched and says why. `ignatius serve` shows the edit as highlights beside the note. `ignatius approve` accepts it. -The skill works out which key convention a model uses from the shape of its existing entities rather than asking you to choose a mode up front, and a model can mix both styles per entity. +Before its first write, the skill checks for a baseline so that the edit shows as a change: -| Convention | Primary key shape | Foreign key placement | +| `.ignatius/baseline.json` | `.ignatius/changes/` | The skill | |---|---|---| -| `key-inherited` | Composite: parent primary-key columns plus a local discriminator | Inside the child's primary key | -| `orm-oriented` | A single surrogate `id`, typically integer or uuid | Outside the primary key, as plain columns | +| valid | any | edits as usual | +| missing or invalid | empty | runs `ignatius approve`, then edits | +| missing or invalid | holds notes | stops and asks you to approve the notes or restore the baseline | -You never set `classification` or `identifying` by hand in either convention. The parser derives both from the key shape you describe. See [what gets derived](/modeling/entities#what-gets-derived). +The skill never runs `approve` after editing. Accepting the edit is your step. The `model` mode writes no note, and a model it creates gets a `.gitignore` containing `.ignatius/`. The note format is in [Change review](https://github.com/noormdev/ignatius/blob/main/docs/guides/change-review.md). -## The verification loop +## What bites -After writing each file the skill runs `ignatius validate <model-root>` and parses the findings from stderr. Findings are reported with fix hints, and you can ask the skill to revise and re-run, up to five attempts. A clean run with no findings confirms the file is valid. - -This is the part that makes agent-authored models trustworthy. The skill is not asked to be right on the first try. It is asked to converge against a linter that checks unknown targets, dangling foreign-key columns, missing discriminators, unknown columns in flow contracts, and unbalanced decompositions. - - -## From model to schema - - -A validated model is a specification, and the same markdown a stakeholder reviewed is what the agent reads to build. - -Once the model is agreed, point Claude Code at the model root and ask it to write the SQL. The entity files carry the keys, the columns, the relationships, and the prose explaining what each one is for, which is more context than a schema request usually gets. From there, [noorm](/getting-started/installation) takes over: SQL files in directories, `noorm run build` to build a fresh database, and changes for existing ones. - -The division of labor is worth stating plainly. ignatius owns the decision about what the data is. noorm owns building and versioning it. Neither tool reaches into the other's job, and both keep the source of truth in files you own. +| Symptom | Cause | Fix | +|---|---|---| +| The skill stops before its first write | `.ignatius/changes/` holds notes and there is no baseline | Read the notes, run `ignatius approve`, then continue | +| A change shows as `unexplained` | The note does not name that entity, column, diagram, or node | Add the name to the note | +| A note shows `named but unchanged` | The note names an entity, column, diagram, or node that did not change | Remove the name, or make the change | +| A note you wanted is gone | `ignatius approve` deletes every note | Copy anything worth keeping into the entity or flow body first | -## Related +## Next -- [Entities and key inheritance](/modeling/entities) for the format the skill writes. -- [Data flows](/modeling/data-flows) for the diagram format. -- [Best practices](/modeling/best-practices) for how to review what the skill produces. +- [Entities and key inheritance](/modeling/entities): the format the skill writes +- [Data flows](/modeling/data-flows): the diagram format +- [From model to SQL](/modeling/model-to-sql): hand the validated model to an agent that writes the DDL +- [Best practices](/modeling/best-practices): review what the skill produces diff --git a/docs/modeling/reverse-engineering.md b/docs/modeling/reverse-engineering.md index bbb049b5..476569ab 100644 --- a/docs/modeling/reverse-engineering.md +++ b/docs/modeling/reverse-engineering.md @@ -1,189 +1,143 @@ --- title: Reverse-engineering an existing database -description: Point an agent at a database you already run and have it extract an ignatius model, using noorm's MCP server for read-only schema access. +description: Point an agent at a database you already run, read through noorm's MCP server, and have the ignatius-modeling skill extract an ignatius model from it. --- # Reverse-engineering an existing database - -Most databases worth modeling already exist. The schema is in production, the reasoning behind it left with someone two jobs ago, and the only description of it is the DDL. - -You do not have to model that by hand. noorm exposes the schema to a coding agent over MCP, and the [`ignatius-modeling` skill](/modeling/modeling-skill) knows how to turn what it reads into entity and flow files. Your job is to set up the access, keep it read-only, and review what comes back. - -The result is an ignatius model of the system you already run: a diagram to hand to people, a dictionary of what every table is for, and a base to plan the next change against. - - -## Where this sits in IDEF1X - - -Extraction is not a detour around the methodology. It is a recognized path inside it. IDEF1X calls this the **bottom-up** approach, inferring a logical model from the physical artifacts that already exist, and the full round trip of reverse-engineer, adjust for new requirements, then forward-engineer is **re-engineering**. - -The discipline is also specific about how far bottom-up gets you, which is worth knowing before you start rather than after. A *data model* describes a physical database. An *information model* describes the logical requirements a system satisfies. Reverse-engineering climbs from the first toward the second in levels: - -| Level | What you get | What it is worth | -|---|---|---| -| 1 | Current-system documentation, a physical model | Guaranteed only to show what **is** | -| 2 | Application-level logical model | The requirements the system currently satisfies, with no claim about whether they are right | -| 3 | Business-level model | A basis for future design, and unreachable without the business participating directly | - -An agent with schema access reaches level 1 quickly and gets a good way into level 2. It cannot reach level 3 on its own, because the information at that level was never in the database. It is in the heads of the people who run the business, and the only way out is to ask them. - -The IDEF1X literature is blunt about the risk, and the warning is the reason this page ends with a review step rather than a success message: **a model of what you have is not a model of what you want.** Assumptions compound as you climb the levels, and their source is the past, mistakes included. That is also why the skill records anti-patterns instead of quietly correcting them. Faithful first, better second, with the "better" being a conversation you have afterward rather than something extraction decides for you. - -The [`idef1x` skill](https://www.skills.sh/damusix/skills/idef1x) carries the methodology itself if you want an agent applying it alongside you. - - -## What you need - -- noorm installed, with a config that connects to the database ([installation](/getting-started/installation)) -- ignatius installed ([installation](/modeling/installation)) -- The `ignatius-modeling` skill, and optionally the noorm skill for SDK and CLI conventions -- An agent harness that speaks MCP, such as Claude Code or Cursor - - -## 1. Create the modeling space - - -A model root is any folder with an `ignatius.yml` in it. Let the skill scaffold one: - -```bash -/ignatius-modeling model +Extract an ignatius model from a database you already run, with an agent that reads the schema through noorm's MCP server. + +The agent reads the database through noorm, writes the model files, and checks them. You review the result: + +```mermaid +sequenceDiagram + participant You + participant Agent + participant MCP as noorm MCP + participant DB as database + participant Files as model files + participant Validate as ignatius validate + You->>Agent: /ignatius-modeling discover + Agent->>MCP: run_noorm_cmd list, detail + MCP->>DB: read system catalogs + DB-->>MCP: tables, columns, keys + MCP-->>Agent: JSON results + Agent->>Files: write entity and flow files + Agent->>Validate: check the model folder + Validate-->>Agent: findings + Agent->>You: questions the schema cannot answer + You->>Files: ignatius serve -o ``` -That writes `ignatius.yml`, the group files, and the directories. To do it by hand, a one-line `ignatius.yml` is a valid model root: +`noorm mcp init` runs once per project. The slash command runs in the agent: ```bash -mkdir -p models/legacy && echo "name: Legacy Schema" > models/legacy/ignatius.yml +noorm mcp init ``` -Put it in the repository whose schema it describes, so the model and the code stay in one history. - -The rest of this page refers to that folder as `models/legacy`. Substitute wherever yours actually landed, or leave the path off entirely: `serve`, `export`, and `validate` all take the model root as an optional argument and find it by searching up and down from the current directory when you omit it. - - -## 2. Connect noorm over MCP - - -noorm runs an MCP server over stdio. Generate the agent's config: - -```bash -noorm mcp init # Claude Code, writes .mcp.json -noorm mcp init --agent cursor # Cursor, writes .cursor/mcp.json +``` +/ignatius-modeling discover ``` -The agent picks up the server on next launch and gets two tools: `noorm_help` to list commands and `run_noorm_cmd` to execute them. The schema explorer, the SQL runner, and config management all arrive through those. -Make sure the config you want the agent reading is the active one, and that it points at the database you mean. Reverse-engineering staging when you meant production produces a confident model of the wrong thing. +## What do I set up first? -## 3. Keep the config read-only +1. A model root: any folder with an `ignatius.yml`. `/ignatius-modeling model` scaffolds one. `mkdir -p models/legacy && echo "name: Legacy Schema" > models/legacy/ignatius.yml` is the minimum. Keep it in the repository whose schema it describes. +2. The agent's MCP config. `noorm mcp init` writes `.mcp.json` for Claude Code, and `noorm mcp init --agent cursor` writes `.cursor/mcp.json`. The agent gets two tools, `noorm_help` and `run_noorm_cmd`. +3. A noorm config for the database you mean. The agent connects to the active config unless it names another. A model of staging is not a model of production. +4. The skill and the binary, both on [Download](/download). -Every config carries a role per channel, `access: { user, agent }`, and the `agent` role governs what a coding agent can do regardless of whether it comes in over MCP or shells out to the `noorm` binary. noorm resolves the channel from the harness that spawned it, so an agent refused over MCP cannot route around the refusal on the command line. +## How do I keep the agent read-only? -| Role | What the agent can do | -|---|---| -| `viewer` | Explore schema, run read-only SQL (`SELECT`, `SHOW`, `DESCRIBE`). `EXPLAIN` inherits the class of the statement it wraps, so `EXPLAIN ANALYZE DELETE ...` is a write and is denied | -| `operator` | Adds `INSERT`/`UPDATE`/`DELETE`; destructive commands stay out of reach | -| `admin` | Adds DDL, change and run commands, and vault writes. The irreversible operations (drop, teardown, truncate, config rewrite, vault propagate, force-unlock) are confirmation-gated, and a confirmation collapses to a denial on the agent channel | -| `false` | The config is invisible on that channel | -`viewer` is all reverse-engineering needs, and it is already the default: a config that never declared `access` gets `{ user: 'admin', agent: 'viewer' }`. So this step is usually a check rather than a change. Confirm it in `noorm ui` under Config → Edit, and lower it back to `viewer` if someone raised it for other work. +A config that never declared `access` gives agents `viewer`: schema exploration and read-only SQL. That is all this job needs, so this step is usually a check. Confirm it in `noorm ui` under Config, then Edit. -Raw SQL is classified by what the statement does rather than which command asked, so an agent on `viewer` gets a `SELECT` through and a same-shaped `INSERT` denied. +The role covers MCP and an agent that shells out to `noorm`. See [MCP](/guide/automation/mcp) for the full role table. ::: warning Roles prevent accidents, not attacks -The `agent` role is a guardrail against a mistake, not a security boundary. It stops an agent from doing damage it never meant to do, which is the realistic failure. It does not stop one that has been talked into it: `NOORM_CHANNEL` is an environment variable, so anything able to influence the agent's instructions or its environment can reach around the role. - -That is worth weighing here specifically, because reverse-engineering points an agent at a large amount of text you did not write. Table comments, column descriptions, and stored procedure bodies are all content a prompt injection can ride in on. - -So for a database you care about, connect noorm with a database user that holds only `SELECT` and catalog permissions. Then the role is the second line of defense rather than the only one, and an instruction that gets past it still cannot write anything. +An agent that shells out can set `NOORM_CHANNEL=user`. Reverse-engineering also feeds the agent table comments, column descriptions, and procedure bodies that you may not have written. Connect noorm with a database user that holds only `SELECT` and catalog permissions, so the role is the second line of defense. ::: -## 4. Give the database user catalog access - - -The explorer reads your database's system catalogs. Which ones depends on the dialect: - -| Dialect | Catalog noorm reads | -|---|---| -| PostgreSQL | `information_schema` views (`columns`, `table_constraints`, `key_column_usage`, `referential_constraints`, `parameters`) | -| MySQL | `information_schema` views (`columns`, `key_column_usage`, `referential_constraints`, `routines`, `parameters`) | -| SQL Server | `sys` catalog views (`sys.columns`, `sys.foreign_keys`, `sys.foreign_key_columns`, `sys.index_columns`) | -| SQLite | `sqlite_master` and `PRAGMA` (file access, no grants) | - -This is the step that quietly ruins a model, because a permissions gap does not raise an error. It returns fewer rows. - -PostgreSQL filters the information schema by privilege: "Only those tables and views are shown that the current user has access to (by way of being the owner or having some privilege)." SQL Server behaves the same way through metadata visibility, where "the visibility of metadata is limited to securables that a user either owns or on which the user has been granted some permission," and a user with no permission on a table gets an empty result set rather than a denial. - -An agent reading through an under-privileged user therefore produces a model that looks finished and is silently missing tables. Grant catalog access before you start: - -```sql --- PostgreSQL: visibility follows privilege -GRANT USAGE ON SCHEMA public TO modeler; -GRANT SELECT ON ALL TABLES IN SCHEMA public TO modeler; +## What does the database user need? --- MySQL: same idea, per schema -GRANT SELECT ON mydb.* TO 'modeler'@'%'; --- SQL Server: metadata visibility without data access -GRANT VIEW DEFINITION TO modeler; -``` - -Then check the count. Ask the agent, or run it yourself, for the table list noorm sees and compare it against what you know is there. A mismatch here is cheaper to catch now than after the model is written. - - -## 5. Ask the agent to reverse-engineer +A permissions gap does not raise an error. The catalogs return fewer rows, so the model looks finished and misses tables. [PostgreSQL](https://www.postgresql.org/docs/current/infoschema-tables.html), [MySQL](https://dev.mysql.com/doc/refman/8.0/en/information-schema-introduction.html), and [SQL Server](https://learn.microsoft.com/en-us/sql/relational-databases/security/metadata-visibility-configuration) each show only objects the user owns or holds a privilege on. +| Dialect | noorm reads | Grant | +|---|---|---| +| PostgreSQL | `information_schema`, `pg_catalog` | `GRANT USAGE ON SCHEMA public TO modeler;` and `GRANT SELECT ON ALL TABLES IN SCHEMA public TO modeler;` | +| MySQL | `information_schema` | `GRANT SELECT ON mydb.* TO 'modeler'@'%';` | +| SQL Server | `sys` catalog views | `GRANT VIEW DEFINITION TO modeler;` | +| SQLite | `sqlite_master`, `PRAGMA` | none, file access | -Launch the harness and point it at both the database and the model root: +Before the agent writes anything, have it run `overview`, which returns a count for each object type. Compare the table count with what you know is there. -``` -/ignatius-modeling discover -``` -`discover` interviews you when nothing exists yet. When a real system is available it reads that instead, and interviews you only about the judgment calls the schema cannot answer. Tell it where the model root is and that noorm's MCP server is connected. +## What does the agent extract? -Code is worth offering alongside the schema. Foreign keys give the agent your entities and relationships, but only the handlers, jobs, and stored procedures that touch the data can give it the processes for the [data flow diagrams](/modeling/data-flows). A `SELECT` inside a request handler is an input flow with a column list; an `INSERT` is an output flow. Point the agent at the repository and the flows come out at column level rather than as guesses. +Foreign keys give the agent entities and relationships. Only code gives it processes, so offer the repository too. -## 6. What a good extraction does +| In the database or code | In the model | +|---|---| +| Table | Entity, named exactly as the table. `sales_orders` stays `sales_orders` | +| Column | Attribute, nullability kept. A `NOT NULL` foreign key is a mandatory parent | +| Primary key | `key-inherited` when it contains a foreign key, `orm-oriented` for a lone surrogate `id` | +| Foreign key | Relationship. Whether it is identifying comes from where the key sits | +| Table whose whole primary key is its parent's | Subtype member | +| `SELECT` in a handler or procedure | Input flow with that column list | +| `INSERT` in a handler or procedure | Output flow with those columns | +| A few real rows per table | `examples:`, with PII and secrets masked | +The skill records anti-patterns instead of fixing them. A junk-drawer table or a surrogate key where the real identity is composite is captured as it is, then raised with you as a question. -The skill's method is worth knowing, because it is what you review against. -**It reads rather than invents.** Entities come from tables, attributes from columns with nullability preserved, relationships from foreign-key constraints, and subtype clusters from tables whose whole primary key equals their parent's. Whether an edge is identifying is derived from the key shape, never declared. +## How far does extraction get? -**It keeps the source's names.** `sales_orders` stays `sales_orders`. The entity id is what every relationship target, wiki link, and `db:` token has to match, so renaming during extraction breaks the correspondence with the real system. A rename is a decision to surface, not a cleanup to perform quietly. -**It detects your key convention instead of imposing one.** A composite primary key containing a foreign key reads as `key-inherited`; a lone surrogate `id` with foreign keys outside the key reads as `orm-oriented`. Mixed schemas stay mixed. See [the two key conventions](/modeling/entities#the-two-key-conventions). +IDEF1X names the approach bottom-up and sorts its results into levels: -**It records anti-patterns rather than fixing them.** Faithful first, better second. A junk-drawer table or a surrogate key where the real identity is composite gets captured as it is, then raised with you as a question. Silent improvement during extraction produces a model of a database you do not have. +| Level | You get | Worth | +|---|---|---| +| 1 | Current-system documentation | Shows what **is** | +| 2 | Application-level logical model | The requirements the system satisfies today, right or wrong | +| 3 | Business-level model | A basis for future design. Needs the business in the room | -**It grounds the model in real rows.** Sample data from the live system makes the best examples, which is why every entity and process should carry some. Mask anything sensitive before it lands in a file you commit. +We read a schema as enough for level 1 and part of level 2. Level 3 lives with the people who run the business. A model of what you have is not a model of what you want. The [`idef1x` skill](https://www.skills.sh/damusix/skills/idef1x) carries the method. -## 7. Verify and review +## How do I review the result? ```bash -ignatius validate models/legacy # or just `ignatius validate` from inside the project +ignatius validate models/legacy ignatius serve models/legacy -o ``` -`validate` exits `1` on errors and `0` otherwise, so it works as a gate. A clean run means the model is internally consistent. It does not mean the model is right, and the two failures worth hunting are ones no linter catches: a table the agent could not see, and a purpose it invented because the schema could not tell it one. +`validate` exits `1` on errors and `0` otherwise, so it also gates a pull request. A clean run means the model is consistent. Two failures pass it: a table the agent could not see, and a purpose the agent had to guess. An entity body that restates its columns is the second one. + +The skill ends the session with a change note. `serve` shows the edit beside it, and `ignatius approve` accepts it; see [the modeling skill](/modeling/modeling-skill). Walk the people who run the business through the served model. -Read the entity bodies with that in mind. Any entity whose description restates its columns rather than explaining what it is for is a question the agent could not answer from the schema, and the answer is in your head or nobody's. Those paragraphs are the part of the model worth the most later, and the part only you can supply. -This is the climb from level 2 to level 3, and it is the work reverse-engineering cannot do for you. An extracted model tells you what the database is. Turning that into what the business means, and then into what it should be, takes the people who run it. Serve the model, walk them through it, and let the arguments start. +## What bites + + +| Symptom | Cause | Fix | +|---|---|---| +| The model is clean and tables are missing | The database user cannot see them | Grant catalog access, then compare `overview` counts | +| The agent cannot find your config | `access.agent` is `false`, which hides the config from agents | Set `agent` to `viewer` | +| Links and `db:` tokens break | An entity id no longer matches its table name | Restore the source name. Raise a rename as a question | +| Real customer values in a committed file | `examples:` come from live rows | Mask them before you commit | +| An entity body restates its columns | The schema cannot say what the entity is for | Write the purpose yourself | -## Related +## Next -- [Entities and key inheritance](/modeling/entities) -- [Data flows](/modeling/data-flows) -- [The modeling skill](/modeling/modeling-skill) -- [MCP (AI agent integration)](/guide/automation/mcp) for the server itself -- [Access roles](/guide/environments/configs#access-roles) for channel detection and the `NOORM_CHANNEL` override +- [The modeling skill](/modeling/modeling-skill): the modes and the change-review loop +- [Entities and key inheritance](/modeling/entities): the files the agent writes +- [Data flows](/modeling/data-flows): processes, externals, and stores +- [MCP](/guide/automation/mcp): the server and its access roles diff --git a/docs/modeling/why.md b/docs/modeling/why.md new file mode 100644 index 00000000..87298f2a --- /dev/null +++ b/docs/modeling/why.md @@ -0,0 +1,95 @@ +--- +title: Why ignatius +description: The case for modeling before building, who reads the model, why ignatius is a separate tool from noorm, and where the name comes from. +--- + +# Why ignatius + + +Most schemas get designed while they are being built, so the shape of the whole lives in migration history and in the head of whoever wrote it. [ignatius](https://github.com/noormdev/ignatius) moves that design earlier. You describe entities, keys, and relationships in markdown, and it renders a diagram people can argue about before anyone writes SQL. It never connects to a database and never generates DDL. + +<!-- Wrapped in a div because `video` is not a CommonMark block-level tag: + left bare, markdown-it splits the multi-line element across paragraphs and + Vue's SFC parser then rejects the orphaned closing tag. --> +<div class="demo-video"> +<video src="/video/ignatius.mp4" poster="/video/ignatius-poster.jpg" width="1316" height="858" controls muted playsinline preload="none"></video> +</div> + +Above: a model of an LLM memory database in all three views. + + +## What does it produce? + + +One folder of markdown drives three views of the same model. + +| View | Notation | Answers | Shows | +|---|---|---|---| +| Data Graph | IDEF1X | What is the data, and how is it identified? | Entities in crow's-foot notation, identifying relationships, compound keys that carry their lineage, subtype clusters | +| Data Dictionary | none | What is each thing for? | Every entity, process, external, and store on one searchable page | +| Data Flows | SSADM data flow diagrams | Who touches the data, and where does it live? | Processes, external actors, and stores, where a store is any place data rests: a table, a cache, a log file, a queue, a paper form | + +See [Entities and key inheritance](/modeling/entities) and [Data flows](/modeling/data-flows). + +You write keys and relationships as text, and ignatius derives cardinality, entity classification, and subtype clusters from the key shape. Change a foreign key and the notation updates on the next reload. The model diffs and reviews like code. + +A table list says what exists. The graph also shows how entities connect, where identity comes from, and which clusters share a basetype. Each entity carries prose on why it exists, plus example rows, and a sample row that violates a constraint you believed in exposes a modeling error no structural check will catch. + + +## Why model before building? + + +The industry mostly skips modeling and goes straight to building. In my experience, modeling first solves about 80% of the problems up front and gives months of valid programming work before anyone opens an editor. + +The other 20% is real: once you start building, you do find problems in the model. What matters is when. While nothing is committed to the system, changing the model costs an edit to a text file. Once it is code, it does not. + +Review sits between writing the model and writing the schema, so a disagreement about the data surfaces while it is cheap: + +```mermaid +flowchart TD + W["Write entity and<br/>flow files"] --> V["ignatius validate"] + V --> S["ignatius serve<br/>or export"] + S --> D{"Stakeholders agree?"} + D -->|no| W + D -->|yes| B["Agent writes the schema<br/>from the same files"] + B --> N["noorm builds and<br/>versions the SQL"] +``` + + +## Who reads the model? + + +| Reader | Gets | How | +|---|---|---| +| Stakeholder | A diagram to explore, a dictionary to search, and prose on business meaning | `ignatius export` writes one self-contained HTML file that opens from `file://` with nothing installed | +| Coding agent | Plain markdown with YAML frontmatter to read and write | The [`ignatius-modeling` skill](/modeling/modeling-skill) authors entity and flow files through guided Q&A and checks each with `ignatius validate` | + + +## Why is it a separate tool from noorm? + + +noorm already explores schemas and runs SQL, so it is fair to ask why modeling is not a `noorm model` subcommand. Three reasons: + +- **It never touches a database.** ignatius carries no database driver and emits no DDL. It is useful before the database exists, when a tool that needs a connection has nothing to offer. +- **It does not need noorm.** Nothing in the format is specific to noorm: markdown in, IDEF1X and SSADM diagrams out, no dialect anywhere. A team on any other migration tool gets the same thing. +- **The audiences differ.** `ignatius export` produces a file for the person approving a design. That reader has no reason to install a schema manager. + + +## Where does the name come from? + + +ignatius is named for Derek Ignatius Asirvadem, who mentored me for a couple of years and introduced me to IDEF1X. NIST published IDEF1X as FIPS 184 in 1993, and it became part of ISO/IEC/IEEE 31320-2 in 2012. Studying it and then practising it convinced me there is no higher standard for modeling data. The [landing page](/modeling/) traces its lineage back to Codd in 1970. + +Before ignatius I drew models in OmniGraffle, on Derek's recommendation. My own obsession with perfect lines and names turned drawing into the bottleneck, and a picture cannot tell an LLM how two entities relate. So the model moved into markdown, where the layout is generated and an agent can read and write the source. + +None of these tools would exist without his guidance. ignatius is a tribute to him. + + +## Next + + +- [ignatius commands](/modeling/installation): install, serve, validate, export +- [Entities and key inheritance](/modeling/entities) +- [Data flows](/modeling/data-flows) +- [Reverse-engineering](/modeling/reverse-engineering): extract a model from a database you already run +- [Best practices](/modeling/best-practices) diff --git a/docs/models/order-to-cash/data/billing/invoice.md b/docs/models/order-to-cash/data/billing/invoice.md new file mode 100644 index 00000000..a39358c2 --- /dev/null +++ b/docs/models/order-to-cash/data/billing/invoice.md @@ -0,0 +1,28 @@ +--- +entity: invoice +group: billing +pk: + - party_id + - invoice_no +columns: + party_id: + type: integer + invoice_no: + type: integer + issued_at: + type: datetime + default: now +examples: + - party_id: 2 + invoice_no: 1 + issued_at: "2026-03-05T14:25:00Z" +relationships: + - target: party + on: + party_id: party_id + predicate: { fwd: owes on, rev: is owed by } +--- + +# invoice + +A bill a [[party]] owes on. diff --git a/docs/models/order-to-cash/data/billing/invoice_line.md b/docs/models/order-to-cash/data/billing/invoice_line.md new file mode 100644 index 00000000..43747844 --- /dev/null +++ b/docs/models/order-to-cash/data/billing/invoice_line.md @@ -0,0 +1,50 @@ +--- +entity: invoice_line +group: billing +pk: + - party_id + - invoice_no + - line_no +columns: + party_id: + type: integer + invoice_no: + type: integer + line_no: + type: integer + order_no: + type: integer + order_line_no: + type: integer + amount: + type: decimal +examples: + - party_id: 2 + invoice_no: 1 + line_no: 1 + order_no: 1 + order_line_no: 1 + amount: 49.00 + - party_id: 2 + invoice_no: 1 + line_no: 2 + order_no: 1 + order_line_no: 2 + amount: 89.00 +relationships: + - target: invoice + on: + party_id: party_id + invoice_no: invoice_no + predicate: { fwd: contains, rev: is part of } + - target: order_line + on: + party_id: party_id + order_no: order_no + order_line_no: line_no + predicate: { fwd: is billed by, rev: bills } +--- + +# invoice_line + +Bills one [[order_line]]. Both foreign keys share `party_id`, so an invoice can only bill its own party's orders. diff --git a/docs/models/order-to-cash/data/billing/payment.md b/docs/models/order-to-cash/data/billing/payment.md new file mode 100644 index 00000000..8aa03201 --- /dev/null +++ b/docs/models/order-to-cash/data/billing/payment.md @@ -0,0 +1,30 @@ +--- +entity: payment +group: billing +pk: + - party_id + - payment_no +columns: + party_id: + type: integer + payment_no: + type: integer + paid_at: + type: datetime + amount: + type: decimal +examples: + - party_id: 2 + payment_no: 1 + paid_at: "2026-03-09T10:00:00Z" + amount: 138.00 +relationships: + - target: party + on: + party_id: party_id + predicate: { fwd: makes, rev: is made by } +--- + +# payment + +Money a [[party]] paid. diff --git a/docs/models/order-to-cash/data/billing/payment_allocation.md b/docs/models/order-to-cash/data/billing/payment_allocation.md new file mode 100644 index 00000000..6f294ee4 --- /dev/null +++ b/docs/models/order-to-cash/data/billing/payment_allocation.md @@ -0,0 +1,47 @@ +--- +entity: payment_allocation +group: billing +pk: + - party_id + - payment_no + - invoice_no + - line_no +columns: + party_id: + type: integer + payment_no: + type: integer + invoice_no: + type: integer + line_no: + type: integer + amount: + type: decimal +examples: + - party_id: 2 + payment_no: 1 + invoice_no: 1 + line_no: 1 + amount: 49.00 + - party_id: 2 + payment_no: 1 + invoice_no: 1 + line_no: 2 + amount: 89.00 +relationships: + - target: payment + on: + party_id: party_id + payment_no: payment_no + predicate: { fwd: is applied as, rev: is drawn from } + - target: invoice_line + on: + party_id: party_id + invoice_no: invoice_no + line_no: line_no + predicate: { fwd: is settled by, rev: settles } +--- + +# payment_allocation + +How much of a [[payment]] settles which [[invoice_line]]. Both parents contribute `party_id` to one column, so a payment from party 1 cannot settle party 2's invoice. The schema enforces that with two foreign keys and no trigger. diff --git a/docs/models/order-to-cash/data/parties/business.md b/docs/models/order-to-cash/data/parties/business.md new file mode 100644 index 00000000..dbfb5c22 --- /dev/null +++ b/docs/models/order-to-cash/data/parties/business.md @@ -0,0 +1,30 @@ +--- +entity: business +group: parties +pk: + - party_id +columns: + party_id: + type: integer + legal_name: + type: text + tax_id: + type: text +ak: + - rule: is identified by tax id + columns: + - tax_id +examples: + - party_id: 1 + legal_name: Meridian Analytics LLC + tax_id: "47-3819204" +relationships: + - target: party + on: + party_id: party_id + predicate: { fwd: is realized as, rev: is a } +--- + +# business + +A [[party]] that is a legal entity. `tax_id` is unique, but it is not the key: identity comes from the party. diff --git a/docs/models/order-to-cash/data/parties/party.md b/docs/models/order-to-cash/data/parties/party.md new file mode 100644 index 00000000..74bf70c9 --- /dev/null +++ b/docs/models/order-to-cash/data/parties/party.md @@ -0,0 +1,40 @@ +--- +entity: party +group: parties +pk: + - party_id +columns: + party_id: + type: integer + desc: "Root identity. A surrogate is right here: nothing above party supplies one." + party_type: + type: text + desc: "Which subtype this party is." + name: + type: text + desc: "Display name." +examples: + - party_id: 1 + party_type: BUSINESS + name: Meridian Analytics + - party_id: 2 + party_type: PERSON + name: Ada Lovelace +subtypes: + - exclusive: true + desc: Every party is exactly one of person or business + members: + person: + party_type: party_type.code.PERSON + business: + party_type: party_type.code.BUSINESS +relationships: + - target: party_type + on: + party_type: code + predicate: { fwd: classifies, rev: is classified by } +--- + +# party + +Anyone who places orders and owes on invoices. Every table in sales and billing inherits `party_id`, which is what lets the database stop one customer's payment from settling another customer's invoice. diff --git a/docs/models/order-to-cash/data/parties/party_type.md b/docs/models/order-to-cash/data/parties/party_type.md new file mode 100644 index 00000000..eecbd5ee --- /dev/null +++ b/docs/models/order-to-cash/data/parties/party_type.md @@ -0,0 +1,22 @@ +--- +entity: party_type +reference: true +group: parties +pk: + - code +columns: + code: + type: text + desc: "PERSON or BUSINESS." + description: + type: text +examples: + - code: PERSON + description: A natural person + - code: BUSINESS + description: A legal entity +--- + +# party_type + +The discriminator values for the [[party]] subtype cluster. New kinds of party are added as rows, not as code. diff --git a/docs/models/order-to-cash/data/parties/person.md b/docs/models/order-to-cash/data/parties/person.md new file mode 100644 index 00000000..81545ba3 --- /dev/null +++ b/docs/models/order-to-cash/data/parties/person.md @@ -0,0 +1,26 @@ +--- +entity: person +group: parties +pk: + - party_id +columns: + party_id: + type: integer + given_name: + type: text + family_name: + type: text +examples: + - party_id: 2 + given_name: Ada + family_name: Lovelace +relationships: + - target: party + on: + party_id: party_id + predicate: { fwd: is realized as, rev: is a } +--- + +# person + +A [[party]] that is a natural person. Shares its identity with the party; it never mints its own. diff --git a/docs/models/order-to-cash/data/sales/order_line.md b/docs/models/order-to-cash/data/sales/order_line.md new file mode 100644 index 00000000..1440ec48 --- /dev/null +++ b/docs/models/order-to-cash/data/sales/order_line.md @@ -0,0 +1,49 @@ +--- +entity: order_line +group: sales +pk: + - party_id + - order_no + - line_no +columns: + party_id: + type: integer + order_no: + type: integer + line_no: + type: integer + product_id: + type: integer + qty: + type: integer + unit_price: + type: decimal + desc: "Price agreed at order time, copied from the catalog." +examples: + - party_id: 2 + order_no: 1 + line_no: 1 + product_id: 10 + qty: 1 + unit_price: 49.00 + - party_id: 2 + order_no: 1 + line_no: 2 + product_id: 11 + qty: 1 + unit_price: 89.00 +relationships: + - target: sales_order + on: + party_id: party_id + order_no: order_no + predicate: { fwd: contains, rev: is part of } + - target: product + on: + product_id: product_id + predicate: { fwd: is sold via, rev: sells } +--- + +# order_line + +One product on a [[sales_order]]. The key carries the whole lineage, so the owning party is on every row without a join. diff --git a/docs/models/order-to-cash/data/sales/product.md b/docs/models/order-to-cash/data/sales/product.md new file mode 100644 index 00000000..962b59ab --- /dev/null +++ b/docs/models/order-to-cash/data/sales/product.md @@ -0,0 +1,32 @@ +--- +entity: product +group: sales +pk: + - product_id +columns: + product_id: + type: integer + sku: + type: text + name: + type: text + list_price: + type: decimal +ak: + - rule: is identified by sku + columns: + - sku +examples: + - product_id: 10 + sku: HW-DONGLE + name: USB security key + list_price: 49.00 + - product_id: 11 + sku: SW-SEAT + name: Desktop seat, yearly + list_price: 89.00 +--- + +# product + +The catalog. Seeded from `sql/09_seeds/products.yml` in the noorm project. diff --git a/docs/models/order-to-cash/data/sales/sales_order.md b/docs/models/order-to-cash/data/sales/sales_order.md new file mode 100644 index 00000000..1f7f2dd8 --- /dev/null +++ b/docs/models/order-to-cash/data/sales/sales_order.md @@ -0,0 +1,24 @@ +--- +entity: sales_order +group: sales +pk: [party_id, order_no] +columns: + party_id: { type: integer } + order_no: { type: integer } + placed_at: { type: datetime, default: now } + status: { type: text, default: open } +examples: + - { party_id: 2, order_no: 1, status: invoiced } + - { party_id: 2, order_no: 2, status: open } + - { party_id: 1, order_no: 1, status: paid } +relationships: + - target: party + on: { party_id: party_id } + predicate: + fwd: places + rev: is placed by +--- + +# sales_order + +An order a [[party]] places. `order_no` counts within the party, so party 2's orders are 1, 2, 3 and party 1's start again at 1. `place_order()` assigns it; it is never a timestamp. diff --git a/docs/models/order-to-cash/externals/Customer.md b/docs/models/order-to-cash/externals/Customer.md new file mode 100644 index 00000000..ce2cdaee --- /dev/null +++ b/docs/models/order-to-cash/externals/Customer.md @@ -0,0 +1,5 @@ +--- +external: Customer +--- + +The person or business placing orders and paying invoices. Recorded as a [[party]] once known. diff --git a/docs/models/order-to-cash/flows/order-to-cash/Collect-Payment.md b/docs/models/order-to-cash/flows/order-to-cash/Collect-Payment.md new file mode 100644 index 00000000..bbfe698c --- /dev/null +++ b/docs/models/order-to-cash/flows/order-to-cash/Collect-Payment.md @@ -0,0 +1,27 @@ +--- +process: Collect payment +number: 3 +description: "Records a payment and allocates it against invoice lines." +inputs: + - from: ext:Customer + label: payment + data: card, amount + - from: db:invoice_line + label: open lines + data: [party_id, invoice_no, line_no, amount] +outputs: + - to: db:payment + label: payment + data: [party_id, payment_no, amount] + - to: db:payment_allocation + label: allocations + data: [party_id, payment_no, invoice_no, line_no, amount] + - to: file:gateway-log + label: gateway response + data: transaction reference, status, raw response + - to: ext:Customer + label: receipt + data: payment number, status +--- + +Writes the [[payment]] and its [[payment_allocation]] rows in one transaction. diff --git a/docs/models/order-to-cash/flows/order-to-cash/Issue-Invoice.md b/docs/models/order-to-cash/flows/order-to-cash/Issue-Invoice.md new file mode 100644 index 00000000..825e5c74 --- /dev/null +++ b/docs/models/order-to-cash/flows/order-to-cash/Issue-Invoice.md @@ -0,0 +1,21 @@ +--- +process: Issue invoice +number: 2 +description: "Bills open order lines." +inputs: + - from: db:order_line + label: unbilled lines + data: [party_id, order_no, line_no, qty, unit_price] +outputs: + - to: db:invoice + label: new invoice + data: [party_id, invoice_no] + - to: db:invoice_line + label: billed lines + data: [party_id, invoice_no, line_no, order_no, order_line_no, amount] + - to: ext:Customer + label: invoice + data: invoice number, lines, total +--- + +Bills every open [[order_line]] for one party onto a new [[invoice]]. diff --git a/docs/models/order-to-cash/flows/order-to-cash/Place-Order.md b/docs/models/order-to-cash/flows/order-to-cash/Place-Order.md new file mode 100644 index 00000000..4a9b1002 --- /dev/null +++ b/docs/models/order-to-cash/flows/order-to-cash/Place-Order.md @@ -0,0 +1,21 @@ +--- +process: Place order +number: 1 +description: "Records an order and its lines, assigning the next order_no for the party." +inputs: + - from: ext:Customer + label: order request + data: party, products, quantities + - from: db:product + label: list price + data: [product_id, list_price] +outputs: + - to: db:sales_order + label: new order + data: [party_id, order_no, status] + - to: db:order_line + label: order lines + data: [party_id, order_no, line_no, product_id, qty, unit_price] +--- + +Runs as the `place_order()` procedure. It assigns `order_no` as one more than the party's highest, inside the same transaction that writes the lines. diff --git a/docs/models/order-to-cash/groups/billing.md b/docs/models/order-to-cash/groups/billing.md new file mode 100644 index 00000000..891dc49c --- /dev/null +++ b/docs/models/order-to-cash/groups/billing.md @@ -0,0 +1,6 @@ +--- +label: Billing +color: "#6E8F4E" +--- + +What is owed and what was paid: invoices, payments, and how payments settle invoice lines. diff --git a/docs/models/order-to-cash/groups/parties.md b/docs/models/order-to-cash/groups/parties.md new file mode 100644 index 00000000..e46426be --- /dev/null +++ b/docs/models/order-to-cash/groups/parties.md @@ -0,0 +1,6 @@ +--- +label: Parties +color: "#916336" +--- + +Who buys. Every customer is a party, and every party is exactly one of a person or a business. diff --git a/docs/models/order-to-cash/groups/sales.md b/docs/models/order-to-cash/groups/sales.md new file mode 100644 index 00000000..c206e691 --- /dev/null +++ b/docs/models/order-to-cash/groups/sales.md @@ -0,0 +1,6 @@ +--- +label: Sales +color: "#454A59" +--- + +What was ordered: the catalog, orders, and their lines. diff --git a/docs/models/order-to-cash/ignatius.yml b/docs/models/order-to-cash/ignatius.yml new file mode 100644 index 00000000..50792313 --- /dev/null +++ b/docs/models/order-to-cash/ignatius.yml @@ -0,0 +1,7 @@ +name: Order to cash +description: The running example used on every page of noorm.dev. +branding: + title: Order to cash + subtitle: the noorm.dev running example + copyright: + holder: noorm diff --git a/docs/models/order-to-cash/stores/gateway-log.md b/docs/models/order-to-cash/stores/gateway-log.md new file mode 100644 index 00000000..fb444719 --- /dev/null +++ b/docs/models/order-to-cash/stores/gateway-log.md @@ -0,0 +1,6 @@ +--- +kind: file +title: Payment gateway log +--- + +Raw responses from the payment gateway, appended by Collect payment. Read only for reconciliation. diff --git a/docs/models/polymorphic/data/content/comments.md b/docs/models/polymorphic/data/content/comments.md deleted file mode 100644 index a595dee3..00000000 --- a/docs/models/polymorphic/data/content/comments.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -entity: comments -group: content -pk: - - comment_id -columns: - comment_id: - type: integer - commentable_type: - type: text - desc: "Which kind of row commentable_id means. 'POST' | 'PHOTO' | 'COMMENT' | 'USER' | 'GROUP'. The database cannot check it." - commentable_id: - type: integer - desc: "References nothing. No foreign key is possible, because the target table changes per row." - created_at: - type: datetime -examples: - - comment_id: 1 - commentable_type: POST - commentable_id: 1 - created_at: "2026-04-02T11:00:00Z" ---- - -# comments - -The table ORMs reach for polymorphism to model. Five possible targets, none of them declared. - -`commentable_id` is an ordinary integer. Point it at a deleted row, or at the right id in the wrong table, and nothing objects. Every rule about what it may reference lives in application code. diff --git a/docs/models/polymorphic/data/content/photos.md b/docs/models/polymorphic/data/content/photos.md deleted file mode 100644 index 38f4139e..00000000 --- a/docs/models/polymorphic/data/content/photos.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -entity: photos -group: content -pk: - - photo_id -columns: - photo_id: - type: integer - subject_type: - type: text - desc: "Which kind of row subject_id means. 'USER' | 'GROUP' | 'PROFILE' | 'POST'. The database cannot check it." - subject_id: - type: integer - desc: "References nothing. No foreign key is possible, because the target table changes per row." - created_at: - type: datetime -examples: - - photo_id: 1 - subject_type: USER - subject_id: 1 - created_at: "2026-04-02T11:00:00Z" ---- - -# photos - -A photo, hanging off any of four things. Which one is a string. - -`subject_id` is an ordinary integer. Point it at a deleted row, or at the right id in the wrong table, and nothing objects. Every rule about what it may reference lives in application code. diff --git a/docs/models/polymorphic/data/content/posts.md b/docs/models/polymorphic/data/content/posts.md deleted file mode 100644 index b3b9cacd..00000000 --- a/docs/models/polymorphic/data/content/posts.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -entity: posts -group: content -pk: - - post_id -columns: - post_id: - type: integer - owner_type: - type: text - desc: "Which kind of row owner_id means. 'USER' | 'GROUP'. The database cannot check it." - owner_id: - type: integer - desc: "References nothing. No foreign key is possible, because the target table changes per row." - created_at: - type: datetime -examples: - - post_id: 1 - owner_type: USER - owner_id: 1 - created_at: "2026-04-02T11:00:00Z" ---- - -# posts - -A post, owned by a user or a group. Which one is a string. - -`owner_id` is an ordinary integer. Point it at a deleted row, or at the right id in the wrong table, and nothing objects. Every rule about what it may reference lives in application code. diff --git a/docs/models/polymorphic/data/content/tags.md b/docs/models/polymorphic/data/content/tags.md deleted file mode 100644 index fc38a4b6..00000000 --- a/docs/models/polymorphic/data/content/tags.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -entity: tags -group: content -pk: - - tag_id -columns: - tag_id: - type: integer - target_type: - type: text - desc: "Which kind of row target_id means. 'POST' | 'PHOTO' | 'COMMENT'. The database cannot check it." - target_id: - type: integer - desc: "References nothing. No foreign key is possible, because the target table changes per row." - created_at: - type: datetime -examples: - - tag_id: 1 - target_type: POST - target_id: 1 - created_at: "2026-04-02T11:00:00Z" ---- - -# tags - -A tag application. Same shape, same silence. - -`target_id` is an ordinary integer. Point it at a deleted row, or at the right id in the wrong table, and nothing objects. Every rule about what it may reference lives in application code. diff --git a/docs/models/polymorphic/data/identity/groups.md b/docs/models/polymorphic/data/identity/groups.md deleted file mode 100644 index 42dcf4dc..00000000 --- a/docs/models/polymorphic/data/identity/groups.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -entity: groups -group: identity -pk: - - group_id -columns: - group_id: - type: integer - name: - type: text -examples: - - group_id: 7 - name: Relational Modelling ---- - -# groups - -Also unchanged, and also never referenced by anything below. diff --git a/docs/models/polymorphic/data/identity/profiles.md b/docs/models/polymorphic/data/identity/profiles.md deleted file mode 100644 index ea3718f9..00000000 --- a/docs/models/polymorphic/data/identity/profiles.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -entity: profiles -group: identity -pk: - - user_id -columns: - user_id: - type: integer - bio: - type: text -examples: - - user_id: 1 - bio: Keys should migrate, not point. -relationships: - - target: users - on: - user_id: user_id - predicate: { fwd: is described by, rev: describes } ---- - -# profiles - -The only real foreign key left in this design, and only because a profile happens to belong to exactly one kind of thing. - -Every other association below wanted to point at more than one kind, so it gave up pointing at all. diff --git a/docs/models/polymorphic/data/identity/users.md b/docs/models/polymorphic/data/identity/users.md deleted file mode 100644 index cb016291..00000000 --- a/docs/models/polymorphic/data/identity/users.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -entity: users -group: identity -pk: - - user_id -columns: - user_id: - type: integer - handle: - type: text -examples: - - user_id: 1 - handle: ada ---- - -# users - -Unchanged from the relational version. The identity tables are never the problem. diff --git a/docs/models/polymorphic/groups/content.md b/docs/models/polymorphic/groups/content.md deleted file mode 100644 index c0430e0e..00000000 --- a/docs/models/polymorphic/groups/content.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -label: Content -color: "#C2873C" ---- - -One table per concept. The variation lives in a string column. diff --git a/docs/models/polymorphic/groups/identity.md b/docs/models/polymorphic/groups/identity.md deleted file mode 100644 index dbe4203b..00000000 --- a/docs/models/polymorphic/groups/identity.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -label: Identity -color: "#4B7398" ---- - -The things that own content, when anything can be said to own it. diff --git a/docs/models/polymorphic/ignatius.yml b/docs/models/polymorphic/ignatius.yml deleted file mode 100644 index 0b212ca3..00000000 --- a/docs/models/polymorphic/ignatius.yml +++ /dev/null @@ -1,8 +0,0 @@ -name: Polymorphic Associations — The Same Domain -description: The ORM shape of the model in "The case for proper relational design" - -branding: - title: Polymorphic associations - subtitle: the same domain, fewer tables - copyright: - holder: noorm diff --git a/docs/models/social/data/content/comment_comments.md b/docs/models/social/data/content/comment_comments.md deleted file mode 100644 index d53dc8b7..00000000 --- a/docs/models/social/data/content/comment_comments.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: comment_comments -group: content -pk: - - comment_id -columns: - comment_id: - type: integer - desc: "The basetype's key, whole." - parent_comment_id: - type: integer - desc: "The comment being replied to. A real foreign key, even though it points back at the same basetype." -examples: - - comment_id: 4 - parent_comment_id: 3 -relationships: - - target: comments - on: - comment_id: comment_id - predicate: { fwd: is realized as, rev: is a } - - target: comments - on: - parent_comment_id: comment_id - predicate: { fwd: is replied to by, rev: replies to } ---- - -# comment_comments - -A reply. It is a subtype of [[comments]] and also references [[comments]] again as its target, which polymorphism models with the same untyped integer it uses for everything else. diff --git a/docs/models/social/data/content/comments.md b/docs/models/social/data/content/comments.md deleted file mode 100644 index 498802d7..00000000 --- a/docs/models/social/data/content/comments.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -entity: comments -group: content -pk: - - comment_id -columns: - comment_id: - type: integer - target_type: - type: text - desc: "Discriminator, drawn from the comment_target_types table rather than a CHECK list. Joins a unique key so it can travel into each subtype's foreign key." - created_at: - type: datetime -examples: - - comment_id: 1 - target_type: USER - created_at: "2026-04-02T11:00:00Z" -subtypes: - - exclusive: true - desc: Every comment targets exactly one kind of thing - members: - user_comments: - target_type: USER - group_comments: - target_type: GROUP - post_comments: - target_type: POST - comment_comments: - target_type: COMMENT ---- - -# comments - -A comment. This is the entity ORMs reach for polymorphism to model, and the one that suffers most for it. diff --git a/docs/models/social/data/content/group_comments.md b/docs/models/social/data/content/group_comments.md deleted file mode 100644 index 0ccc49a0..00000000 --- a/docs/models/social/data/content/group_comments.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: group_comments -group: content -pk: - - comment_id -columns: - comment_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - group_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - comment_id: 1 - group_id: 1 -relationships: - - target: comments - on: - comment_id: comment_id - predicate: { fwd: is realized as, rev: is a } - - target: groups - on: - group_id: group_id - predicate: { fwd: is left on, rev: is is by } ---- - -# group_comments - -A [[comments]] belonging to a [[groups]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/group_photos.md b/docs/models/social/data/content/group_photos.md deleted file mode 100644 index 6a2ebe0f..00000000 --- a/docs/models/social/data/content/group_photos.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: group_photos -group: content -pk: - - photo_id -columns: - photo_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - group_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - photo_id: 1 - group_id: 1 -relationships: - - target: photos - on: - photo_id: photo_id - predicate: { fwd: is realized as, rev: is a } - - target: groups - on: - group_id: group_id - predicate: { fwd: hosts, rev: is hosts by } ---- - -# group_photos - -A [[photos]] belonging to a [[groups]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/group_posts.md b/docs/models/social/data/content/group_posts.md deleted file mode 100644 index 60ba67eb..00000000 --- a/docs/models/social/data/content/group_posts.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: group_posts -group: content -pk: - - post_id -columns: - post_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - group_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - post_id: 1 - group_id: 1 -relationships: - - target: posts - on: - post_id: post_id - predicate: { fwd: is realized as, rev: is a } - - target: groups - on: - group_id: group_id - predicate: { fwd: publishes, rev: is publishes by } ---- - -# group_posts - -A [[posts]] belonging to a [[groups]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/photos.md b/docs/models/social/data/content/photos.md deleted file mode 100644 index 5a895acf..00000000 --- a/docs/models/social/data/content/photos.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -entity: photos -group: content -pk: - - photo_id -columns: - photo_id: - type: integer - subject_type: - type: text - desc: "Discriminator, drawn from the photo_subject_types table rather than a CHECK list. Joins a unique key so it can travel into each subtype's foreign key." - created_at: - type: datetime -examples: - - photo_id: 1 - subject_type: USER - created_at: "2026-04-02T11:00:00Z" -subtypes: - - exclusive: true - desc: Every photo hangs off exactly one kind of subject - members: - user_photos: - subject_type: USER - group_photos: - subject_type: GROUP - profile_photos: - subject_type: PROFILE - user_post_photos: - subject_type: USER_POST ---- - -# photos - -A photo. Four things can hold one, and each gets a real foreign key rather than a shared integer column. diff --git a/docs/models/social/data/content/post_comments.md b/docs/models/social/data/content/post_comments.md deleted file mode 100644 index 8e1aa938..00000000 --- a/docs/models/social/data/content/post_comments.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: post_comments -group: content -pk: - - comment_id -columns: - comment_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - post_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - comment_id: 1 - post_id: 1 -relationships: - - target: comments - on: - comment_id: comment_id - predicate: { fwd: is realized as, rev: is a } - - target: posts - on: - post_id: post_id - predicate: { fwd: replies to, rev: is replies by } ---- - -# post_comments - -A [[comments]] belonging to a [[posts]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/posts.md b/docs/models/social/data/content/posts.md deleted file mode 100644 index fb9ff434..00000000 --- a/docs/models/social/data/content/posts.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -entity: posts -group: content -pk: - - post_id -columns: - post_id: - type: integer - owner_type: - type: text - desc: "Discriminator, drawn from the post_owner_types table rather than a CHECK list. Joins a unique key so it can travel into each subtype's foreign key." - created_at: - type: datetime -examples: - - post_id: 1 - owner_type: USER - created_at: "2026-04-02T11:00:00Z" -subtypes: - - exclusive: true - desc: Every post is owned by exactly one of a user or a group - members: - user_posts: - owner_type: USER - group_posts: - owner_type: GROUP ---- - -# posts - -A post. Everything common to one lives here; who owns it is structural, not a column you branch on. diff --git a/docs/models/social/data/content/profile_photos.md b/docs/models/social/data/content/profile_photos.md deleted file mode 100644 index 841e0dec..00000000 --- a/docs/models/social/data/content/profile_photos.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: profile_photos -group: content -pk: - - photo_id -columns: - photo_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - user_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - photo_id: 1 - user_id: 1 -relationships: - - target: photos - on: - photo_id: photo_id - predicate: { fwd: is realized as, rev: is a } - - target: profiles - on: - user_id: user_id - predicate: { fwd: illustrates, rev: is illustrates by } ---- - -# profile_photos - -A [[photos]] belonging to a [[profiles]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/user_comments.md b/docs/models/social/data/content/user_comments.md deleted file mode 100644 index fe1de46c..00000000 --- a/docs/models/social/data/content/user_comments.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: user_comments -group: content -pk: - - comment_id -columns: - comment_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - user_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - comment_id: 1 - user_id: 1 -relationships: - - target: comments - on: - comment_id: comment_id - predicate: { fwd: is realized as, rev: is a } - - target: users - on: - user_id: user_id - predicate: { fwd: is left on, rev: is is by } ---- - -# user_comments - -A [[comments]] belonging to a [[users]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/user_photos.md b/docs/models/social/data/content/user_photos.md deleted file mode 100644 index b06b45e4..00000000 --- a/docs/models/social/data/content/user_photos.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: user_photos -group: content -pk: - - photo_id -columns: - photo_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - user_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - photo_id: 1 - user_id: 1 -relationships: - - target: photos - on: - photo_id: photo_id - predicate: { fwd: is realized as, rev: is a } - - target: users - on: - user_id: user_id - predicate: { fwd: uploads, rev: is uploads by } ---- - -# user_photos - -A [[photos]] belonging to a [[users]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/user_post_photos.md b/docs/models/social/data/content/user_post_photos.md deleted file mode 100644 index 7e17a86c..00000000 --- a/docs/models/social/data/content/user_post_photos.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: user_post_photos -group: content -pk: - - photo_id -columns: - photo_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - post_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - photo_id: 1 - post_id: 1 -relationships: - - target: photos - on: - photo_id: photo_id - predicate: { fwd: is realized as, rev: is a } - - target: user_posts - on: - post_id: post_id - predicate: { fwd: is attached to, rev: is is by } ---- - -# user_post_photos - -A [[photos]] belonging to a [[user_posts]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/content/user_posts.md b/docs/models/social/data/content/user_posts.md deleted file mode 100644 index 8399cad5..00000000 --- a/docs/models/social/data/content/user_posts.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: user_posts -group: content -pk: - - post_id -columns: - post_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - user_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - post_id: 1 - user_id: 1 -relationships: - - target: posts - on: - post_id: post_id - predicate: { fwd: is realized as, rev: is a } - - target: users - on: - user_id: user_id - predicate: { fwd: authors, rev: is authors by } ---- - -# user_posts - -A [[posts]] belonging to a [[users]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/identity/groups.md b/docs/models/social/data/identity/groups.md deleted file mode 100644 index 47898ab4..00000000 --- a/docs/models/social/data/identity/groups.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -entity: groups -group: identity -pk: - - group_id -columns: - group_id: - type: integer - name: - type: text -examples: - - group_id: 7 - name: Relational Modelling ---- - -# groups - -Independent, like [[users]]. The other thing that can own content. diff --git a/docs/models/social/data/identity/profiles.md b/docs/models/social/data/identity/profiles.md deleted file mode 100644 index a3587b48..00000000 --- a/docs/models/social/data/identity/profiles.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -entity: profiles -group: identity -pk: - - user_id -columns: - user_id: - type: integer - desc: "The user's key, whole. A profile has no identity of its own." - bio: - type: text -examples: - - user_id: 1 - bio: Keys should migrate, not point. -relationships: - - target: users - on: - user_id: user_id - predicate: { fwd: is described by, rev: describes } ---- - -# profiles - -Dependent. Its key *is* the user's key, so a profile cannot exist without one and cannot be moved to another. diff --git a/docs/models/social/data/identity/users.md b/docs/models/social/data/identity/users.md deleted file mode 100644 index 7d6eefbb..00000000 --- a/docs/models/social/data/identity/users.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -entity: users -group: identity -pk: - - user_id -columns: - user_id: - type: integer - handle: - type: text -examples: - - user_id: 1 - handle: ada ---- - -# users - -Independent. Nothing above it supplies identity, so a surrogate key is the right call here and nowhere else. diff --git a/docs/models/social/data/tagging/comment_tags.md b/docs/models/social/data/tagging/comment_tags.md deleted file mode 100644 index 3ad1fe0b..00000000 --- a/docs/models/social/data/tagging/comment_tags.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: comment_tags -group: tagging -pk: - - tag_id -columns: - tag_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - comment_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - tag_id: 1 - comment_id: 1 -relationships: - - target: tags - on: - tag_id: tag_id - predicate: { fwd: is realized as, rev: is a } - - target: comments - on: - comment_id: comment_id - predicate: { fwd: labels, rev: is labels by } ---- - -# comment_tags - -A [[tags]] belonging to a [[comments]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/tagging/photo_tags.md b/docs/models/social/data/tagging/photo_tags.md deleted file mode 100644 index 6a989bea..00000000 --- a/docs/models/social/data/tagging/photo_tags.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: photo_tags -group: tagging -pk: - - tag_id -columns: - tag_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - photo_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - tag_id: 1 - photo_id: 1 -relationships: - - target: tags - on: - tag_id: tag_id - predicate: { fwd: is realized as, rev: is a } - - target: photos - on: - photo_id: photo_id - predicate: { fwd: labels, rev: is labels by } ---- - -# photo_tags - -A [[tags]] belonging to a [[photos]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/tagging/post_tags.md b/docs/models/social/data/tagging/post_tags.md deleted file mode 100644 index 877a1103..00000000 --- a/docs/models/social/data/tagging/post_tags.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: post_tags -group: tagging -pk: - - tag_id -columns: - tag_id: - type: integer - desc: "The basetype's key, whole. That is what makes this a subtype rather than a child." - post_id: - type: integer - desc: "A real foreign key to a real table, which polymorphism cannot give you." -examples: - - tag_id: 1 - post_id: 1 -relationships: - - target: tags - on: - tag_id: tag_id - predicate: { fwd: is realized as, rev: is a } - - target: posts - on: - post_id: post_id - predicate: { fwd: labels, rev: is labels by } ---- - -# post_tags - -A [[tags]] belonging to a [[posts]] row. Its key is the basetype's key. diff --git a/docs/models/social/data/tagging/tags.md b/docs/models/social/data/tagging/tags.md deleted file mode 100644 index 986f4f23..00000000 --- a/docs/models/social/data/tagging/tags.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -entity: tags -group: tagging -pk: - - tag_id -columns: - tag_id: - type: integer - target_type: - type: text - desc: "Discriminator, drawn from the tag_target_types table rather than a CHECK list. Joins a unique key so it can travel into each subtype's foreign key." - created_at: - type: datetime -examples: - - tag_id: 1 - target_type: POST - created_at: "2026-04-02T11:00:00Z" -subtypes: - - exclusive: true - desc: Every tag application points at exactly one kind of thing - members: - post_tags: - target_type: POST - photo_tags: - target_type: PHOTO - comment_tags: - target_type: COMMENT ---- - -# tags - -A tag application. The label itself is common; what was tagged is the subtype. diff --git a/docs/models/social/groups/content.md b/docs/models/social/groups/content.md deleted file mode 100644 index 956b96ac..00000000 --- a/docs/models/social/groups/content.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -label: Content -color: "#E05742" ---- - -Posts, photos and comments, each a basetype with its own subtype cluster. diff --git a/docs/models/social/groups/identity.md b/docs/models/social/groups/identity.md deleted file mode 100644 index 382e2f75..00000000 --- a/docs/models/social/groups/identity.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -label: Identity -color: "#4B7398" ---- - -The things that can own content. diff --git a/docs/models/social/groups/tagging.md b/docs/models/social/groups/tagging.md deleted file mode 100644 index 7cbd1679..00000000 --- a/docs/models/social/groups/tagging.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -label: Tagging -color: "#6E8F4E" ---- - -Tag applications, one subtype per taggable thing. diff --git a/docs/models/social/ignatius.yml b/docs/models/social/ignatius.yml deleted file mode 100644 index 94c673a5..00000000 --- a/docs/models/social/ignatius.yml +++ /dev/null @@ -1,12 +0,0 @@ -name: Basetype-Subtypes — A Social Graph -description: The exhaustive example from "The case for proper relational design" on noorm.dev - -branding: - title: Basetype-subtypes - subtitle: what polymorphism hides - copyright: - holder: noorm - -theme: - spacing: - nodeSep: 26 diff --git a/docs/models/todo-list/data/core/todo_items.md b/docs/models/todo-list/data/core/todo_items.md deleted file mode 100644 index c6f52646..00000000 --- a/docs/models/todo-list/data/core/todo_items.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -entity: todo_items -group: core -pk: - - user_id - - created_at - - item_index -columns: - user_id: - type: integer - desc: "Inherited from users, two levels up. Still part of this row's identity." - created_at: - type: datetime - desc: "Inherited from todos. Names which list this item belongs to." - item_index: - type: integer - desc: "Local discriminator. Position within its own list." - body: - type: text - desc: "The task." - done: - type: boolean - default: "false" - desc: "Whether it has been completed." -examples: - - user_id: 1 - created_at: "2026-03-01T10:00:00Z" - item_index: 1 - body: Oat milk - done: true - - user_id: 1 - created_at: "2026-03-01T10:00:00Z" - item_index: 2 - body: Coffee - done: false - - user_id: 2 - created_at: "2026-03-02T08:05:00Z" - item_index: 1 - body: Tomatoes - done: false -relationships: - - target: todos - on: - user_id: user_id - created_at: created_at - predicate: { fwd: contains, rev: is part of } ---- - -# todo_items - -An item on a [[todos]] list, and the entity that makes the argument. - -Its key is `user_id, created_at, item_index`: the whole ancestry, plus one local discriminator. Read the example rows and the lineage is visible without a join. `user_id` is right there on every row, two levels down from where it was introduced. - -That is the property the surrogate-key version gives up. With `todo_item_id` referencing `todo_id` referencing `user_id`, answering "whose item is this?" costs two joins, and nothing in the schema stops an item from being attached to a list that belongs to someone else. diff --git a/docs/models/todo-list/data/core/todos.md b/docs/models/todo-list/data/core/todos.md deleted file mode 100644 index cc94b083..00000000 --- a/docs/models/todo-list/data/core/todos.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -entity: todos -group: core -pk: - - user_id - - created_at -columns: - user_id: - type: integer - desc: "Inherited from users. Part of this entity's own identity, not a pointer to it." - created_at: - type: datetime - desc: "Local discriminator. Distinguishes one list from another within the same user." - title: - type: text - desc: "What the list is called." -examples: - - user_id: 1 - created_at: "2026-03-01T10:00:00Z" - title: Groceries - - user_id: 1 - created_at: "2026-03-04T18:22:00Z" - title: Sprint 14 - - user_id: 2 - created_at: "2026-03-02T08:05:00Z" - title: Groceries -relationships: - - target: users - on: - user_id: user_id - predicate: { fwd: keeps, rev: is kept by } ---- - -# todos - -A list belonging to a [[users]] row. There is no `todo_id`. - -Identity is `user_id` plus the moment the list was created. That is what makes the relationship **identifying**: a todo cannot be identified without knowing whose it is, so the parent's key lives inside the child's. - -Two users can both have a list created at the same instant, and two lists can share a title. Neither collides, because the key carries the owner. diff --git a/docs/models/todo-list/data/core/users.md b/docs/models/todo-list/data/core/users.md deleted file mode 100644 index f4d60401..00000000 --- a/docs/models/todo-list/data/core/users.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -entity: users -group: core -pk: - - user_id -columns: - user_id: - type: integer - desc: "Surrogate identity. The root of the key spine, so this one is allowed to be arbitrary." - email: - type: text - desc: "Login identity. Unique, but not the primary key: it changes, and keys should not." - created_at: - type: datetime - desc: "When the account was opened." -examples: - - user_id: 1 - email: ada@example.com - created_at: "2026-01-04T09:12:00Z" - - user_id: 2 - email: grace@example.com - created_at: "2026-02-11T16:40:00Z" ---- - -# users - -The root of the model, and the only entity here with an identity of its own. - -A surrogate key is the right call at the root: nothing above it can supply identity, and the natural candidate (`email`) is something people change. Everything below inherits `user_id` rather than minting its own. diff --git a/docs/models/todo-list/groups/core.md b/docs/models/todo-list/groups/core.md deleted file mode 100644 index b037c63e..00000000 --- a/docs/models/todo-list/groups/core.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -label: Core -color: "#E05742" ---- - -The whole model. Three entities, one identity spine. diff --git a/docs/models/todo-list/ignatius.yml b/docs/models/todo-list/ignatius.yml deleted file mode 100644 index ccdc2c8c..00000000 --- a/docs/models/todo-list/ignatius.yml +++ /dev/null @@ -1,8 +0,0 @@ -name: Inherited Keys — A Todo List -description: The worked example from "The case for proper relational design" on noorm.dev - -branding: - title: Inherited keys - subtitle: a worked example - copyright: - holder: noorm diff --git a/docs/parts/ignatius-cmds.md b/docs/parts/ignatius-cmds.md new file mode 100644 index 00000000..72dbac30 --- /dev/null +++ b/docs/parts/ignatius-cmds.md @@ -0,0 +1,21 @@ +<div class="nh-cmds"> +<div> + +```bash +# check the model in CI +$ ignatius validate model/ +✓ /home/you/shop/model: valid — 11 entities, no findings. +``` + +</div> +<div> + +```text +# model with an agent, in Claude Code +> /ignatius-modeling discover + interviews you about how the business runs, + then writes the entities and flows +``` + +</div> +</div> diff --git a/docs/parts/ignatius-dfd.md b/docs/parts/ignatius-dfd.md new file mode 100644 index 00000000..1d83effc --- /dev/null +++ b/docs/parts/ignatius-dfd.md @@ -0,0 +1,19 @@ +<div class="nh-dfd-compare"> +<div class="nh-panel"> +<div class="nh-panel-head"><span>logic flowchart</span><span>what teams draw</span></div> +<pre class="nh-panel-body nh-ascii" role="img" aria-label="Logic flowchart: charge card, then approved? yes leads to mark paid, no leads to show error."> ┌─────────────┐ + │ charge card │ + └──────┬──────┘ + │ + approved? + ┌───┴───┐ + yes│ │no + ┌──────▼───┐ ┌─▼──────────┐ + │ mark paid│ │ show error │ + └──────────┘ └────────────┘</pre> +<p class="nh-panel-foot">What the code decides. No table, log file, or person appears in it.</p> +</div> +<ModelViewer class="nh-dfd-frame" model="order-to-cash" hash="view=flow&dfd=order-to-cash&flowview=connected" :tabs="false" label="data flow diagram · order to cash" title="Order-to-cash data flow diagram rendered by ignatius"> +<p class="nh-panel-foot">Every place an order's data enters, rests, and leaves, including the gateway log that no table holds.</p> +</ModelViewer> +</div> diff --git a/docs/parts/ignatius-problems.md b/docs/parts/ignatius-problems.md new file mode 100644 index 00000000..204f6ec6 --- /dev/null +++ b/docs/parts/ignatius-problems.md @@ -0,0 +1,22 @@ +<div class="nh-rows"> +<div class="nh-r"> +<p class="nh-pain">The ER diagram in your wiki stopped matching the database months ago.</p> +<p class="nh-fix">The diagram renders from markdown in the repo, so it changes in the same pull request as the schema.</p> +<code class="nh-proof">ignatius serve model/ -o</code> +</div> +<div class="nh-r"> +<p class="nh-pain">Nobody remembers why a table exists.</p> +<p class="nh-fix">Each entity carries its purpose and example rows beside its keys, searchable in the dictionary.</p> +<code class="nh-proof">examples:</code> +</div> +<div class="nh-r"> +<p class="nh-pain">The person who signs off on the design can't read DDL.</p> +<p class="nh-fix">Export the whole model to one HTML file. They open it from their desktop, with nothing installed.</p> +<code class="nh-proof">ignatius export -o model.html</code> +</div> +<div class="nh-r"> +<p class="nh-pain">You inherited a database nobody documented.</p> +<p class="nh-fix">An agent reads it through noorm's read-only MCP server, and the ignatius-modeling skill writes the model.</p> +<code class="nh-proof">/ignatius-modeling</code> +</div> +</div> diff --git a/docs/parts/ignatius-viewer.md b/docs/parts/ignatius-viewer.md new file mode 100644 index 00000000..58852124 --- /dev/null +++ b/docs/parts/ignatius-viewer.md @@ -0,0 +1,32 @@ +<div class="nh-viewer"> +<div class="nh-file"> +<div class="nh-panel-head"><span>model/data/sales/sales_order.md</span><span>you write this</span></div> + +```yaml +--- +entity: sales_order +group: sales +pk: [party_id, order_no] +columns: + party_id: { type: integer } + order_no: { type: integer } + placed_at: { type: datetime, default: now } + status: { type: text, default: open } +examples: + - { party_id: 2, order_no: 1, status: invoiced } + - { party_id: 2, order_no: 2, status: open } + - { party_id: 1, order_no: 1, status: paid } +relationships: + - target: party + on: { party_id: party_id } + predicate: + fwd: places + rev: is placed by +--- +``` + +</div> +<ModelViewer model="order-to-cash" title="Order-to-cash model rendered by ignatius" caption="ignatius draws this" /> +</div> + +<p class="nh-note">Cardinality and identifying edges come from the keys, and subtype clusters from the <code>subtypes:</code> block. Nothing is drawn by hand, and saving a file reloads the open browser.</p> diff --git a/docs/public/icons/bolt.svg b/docs/public/icons/bolt.svg deleted file mode 100644 index c0127ccb..00000000 --- a/docs/public/icons/bolt.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 640"><!--!Font Awesome Free v7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2026 Fonticons, Inc.--><path fill="#06B6D4" d="M434.8 54.1C446.7 62.7 451.1 78.3 445.7 91.9L367.3 288L512 288C525.5 288 537.5 296.4 542.1 309.1C546.7 321.8 542.8 336 532.5 344.6L244.5 584.6C233.2 594 217.1 594.5 205.2 585.9C193.3 577.3 188.9 561.7 194.3 548.1L272.7 352L128 352C114.5 352 102.5 343.6 97.9 330.9C93.3 318.2 97.2 304 107.5 295.4L395.5 55.4C406.8 46 422.9 45.5 434.8 54.1z"/></svg> diff --git a/docs/public/icons/cubes.svg b/docs/public/icons/cubes.svg deleted file mode 100644 index 15cf9d93..00000000 --- a/docs/public/icons/cubes.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="none" stroke="#E05742" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M2.97 12.92A2 2 0 0 0 2 14.63v3.24a2 2 0 0 0 1.03 1.75l3 1.68a2 2 0 0 0 1.94 0l3.06-1.68a2 2 0 0 0 1.03-1.75v-3.24a2 2 0 0 0-.97-1.71l-3-1.68a2 2 0 0 0-1.94 0z"></path><path d="M7 16.5l-4.74-2.85"></path><path d="M7 16.5v5.17"></path><path d="M7 16.5l4.74-2.85"></path><path d="M14.97 12.92A2 2 0 0 0 14 14.63v3.24a2 2 0 0 0 1.03 1.75l3 1.68a2 2 0 0 0 1.94 0l3.06-1.68a2 2 0 0 0 1.03-1.75v-3.24a2 2 0 0 0-.97-1.71l-3-1.68a2 2 0 0 0-1.94 0z"></path><path d="M19 16.5l-4.74-2.85"></path><path d="M19 16.5v5.17"></path><path d="M19 16.5l4.74-2.85"></path><path d="M8.97 4.92A2 2 0 0 0 8 6.63v3.24a2 2 0 0 0 1.03 1.75l3 1.68a2 2 0 0 0 1.94 0l3.06-1.68a2 2 0 0 0 1.03-1.75V6.63a2 2 0 0 0-.97-1.71l-3-1.68a2 2 0 0 0-1.94 0z"></path><path d="M13 8.5l-4.74-2.85"></path><path d="M13 8.5l4.74-2.85"></path></svg> \ No newline at end of file diff --git a/docs/public/icons/database.svg b/docs/public/icons/database.svg deleted file mode 100644 index b6070291..00000000 --- a/docs/public/icons/database.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="none" stroke="#E05742" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M4 6c0-1.66 3.58-3 8-3s8 1.34 8 3-3.58 3-8 3-8-1.34-8-3z"></path><path d="M4 6v6c0 1.66 3.58 3 8 3s8-1.34 8-3V6"></path><path d="M4 12v6c0 1.66 3.58 3 8 3s8-1.34 8-3v-6"></path></svg> \ No newline at end of file diff --git a/docs/public/icons/fast-forward.svg b/docs/public/icons/fast-forward.svg deleted file mode 100644 index 5d215574..00000000 --- a/docs/public/icons/fast-forward.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="none" stroke="#E05742" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polygon points="13 19 22 12 13 5 13 19"></polygon><polygon points="2 19 11 12 2 5 2 19"></polygon></svg> \ No newline at end of file diff --git a/docs/public/icons/flask.svg b/docs/public/icons/flask.svg deleted file mode 100644 index 37d16210..00000000 --- a/docs/public/icons/flask.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 640"><!--!Font Awesome Free v7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2026 Fonticons, Inc.--><path fill="#EC4899" d="M384 64L224 64C206.3 64 192 78.3 192 96C192 113.7 206.3 128 224 128L224 279.5L103.5 490.3C98.6 499 96 508.7 96 518.7C96 550.4 121.6 576 153.3 576L486.7 576C518.3 576 544 550.4 544 518.7C544 508.7 541.4 498.9 536.5 490.3L416 279.5L416 128C433.7 128 448 113.7 448 96C448 78.3 433.7 64 416 64L384 64zM288 279.5L288 128L352 128L352 279.5C352 290.6 354.9 301.6 360.4 311.3L402 384L238 384L279.6 311.3C285.1 301.6 288 290.7 288 279.5z"/></svg> diff --git a/docs/public/icons/git-branch.svg b/docs/public/icons/git-branch.svg deleted file mode 100644 index 2a8e4ef1..00000000 --- a/docs/public/icons/git-branch.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="none" stroke="#E05742" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><line x1="6" y1="3" x2="6" y2="15"></line><circle cx="18" cy="6" r="3"></circle><circle cx="6" cy="18" r="3"></circle><path d="M18 9a9 9 0 0 1-9 9"></path></svg> \ No newline at end of file diff --git a/docs/public/icons/lock.svg b/docs/public/icons/lock.svg deleted file mode 100644 index 954d58ac..00000000 --- a/docs/public/icons/lock.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="none" stroke="#E05742" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><rect x="3" y="11" width="18" height="11" rx="2" ry="2"></rect><path d="M7 11V7a5 5 0 0 1 10 0v4"></path></svg> \ No newline at end of file diff --git a/docs/public/icons/terminal.svg b/docs/public/icons/terminal.svg deleted file mode 100644 index 26a5a5b1..00000000 --- a/docs/public/icons/terminal.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 640"><!--!Font Awesome Free v7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2026 Fonticons, Inc.--><path fill="#F59E0B" d="M73.4 182.6C60.9 170.1 60.9 149.8 73.4 137.3C85.9 124.8 106.2 124.8 118.7 137.3L278.7 297.3C291.2 309.8 291.2 330.1 278.7 342.6L118.7 502.6C106.2 515.1 85.9 515.1 73.4 502.6C60.9 490.1 60.9 469.8 73.4 457.3L210.7 320L73.4 182.6zM288 448L544 448C561.7 448 576 462.3 576 480C576 497.7 561.7 512 544 512L288 512C270.3 512 256 497.7 256 480C256 462.3 270.3 448 288 448z"/></svg> diff --git a/docs/public/icons/toolbox.svg b/docs/public/icons/toolbox.svg deleted file mode 100644 index de2c8e44..00000000 --- a/docs/public/icons/toolbox.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24" fill="none" stroke="#E05742" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M14.7 6.3a1 1 0 0 0 0 1.4l1.6 1.6a1 1 0 0 0 1.4 0l3.77-3.77a6 6 0 0 1-7.94 7.94l-6.91 6.91a2.12 2.12 0 0 1-3-3l6.91-6.91a6 6 0 0 1 7.94-7.94l-3.76 3.76z"></path></svg> \ No newline at end of file diff --git a/docs/public/icons/users.svg b/docs/public/icons/users.svg deleted file mode 100644 index dd04e82f..00000000 --- a/docs/public/icons/users.svg +++ /dev/null @@ -1 +0,0 @@ -<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 640"><!--!Font Awesome Free v7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2026 Fonticons, Inc.--><path fill="#8B5CF6" d="M320 80C377.4 80 424 126.6 424 184C424 241.4 377.4 288 320 288C262.6 288 216 241.4 216 184C216 126.6 262.6 80 320 80zM96 152C135.8 152 168 184.2 168 224C168 263.8 135.8 296 96 296C56.2 296 24 263.8 24 224C24 184.2 56.2 152 96 152zM0 480C0 409.3 57.3 352 128 352C140.8 352 153.2 353.9 164.9 357.4C132 394.2 112 442.8 112 496L112 512C112 523.4 114.4 534.2 118.7 544L32 544C14.3 544 0 529.7 0 512L0 480zM521.3 544C525.6 534.2 528 523.4 528 512L528 496C528 442.8 508 394.2 475.1 357.4C486.8 353.9 499.2 352 512 352C582.7 352 640 409.3 640 480L640 512C640 529.7 625.7 544 608 544L521.3 544zM472 224C472 184.2 504.2 152 544 152C583.8 152 616 184.2 616 224C616 263.8 583.8 296 544 296C504.2 296 472 263.8 472 224zM160 496C160 407.6 231.6 336 320 336C408.4 336 480 407.6 480 496L480 512C480 529.7 465.7 544 448 544L192 544C174.3 544 160 529.7 160 512L160 496z"/></svg> diff --git a/docs/public/ignatius.sh b/docs/public/ignatius.sh new file mode 100644 index 00000000..d5264a41 --- /dev/null +++ b/docs/public/ignatius.sh @@ -0,0 +1,10 @@ +#!/bin/sh +# Short URL for the ignatius installer; runs the one copy kept in the ignatius repo. +# Variables set on this sh reach it: curl -fsSL https://noorm.dev/ignatius.sh | IGNATIUS_VERSION=v0.23.0 sh +set -eu + +installer=$(mktemp) +trap 'rm -f "$installer"' EXIT + +curl -fsSL https://raw.githubusercontent.com/noormdev/ignatius/main/install.sh -o "$installer" +sh "$installer" diff --git a/docs/public/image/favicon.svg b/docs/public/image/favicon.svg new file mode 100644 index 00000000..58318f9e --- /dev/null +++ b/docs/public/image/favicon.svg @@ -0,0 +1,9 @@ +<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 256" width="256" height="256"><title>noorm + + + + + + + + \ No newline at end of file diff --git a/docs/public/image/logo-dark.svg b/docs/public/image/logo-dark.svg new file mode 100644 index 00000000..759fc78c --- /dev/null +++ b/docs/public/image/logo-dark.svg @@ -0,0 +1,9 @@ + + noorm symbol, reversed + + + + + + + diff --git a/docs/public/image/logo.png b/docs/public/image/logo.png index 3d069b3c..d69bb090 100644 Binary files a/docs/public/image/logo.png and b/docs/public/image/logo.png differ diff --git a/docs/public/image/logo.svg b/docs/public/image/logo.svg index aa844b1f..d36d0fa8 100644 --- a/docs/public/image/logo.svg +++ b/docs/public/image/logo.svg @@ -1,6 +1,9 @@ - - - - - + + noorm symbol + + + + + + diff --git a/docs/public/image/og.png b/docs/public/image/og.png index 7715c434..1dfaa91c 100644 Binary files a/docs/public/image/og.png and b/docs/public/image/og.png differ diff --git a/docs/public/models/order-to-cash-dark.html b/docs/public/models/order-to-cash-dark.html new file mode 100644 index 00000000..d6736e03 --- /dev/null +++ b/docs/public/models/order-to-cash-dark.html @@ -0,0 +1,640 @@ + + + + + + Order to cash + + + +
+ + + + diff --git a/docs/public/models/order-to-cash.html b/docs/public/models/order-to-cash.html new file mode 100644 index 00000000..d325cb51 --- /dev/null +++ b/docs/public/models/order-to-cash.html @@ -0,0 +1,640 @@ + + + + + + Order to cash + + + +
+ + + + diff --git a/docs/public/models/polymorphic.html b/docs/public/models/polymorphic.html deleted file mode 100644 index 49ab18c0..00000000 --- a/docs/public/models/polymorphic.html +++ /dev/null @@ -1,629 +0,0 @@ - - - - - - Polymorphic Associations — The Same Domain - - - -
- - - - diff --git a/docs/public/models/social.html b/docs/public/models/social.html deleted file mode 100644 index 5c5d6442..00000000 --- a/docs/public/models/social.html +++ /dev/null @@ -1,629 +0,0 @@ - - - - - - Basetype-Subtypes — A Social Graph - - - -
- - - - diff --git a/docs/public/models/todo-list.html b/docs/public/models/todo-list.html deleted file mode 100644 index b799db7f..00000000 --- a/docs/public/models/todo-list.html +++ /dev/null @@ -1,629 +0,0 @@ - - - - - - Inherited Keys — A Todo List - - - -
- - - - diff --git a/docs/reference/sdk.md b/docs/reference/sdk.md index e7e012ba..a6db5498 100644 --- a/docs/reference/sdk.md +++ b/docs/reference/sdk.md @@ -1,1707 +1,1105 @@ -# SDK Reference +--- +title: SDK reference +description: createContext, the Kysely-backed Context, and the ten ctx.noorm namespaces, with signatures, access gates, result shapes, events, and errors. +--- +# SDK reference -## Overview -The noorm SDK provides programmatic access to noorm-managed databases. Use it for: - -- **Application code** - Query and mutate data from your apps -- **Test suites** - Reset and seed databases between tests -- **Scripts** - Data transforms, exports, and automation -- **CI/CD** - Headless database operations - - -## Installation +Query a noorm-managed database from TypeScript, and run builds, changes, locks, and the vault from the same context. ```bash -pnpm add @noormdev/sdk +npm install @noormdev/sdk kysely pg ``` - -## Quick Start +`kysely` and your dialect's driver are peer dependencies: `pg` for postgres, `mysql2` for mysql, `tedious` and `tarn` for mssql, `better-sqlite3` for sqlite. Node 22.13 or newer. ```typescript +// scripts/open-orders.ts import { createContext } from '@noormdev/sdk'; -const ctx = await createContext<{ users: { id: number; name: string } }>({ - config: 'dev', -}); -await ctx.connect(); - -// Top-level — SQL focused -const users = await ctx.kysely - .selectFrom('users') - .select(['id', 'name']) - .execute(); - -// Noorm operations — under namespace -await ctx.noorm.changes.ff(); - -await ctx.disconnect(); -``` - - -## createContext(options) - -Creates an SDK context for programmatic database access. - -```typescript -interface CreateContextOptions { - config?: string; // Config name (or use NOORM_CONFIG env var) - projectRoot?: string; // Defaults to process.cwd() - requireTest?: boolean; // Refuse if config.isTest !== true - channel?: Channel; // 'user' (default) or 'agent' — which access role applies - stage?: string; // Stage name for stage defaults - yes?: boolean; // Pre-confirm operations a policy 'confirm' cell would block +interface OrderToCash { + sales_order: { + party_id: number; + order_no: number; + placed_at: Date; + status: string; + }; } -const ctx = await createContext({ - config: 'test', - requireTest: true, -}); -``` - -**Options:** - -| Option | Type | Description | -| ---------------- | --------- | ---------------------------------------------------------------- | -| `config` | `string` | Config name to use. Falls back to `NOORM_CONFIG` env var. | -| `projectRoot` | `string` | Path to noorm project. Defaults to `process.cwd()`. | -| `requireTest` | `boolean` | Throws `RequireTestError` if config doesn't have `isTest: true`. | -| `channel` | `Channel` | Which caller channel this context represents for access-policy checks. Defaults to `'user'`. A destructive operation the config's role denies — or that resolves to "requires confirmation," which the SDK can't prompt for — throws `ProtectedConfigError`. | -| `stage` | `string` | Stage name for inheriting stage defaults. | -| `yes` | `boolean` | Pre-confirms operations that a policy "requires confirmation" cell would otherwise block, the programmatic equivalent of the CLI's `--yes`. Defaults to `false`. Only meaningful on the `user` channel: on `agent`, confirmation collapses to denial before `yes` is read, so an agent cannot use it to pass a gate meant for a human. | - -There is no `connection` option. Connection details come either from a stored config or, when `NOORM_CONNECTION_DIALECT` and `NOORM_CONNECTION_DATABASE` are set, from the environment — in which case `createContext()` needs no `.noorm/` directory and no identity at all. That is the mode to use in production; see [Deploying an Application](/guide/deployment). - -> Access is per-config, not per-context: each config declares `access: { user, agent }` (roles `viewer`/`operator`/`admin`, or `agent: false` to hide the config from agents entirely, over MCP and the CLI alike). `channel` tells `createContext` which half of that grant to enforce — see [Access Roles](/guide/environments/configs#access-roles). - -The gate sits at the core seam, so it applies to every method on `ctx.noorm.run`, `.db`, `.changes`, `.lock.forceRelease`, `.vault`, `.secrets`, `.templates`, `.transfer`, and `.dt.importFile`. Raw `ctx.kysely` queries are not gated. Four permissions require confirmation even for `admin` (`db:truncate`, `db:teardown`, `vault:propagate`, `lock:force`), so `ctx.noorm.db.truncate()` on an admin config still throws `ProtectedConfigError` unless the context was created with `yes: true`. - - -## Top-Level Context Properties - -| Property | Type | Description | -| ----------- | ------------ | ------------------------------------------------- | -| `kysely` | `Kysely` | Direct Kysely access (requires `connect()`) | -| `dialect` | `Dialect` | Database dialect (postgres, mysql, sqlite, mssql) | -| `connected` | `boolean` | Whether currently connected | -| `noorm` | `NoormOps` | Noorm management operations (lazy singleton) | - - -## Lifecycle Methods - - -### connect(retryOptions?) - -Establishes the database connection. Calling it on an already-connected context is a no-op. - -```typescript +const ctx = await createContext({ config: 'dev' }); await ctx.connect(); -// Fail fast instead of hanging on an unreachable database. -await ctx.connect({ retries: 1, delay: 0 }); -``` - -`retryOptions` is `ConnectionRetryOptions`: `{ retries?: number; delay?: number; backoff?: number }`, defaulting to `3` attempts, `1000`ms delay, and a `2x` backoff multiplier. Retries cover transient failures (`ECONNREFUSED`, `ETIMEDOUT`); authentication failures and missing drivers fail on the first attempt. - +const applied = await ctx.noorm.changes.ff(); -### disconnect() +const open = await ctx.kysely + .selectFrom('sales_order') + .select(['party_id', 'order_no']) + .where('status', '=', 'open') + .execute(); -Closes the database connection. +console.log(`${applied.executed} changes applied, ${open.length} open orders`); -```typescript await ctx.disconnect(); ``` +`createContext` resolves a config and returns an unconnected `Context`. `connect()` opens the pool. `changes.ff()` applies every pending change and returns a `BatchChangeResult`. `ctx.kysely` is a Kysely instance typed by your schema interface. `disconnect()` closes the pool. -## Transactions - - -### transaction(fn) - -Execute operations within a database transaction. The callback receives a full Kysely `Transaction` with query builder, `sql` template literal, and all Kysely features. - -```typescript -import { sql } from 'kysely'; - -const result = await ctx.transaction(async (trx) => { - await trx - .updateTable('accounts') - .set({ balance: sql`balance - ${amount}` }) - .where('id', '=', fromId) - .execute(); - await trx - .updateTable('accounts') - .set({ balance: sql`balance + ${amount}` }) - .where('id', '=', toId) - .execute(); - return { transferred: amount }; -}); -``` - - -## Impersonation +## How is a context laid out? -### impersonate(username, fn?) -Run queries as a different database principal. The SDK borrows one connection from the pool, switches identity on it, and hands you a scope whose `kysely`, `proc`, `func`, `tvf`, and `transaction` all route through that connection. +Everything on `ctx` runs SQL. Everything noorm itself does hangs off `ctx.noorm`: -Supported on MSSQL (`EXECUTE AS USER` / `REVERT`) and PostgreSQL (`SET ROLE` / `RESET ROLE`). MySQL and SQLite throw `ImpersonationError`, as does a username containing anything outside alphanumerics, `_`, `@`, `.`, `-`, and `\`. - -```typescript -import { ImpersonationError, type ImpersonatedScope } from '@noormdev/sdk'; - -// Callback mode: reverts on the way out, including when the callback throws. -const rows = await ctx.impersonate('app_readonly', async (scope) => { - return scope.kysely.selectFrom('users').selectAll().execute(); -}); - -// Explicit mode: you own the lifecycle. -const scope = await ctx.impersonate('app_readonly'); -const users = await scope.kysely.selectFrom('users').selectAll().execute(); -await scope.revert(); +```mermaid +flowchart TD + CC["createContext()"] --> CTX["Context"] + CTX --> SQL["kysely, transaction()"] + CTX --> RT["proc(), func(), tvf()
impersonate()"] + CTX --> WS["withSchema()"] + CTX --> NO["ctx.noorm"] + NO --> A["run, changes, db"] + NO --> B["lock, vault, secrets"] + NO --> C["templates, transfer,
dt, utils"] ``` -Explicit mode holds a pooled connection until you call `revert()`. `ctx.disconnect()` releases any scope you never reverted, so a forgotten `revert()` cannot deadlock shutdown, but it does keep a connection out of the pool until then. Prefer callback mode unless you need the scope to outlive one function. - -An `ImpersonatedScope` carries no `noorm` namespace and no lifecycle methods. Management operations stay on `ctx.noorm`, running as the context's own principal. +A context is either connected or not. `connect()` moves it to connected, `disconnect()` moves it back, and both are no-ops when already there. Calls that need the pool throw `NotConnectedError` otherwise: +| Area | Needs `connect()` | Works without it | +|---|---|---| +| `ctx` | `kysely`, `transaction`, `proc`, `func`, `tvf`, `impersonate` | `dialect`, `connected`, `withSchema`, `noorm` | +| `run` | `preview`, `file`, `files`, `dir`, `build` | `discover` | +| `changes` | `apply`, `revert`, `ff`, `next`, `rewind`, `status`, `pending`, `history`, `historyForChange`, `getFileHistory` | `create`, `addFile`, `removeFile`, `renameFile`, `reorderFiles`, `delete`, `discover`, `parse`, `validate` | +| `db`, `lock`, `dt` | every method | none | +| `vault` | every method except `copy` | `copy` | +| `transfer` | none | `to` and `plan` open their own connections to both configs | +| `secrets`, `utils` | none | every method | +| `templates` | none | `render`; the vault tier of `$.secrets` resolves only when connected | -## Schema Scoping +Failures arrive as thrown errors, as fields on a result, or as returned data. [Error handling](#error-handling) lists which call does which. [Access policy](#access-policy) explains why a call can throw `ProtectedConfigError` before it touches the database. -### withSchema(name) +## createContext(options) -Derive a `Context` scoped to one schema. Same connection, same pool, same lifecycle as the parent — `withSchema` is a typed wrapper over Kysely's own `withSchema`, not a new connection or config-level state. ```typescript -const acct = ctx.withSchema('accounting'); - -const invoices = await acct.kysely - .selectFrom('invoices') - .selectAll() - .execute(); -// select * from "accounting"."invoices" — typed against AcctDB - -await acct.proc('rebuild_ledger', { year: 2026 }); -// CALL "accounting"."rebuild_ledger"("year" => $1) - -await acct.proc('billing.close_period'); -// CALL "billing"."close_period"() — caller's qualification wins, no prefix added - -await ctx.kysely.selectFrom('users').execute(); -// select * from "users" — the parent is untouched, no prefix - -await ctx.disconnect(); // one lifecycle closes both +createContext( + options?: CreateContextOptions, +): Promise> ``` -`withSchema(name)` takes up to four generics, one per routine kind, the same shape as `createContext`, and returns a `Context` typed to that schema's own tables and routines. - -**Derivation semantics.** The derived context shares the parent's connection, pool, and lifecycle — `connect()`/`disconnect()` on either side act on both — rather than opening a new connection. `ctx.kysely` re-derives from the bare Kysely instance on every access instead of caching a wrapped copy, so a derived context can never leak a stale schema wrap across a reconnect. - -**Replace, not stack.** Calling `withSchema` again on an already-derived context swaps the schema instead of nesting it: `ctx.withSchema('a').withSchema('b')` resolves against `b` only. - -**`proc`/`func`/`tvf` prefixing.** A routine name with no `.` is qualified with the context's schema before it reaches the query builder — `acct.proc('rebuild_ledger', …)` becomes `accounting.rebuild_ledger`. A name that already contains a `.` passes through unqualified, so `acct.proc('billing.close_period')` still resolves against `billing`. - -**Transaction and impersonation composition.** `derived.transaction(fn)` and `derived.impersonate(username, fn)` both inherit the derived schema — every query inside either stays qualified against it, no extra wiring needed. An impersonation scope opened through a derived context still releases when `parent.disconnect()` runs, because `#heldConnections` is shared between parent and derived instances, not per-instance. - -**Raw SQL caveat.** `withSchema` does not rewrite unqualified identifiers inside `` sql`…` `` fragments — they resolve against the connection's default schema regardless of which context ran them. This is inherent to Kysely's plugin model, since raw `sql` bypasses the query builder plugins entirely; qualify by hand inside raw fragments. - -**Non-goals.** - -- No config/connection-level schema field. The connection stays schema-agnostic — `withSchema` is per-call-site sugar, not connection state. -- No raw-SQL rewriting (see the caveat above). -- No schema-defaulting of `ctx.noorm.db.describe*`'s `schema?` argument — pass it explicitly. -- No per-dialect gating. The qualifier means whatever Kysely's `withSchema` means for the active dialect: a schema on PostgreSQL/MSSQL, a database on MySQL, an ATTACHed database name on SQLite. - - -## Stored Procedures, Functions & TVFs +`DB` types `ctx.kysely`. `Procs`, `Funcs`, and `Tvfs` type `proc()`, `func()`, and `tvf()`; see [Stored procedures, functions and TVFs](#stored-procedures-functions-and-tvfs). -Type-safe helpers for calling stored procedures, database functions, and table-valued functions. Define your signatures as interfaces, then pass them as generics to `createContext`: +| Option | Type | Default | Effect | +|---|---|---|---| +| `config` | `string` | `NOORM_CONFIG`, then the active config, then env-only mode | Stored config to use | +| `projectRoot` | `string` | `process.cwd()` | Directory that holds `.noorm/` and the paths in `settings.yml` | +| `requireTest` | `boolean` | `false` | Throws `RequireTestError` unless the resolved config has `isTest: true` | +| `channel` | `'user' \| 'agent'` | `'user'` | Which half of the config's `access` applies; see [Access policy](#access-policy) | +| `stage` | `string` | the stage named like the config | Stage whose `protected: true` caps the config's access. An env-only config also takes the stage's `isTest` | +| `yes` | `boolean` | `false` | Pre-confirms `confirm` cells on the `user` channel. Ignored on `agent` | -```typescript -interface MyProcs { - 'get_users': [{ department_id: number; active: boolean }, User]; - 'simple_proc': [[number, string], void]; - 'refresh_cache': void; -} - -interface MyFuncs { - 'calc_total': [{ order_id: number }, { total: number }]; - 'add_numbers': [[number, number], { result: number }]; - 'get_version': void; -} +The context is not connected when it returns. The config name comes first, then the config is merged from built-in defaults, the stored config, and `NOORM_*` variables: -interface MyTvfs { - 'get_team_members': [{ team_id: number }, TeamMember]; - 'generate_series': [[number, number], { value: number }]; -} - -const ctx = await createContext({ config: 'dev' }); +```mermaid +flowchart TD + A["createContext(options)"] --> B{"config named?"} + B -->|"yes: options.config,
NOORM_CONFIG, active"| C["stored config
(unknown name throws)"] + B -->|"no, env has dialect
and database"| D["env-only config"] + B -->|"no"| E["throws: No config
available"] + C --> F["merge: defaults,
stored, NOORM_*"] + D --> G["merge: defaults,
NOORM_*"] ``` -Each entry maps a name to an `[Args, ReturnType]` tuple. For positional params, wrap in a nested tuple: `[[number, string], ReturnType]`. Plain `void` is shorthand for `[void, void]`. - -When `Procs`, `Funcs`, or `Tvfs` are not provided, the corresponding method (`proc()`, `func()`, `tvf()`) cannot be called — the type system enforces that you define signatures first. - +The later layer wins. Env-only mode names the config `__env__`, which shows up in error messages and event payloads. It reads no stored config, so it needs no `.noorm/` directory: a missing `state.enc` resolves to empty state. [Deploying an application](/guide/deployment) covers that setup. -### proc(name, params?) - -Call a stored procedure and return the result set rows. Generates dialect-specific SQL: - -| Dialect | Named Params | Positional | No Params | -| ---------- | ------------------------------------- | --------------- | ------------- | -| MSSQL | `EXEC name @k = $1` | `EXEC name $1` | `EXEC name` | -| PostgreSQL | `CALL name(k => $1)` | `CALL name($1)` | `CALL name()` | -| MySQL | `CALL name($1)` (positional fallback) | `CALL name($1)` | `CALL name()` | - -```typescript -// Named params — return type inferred from tuple -const users = await ctx.proc('get_users', { department_id: 1, active: true }); -// users: User[] - -// Positional params -await ctx.proc('simple_proc', [42, 'hello']); +| Variable | Effect | +|---|---| +| `NOORM_CONFIG` | Config name when `options.config` is unset | +| `NOORM_CONNECTION_DIALECT` | `postgres`, `mysql`, `sqlite`, or `mssql`. Any other value throws | +| `NOORM_CONNECTION_DATABASE` | With the dialect, switches on env-only mode | +| `NOORM_CONNECTION_HOST`, `_PORT`, `_USER`, `_PASSWORD`, `_SSL`, `_POOL_MIN`, `_POOL_MAX` | Override the matching `connection` field of the resolved config | +| `NOORM_PATHS_SQL`, `NOORM_PATHS_CHANGES` | Override `paths.sql` and `paths.changes` from `settings.yml` | +| `NOORM_IDENTITY_PRIVATE_KEY`, `_NAME`, `_EMAIL` | CI identity. All three are required, otherwise they are ignored. They let the context decrypt state and vault keys with no key file on disk | +| `NOORM_YES` | Any value except empty, `0`, or `false` confirms `confirm` cells on the `user` channel, like `yes: true` | -// No params -await ctx.proc('refresh_cache'); +An env-only config takes the default access, `{ user: 'admin', agent: 'viewer' }`. -// Override return type explicitly -const custom = await ctx.proc<'get_users', CustomUser>('get_users', { department_id: 1, active: true }); -// custom: CustomUser[] -``` +| Throws | When | +|---|---| +| `Config "" not found` | The name matches no stored config | +| `No config available...` | No name, and the env lacks a dialect or database | +| `RequireTestError` | `requireTest: true` on a config without `isTest: true` | +| `Private key required to decrypt state` | A `state.enc` exists and no identity key or `NOORM_IDENTITY_*` is available | -**Returns:** `Promise` — the result set rows. `T` is inferred from the `[Args, ReturnType]` tuple, or overridden via the second generic. -**Throws** on SQLite (no stored procedure support). +## Context members -Procedure, function, and column names are quoted per dialect (`"name"` on PostgreSQL, `[name]` on MSSQL, `` `name` `` on MySQL), so mixed-case and reserved-word names survive. A schema-qualified name splits on the first `.` and each half is quoted separately. -On PostgreSQL, `proc()` falls back to a function call when `CALL` reports the target is not a procedure (SQLSTATE `42809` or `42883`). The SDK retries as `SELECT * FROM name(...)`, so `proc()` reaches both `CREATE PROCEDURE` and `CREATE FUNCTION` objects. +| Member | Returns | Notes | +|---|---|---| +| `kysely` | `Kysely` | Standard Kysely. Re-derived on every access, so a reconnect never leaves a stale schema wrap. Not gated by access policy | +| `dialect` | `Dialect` | `postgres`, `mysql`, `sqlite`, or `mssql` | +| `connected` | `boolean` | `true` between `connect()` and `disconnect()` | +| `noorm` | `NoormOps` | Created on first access, one per context | +| `connect(retryOptions?)` | `Promise` | See [Lifecycle](#lifecycle) | +| `disconnect()` | `Promise` | See [Lifecycle](#lifecycle) | +| `transaction(fn)` | `Promise` | See [Transactions](#transactions) | +| `proc(name, params?)` | `Promise` | See [proc](#proc-name-params) | +| `func(name, params?, column)` | `Promise` | See [func](#func-name-params-column) | +| `tvf(name, params?)` | `Promise` | See [tvf](#tvf-name-params) | +| `impersonate(username, fn?)` | `Promise` or `Promise` | See [Impersonation](#impersonation) | +| `withSchema(name)` | `Context` | See [Schema scoping](#schema-scoping) | +`ctx.noorm` also exposes three read-only properties: -### Parameter handling and NULL semantics +| Property | Type | Holds | +|---|---|---| +| `config` | `Config` | The resolved config | +| `settings` | `Settings` | Project settings: paths, build rules, stages, teardown | +| `identity` | `Identity` | The audit identity recorded on every change and lock | -The SDK picks **one** wire-level meaning for "absent value" and sticks -to it: every key whose value is `undefined` or `null` in the params -object serializes as SQL `NULL`. The SDK does NOT silently drop -`undefined` keys, and it does NOT distinguish `null` from `undefined` -when generating named-parameter SQL. If the key is present in the -object, the SDK emits `@key = NULL` (or the dialect-specific -equivalent). -This matters most for MSSQL stored procedures with `DEFAULT` values. -MSSQL applies a parameter's `DEFAULT` only when the parameter is -*omitted from the call*, not when it's explicitly `NULL`. Sending -`@was_inferred = NULL` against `@was_inferred BIT = 0` will overwrite -the default with NULL — and then fail an INSERT into a `NOT NULL` -column. +### Lifecycle -#### Why this convention - -JavaScript carries two values that mean "no value": `undefined` and -`null`. Mapping both to SQL `NULL` makes the wire-level behavior -predictable across `JSON.stringify` round-trips, Zod schemas with -`.optional()`, and conditional spreads (`...maybeKey && { key }`). -The alternative — treating `undefined` as "omit the key" and `null` -as "send NULL" — works in JavaScript but breaks for HTTP request -bodies and any serializer that drops `undefined` on the way in. - -The trade-off is that you can't rely on the proc's `DEFAULT` -mechanism through an optional Zod field. Two patterns avoid the -surprise: - ```typescript -// Pattern A — encode the default in the schema. -const memoryFlags = z.object({ - wasInferred: z.boolean().default(false), - wasObserved: z.boolean().default(false), -}); - -// Pattern B — omit the key entirely when "absent". -const params: Record = { content: 'x' }; -if (wasInferred !== undefined) { - params.wasInferred = wasInferred; -} -// SDK only emits @wasInferred when the key is present. -await ctx.proc('sp_Memory_Create', params); -``` - -Pattern A pushes the default value into the TypeScript layer; pattern -B keeps it in SQL and trusts MSSQL's `DEFAULT` mechanism. - - -#### Zod cheat-sheet - -| Schema | Parses missing key to | SDK emits | -|--------|----------------------|-----------| -| `z.string()` | (rejects) | n/a — validation fails before SDK call | -| `z.string().nullable()` | `null` if value is `null` | `@key = NULL` | -| `z.string().optional()` | `undefined` | `@key = NULL` (key still present) | -| `z.string().default('')` | `''` | `@key = ''` | - -The pitfall: `z.string().optional()` reads like "the SDK will leave -the parameter alone" but actually means "the SDK will send NULL." -If the proc parameter is `NOT NULL` (or the underlying column is), -prefer `.default(...)` or build the params object manually. - - -#### When NULL is what you actually want - -For columns that genuinely store `NULL`, this convention is exactly -what you want: +await ctx.connect(); -```typescript -// Either path produces @description = NULL. -await ctx.proc('sp_Project_Update', { id: 1, description: undefined }); -await ctx.proc('sp_Project_Update', { id: 1, description: null }); +// Fail fast on an unreachable database. +await ctx.connect({ retries: 1, delay: 0 }); ``` -Both are equivalent. Pick the one that reads better at the call site -— usually `undefined` for "the input didn't carry this field" and -`null` for "explicitly clear the field." - - -### Table-Valued Parameters (TVP) +`connect(retryOptions?)` takes `ConnectionRetryOptions` (not exported; declare it inline): -Pass structured table data to MSSQL stored procedures, scalar functions, and table-valued functions using the `tvp()` helper. TVPs are an MSSQL-only feature — calling with TVP params on other dialects throws. +| Field | Default | Meaning | +|---|---|---| +| `retries` | `3` | Attempts before giving up | +| `delay` | `1000` | Milliseconds before the first retry | +| `backoff` | `2` | Multiplier applied to the delay on each retry | -Instead of binding TVP objects through Kysely's driver (which lacks native TVP type detection), the SDK generates a DECLARE/INSERT batch. All user values remain parameterized — no SQL injection risk. +Only transient failures retry: messages containing `econnrefused`, `etimedout`, `too many connections`, or `connection reset`. Authentication, password, login, access-denied, missing-database, and missing-driver errors fail on the first attempt. After the last attempt, `connect()` throws the server's own error. -```typescript -import { tvp, type TvpValue } from '@noormdev/sdk'; -``` +`disconnect()` releases any impersonation scope you never reverted, destroys the pool, and clears the connection. Calling it while disconnected is a no-op. A disconnected context can `connect()` again. -`TvpValue` is generic — `TvpValue` preserves the row type through your proc, func, and tvf signatures so the compiler catches mismatched columns at call sites. The default `TvpValue` (without a type parameter) is equivalent to `TvpValue>`, so existing unparameterized signatures continue to work. -```typescript -interface BatchItem { title: string; priority: number; list_id: string } +### Transactions -interface MyProcs { - 'batch_insert': [{ user_id: string; items: TvpValue }, { count: number }]; - 'bulk_process': [[number, TvpValue], void]; -} - -interface MyFuncs { - 'score_items': [{ multiplier: number; items: TvpValue }, { total: number }]; -} - -interface MyTvfs { - 'match_items': [{ user_id: string; items: TvpValue }, MatchedItem]; -} - -const ctx = await createContext({ config: 'dev' }); -await ctx.connect(); -``` -Call with `tvp(typeName, rows)` — the type name is the SQL Server table type, and rows is an array of objects whose keys match the type's columns. The generic parameter is inferred from the row objects, so `tvp('ItemBatch', items)` returns `TvpValue` when `items` is `BatchItem[]`: +`transaction(fn)` hands `fn` a Kysely `Transaction` and resolves with its return value. A throw inside `fn` rolls back, so do not swallow errors in the callback. ```typescript -const items: BatchItem[] = [ - { title: 'Task A', priority: 1, list_id: '...' }, - { title: 'Task B', priority: 2, list_id: '...' }, -]; - -// proc — named params (items type-checked against TvpValue) -await ctx.proc('batch_insert', { - user_id: '123', - items: tvp('ItemBatch', items), -}); - -// proc — positional params -await ctx.proc('bulk_process', [42, tvp('ItemBatch', items)]); +const cancelled = await ctx.transaction(async (trx) => { + const open = await trx + .selectFrom('sales_order') + .select('order_no') + .where('party_id', '=', 2) + .where('status', '=', 'open') + .execute(); -// func — scalar result from TVP (uses EXEC @result = func pattern) -const result = await ctx.func('score_items', { - multiplier: 2, - items: tvp('ItemBatch', items), -}, 'total'); + await trx + .updateTable('sales_order') + .set({ status: 'cancelled' }) + .where('party_id', '=', 2) + .where('status', '=', 'open') + .execute(); -// tvf — row set from TVP (uses SELECT * FROM tvf(...)) -const rows = await ctx.tvf('match_items', { - user_id: '123', - items: tvp('ItemBatch', items), + return open.length; }); ``` -**Generated SQL by method:** - -```sql --- proc (named) -DECLARE @__tvp_items ItemBatch; -INSERT INTO @__tvp_items ([title], [priority]) VALUES (@1, @2), (@3, @4); -EXEC batch_insert @user_id = @5, @items = @__tvp_items - --- func (EXEC @result pattern) -DECLARE @__tvp_items ItemBatch; -INSERT INTO @__tvp_items ([title], [priority]) VALUES (@1, @2), (@3, @4); -DECLARE @__result sql_variant; EXEC @__result = score_items @multiplier = @5, @items = @__tvp_items; SELECT @__result AS total - --- tvf (SELECT * FROM) -DECLARE @__tvp_items ItemBatch; -INSERT INTO @__tvp_items ([title], [priority]) VALUES (@1, @2), (@3, @4); -SELECT * FROM match_items(@5, @__tvp_items) -``` - -**Features:** - -| Feature | Example | -| --- | --- | -| Works with proc, func, tvf | `ctx.proc(...)`, `ctx.func(...)`, `ctx.tvf(...)` | -| Schema-qualified type | `tvp('dbo.ItemBatch', rows)` | -| Empty TVP (zero rows) | `tvp('ItemBatch', [])` — passes empty table | -| Multiple TVPs per call | Both `Orders: tvp(...)` and `Items: tvp(...)` in one call | -| Impersonated scopes | `scope.proc('...', { items: tvp(...) })` | -| Many rows | Tested with 50+ rows per TVP | - -**Validation:** - -The `tvp()` helper validates inputs at creation time: - -- **Row key consistency** — all rows must have the same keys as the first row. Throws with a clear message identifying the mismatched row. -- **Empty type name** — throws `TVP type name is required.` -The SQL builders validate before generating SQL: +### Impersonation -- **Parameter count limit** — MSSQL supports at most 2,100 bound parameters per batch. The SDK counts `(TVP rows × columns) + scalar params` and throws early: `TVP parameter count (2401) exceeds MSSQL limit of 2100. Split your TVP rows into smaller batches and call the procedure multiple times.` -- **Dialect guard** — throws `Table-valued parameters (TVP) are only supported on MSSQL.` on non-MSSQL dialects. +`impersonate(username, fn?)` runs queries as another database principal. It borrows one connection from the pool, switches identity on it, and returns a scope whose `kysely`, `proc`, `func`, `tvf`, and `transaction` use that connection. -### func(name, params?, column) - -Call a database function and return the scalar result. Generates `SELECT name(...) AS column`, except on MSSQL with named params, where T-SQL has no named-argument form for `SELECT` and the SDK uses `EXEC @var = name` instead. MySQL rejects named params outright. - -| Dialect | Named Params | Positional | No Params | -| ---------- | ---------------------------------------------- | ------------------------ | ---------------------- | -| MSSQL | `EXEC @var = name @k = $1; SELECT @var AS col` | `SELECT name($1) AS col` | `SELECT name() AS col` | -| PostgreSQL | `SELECT name(k => $1) AS col` | `SELECT name($1) AS col` | `SELECT name() AS col` | -| MySQL | throws (pass an array instead) | `SELECT name($1) AS col` | `SELECT name() AS col` | - -Passing an object to `func()` on MySQL throws `MySQL does not support named parameters in function calls. Use positional parameters (array) instead.` This is the one place where a params object is not silently reordered into positional form: `proc()` on MySQL does flatten an object to `Object.values()` order, which is why `func()` refuses rather than guessing. +| Dialect | Switch | Revert | +|---|---|---| +| `mssql` | `EXECUTE AS USER = ''` | `REVERT` | +| `postgres` | `SET ROLE ''` | `RESET ROLE` | +| `mysql`, `sqlite` | throws `ImpersonationError` | | -```typescript -// Named params + column alias — return type inferred from tuple -const result = await ctx.func('calc_total', { order_id: 42 }, 'total'); -// result: { total: number } - -// Positional params + column alias -const sum = await ctx.func('add_numbers', [1, 2], 'result'); -// sum: { result: number } +A username may contain letters, digits, `_`, `@`, `.`, `-`, and `\`. Anything else throws `ImpersonationError`. -// No params — just column alias -const ver = await ctx.func('get_version', 'v'); +The two modes differ in who ends the borrow: -// Override return type explicitly -const custom = await ctx.func<'calc_total', { amount: number }>('calc_total', { order_id: 42 }, 'amount'); -// custom: { amount: number } +```mermaid +sequenceDiagram + participant App + participant Context + participant Database + App->>Context: impersonate("app_readonly", fn) + Context->>Database: borrow one connection, SET ROLE + Context->>App: fn(scope) + App->>Database: scope.kysely queries on the same connection + App-->>Context: fn returns or throws + Context->>Database: RESET ROLE, release connection ``` -**Returns:** `Promise` — the first row (scalar value as `{ column: value }`). `T` is inferred from the `[Args, ReturnType]` tuple, or overridden via the second generic. - -**Throws** on SQLite (no database function call support). - - -### tvf(name, params?) - -Call a table-valued function and return the result set rows. Supported on MSSQL and PostgreSQL only. - -| Dialect | Named Params | Positional | No Params | -| ---------- | ----------------------------------------- | ------------------------- | ----------------------- | -| MSSQL | `SELECT * FROM name($1)` (flattened) | `SELECT * FROM name($1)` | `SELECT * FROM name()` | -| PostgreSQL | `SELECT * FROM name(k => $1)` | `SELECT * FROM name($1)` | `SELECT * FROM name()` | - -A T-SQL `FROM` clause takes only positional arguments, so on MSSQL a params object is flattened to its values in key order. Declare MSSQL TVF params as an array when the order matters to you rather than relying on object key order. - ```typescript -// Named params — return type inferred from tuple -const members = await ctx.tvf('get_team_members', { team_id: 5 }); -// members: TeamMember[] +import { type ImpersonatedScope } from '@noormdev/sdk'; -// Positional params -const series = await ctx.tvf('generate_series', [1, 10]); -// series: { value: number }[] - -// Override return type explicitly -const custom = await ctx.tvf<'get_team_members', CustomMember>('get_team_members', { team_id: 5 }); -// custom: CustomMember[] -``` - -**Returns:** `Promise` — the result set rows. `T` is inferred from the `[Args, ReturnType]` tuple, or overridden via the second generic. - -**Throws** on SQLite and MySQL (no table-valued function support). - - -## ctx.noorm — Noorm Operations - - -### Properties - -| Property | Type | Description | -| ---------- | ---------------- | --------------------------------------- | -| `config` | `Config` | The resolved config object | -| `settings` | `Settings` | Project settings (paths, rules, stages) | -| `identity` | `Identity` | Current operator identity | - -There is no `ctx.noorm.observer`. Events come off the process-global `noormObserver` export instead. See [Event Subscriptions](#event-subscriptions). - - -### ctx.noorm.run — Run Operations - - -#### run.build(options?) - -Execute all SQL files in the schema directory. +// Callback mode: reverts on the way out, including when fn throws. +const rows = await ctx.impersonate('app_readonly', (scope) => + scope.kysely.selectFrom('sales_order').selectAll().execute(), +); -```typescript -const result = await ctx.noorm.run.build({ force: true }); -console.log(`Ran ${result.filesRun} files`); +// Explicit mode: you own the lifecycle. +const scope: ImpersonatedScope = await ctx.impersonate('app_readonly'); +const orders = await scope.kysely.selectFrom('sales_order').selectAll().execute(); +await scope.revert(); ``` -**Options (`BuildOptions`):** `{ force?: boolean; dryRun?: boolean }`. +| Behavior | Detail | +|---|---| +| Scope members | `kysely`, `proc`, `func`, `tvf`, `transaction`, `revert`. No `noorm` and no lifecycle methods; management operations stay on `ctx.noorm`, running as the context's own principal | +| Explicit mode holds a pooled connection | Until `revert()` or `ctx.disconnect()`. `disconnect()` releases an unreverted scope without running the revert SQL, because it destroys the pool | +| `revert()` | Idempotent after it succeeds. After a failure the scope stays revertible, so a retry re-runs the revert | -**Returns:** `Promise`. See [Result shape](#run-result-shape) for the `error` and `skipReason` fields you'll find on each entry of `result.files`. -`build()` is the one run method that applies `settings.build.include` / `exclude` and `settings.rules`, so headless callers see the same file set the TUI's Run Build screen shows. Include or exclude entries that matched no file come back as `result.unmatchedInclude` / `result.unmatchedExclude` rather than as a failure: a build over zero files still succeeds, so these arrays are how a mistyped pattern surfaces. +### Schema scoping -#### run.file(filepath, options?) - -Execute a single SQL file. +`withSchema(name)` returns a `Context` scoped to one schema. It wraps Kysely's own `withSchema`; it is not a new connection. ```typescript -await ctx.noorm.run.file('seeds/test-data.sql'); -``` +const billing = ctx.withSchema('billing'); -**Options (`RunOptions`):** `{ force?: boolean; abortOnError?: boolean; dryRun?: boolean; preview?: boolean; output?: string | null }`. `preview` renders the file without executing it or writing history. +await billing.kysely.selectFrom('invoice').selectAll().execute(); +// select * from "billing"."invoice", typed against BillingDB -**Returns:** `Promise`. On failure `result.error` carries the SQL/load error message; on skip `result.skipReason` carries `'unchanged'`. A skipped file writes no execution history. +await billing.proc('post_invoices', { party_id: 2 }); +// CALL "billing"."post_invoices"("party_id" => $1) +await billing.proc('sales.close_order', [2, 1]); +// CALL "sales"."close_order"($1, $2), a name with a dot is used as given -#### run.files(filepaths, options?) - -Execute multiple SQL files sequentially. - -```typescript -await ctx.noorm.run.files([ - 'functions/utils.sql', - 'triggers/audit.sql', -]); +await ctx.kysely.selectFrom('sales_order').selectAll().execute(); +// the parent is untouched, no prefix ``` -**Returns:** `Promise`. Per-file errors and skip reasons live on `result.files[]`. +`BillingDB` and `BillingProcs` are interfaces you declare, like `OrderToCash` in the first example. The routine generics default to the parent's. +| Rule | Behavior | +|---|---| +| Lifecycle | The derived context shares the parent's connection, pool, and held-connection set. `connect()` and `disconnect()` on either act on both | +| Name check | `name` must match letters, digits, `_`, and `-`. Anything else, including `.`, throws `Invalid schema name` immediately, before any connection is touched | +| Replace, not stack | `ctx.withSchema('a').withSchema('b')` resolves against `b` only | +| Routine names | `proc`, `func`, and `tvf` prefix a name that has no `.` with the schema. A name with a `.` passes through | +| `transaction` | Inherits the schema | +| `impersonate` | `scope.kysely` and `scope.transaction` stay schema-qualified. `scope.proc`, `scope.func`, and `scope.tvf` call the builders with the name as given, so qualify routine names yourself inside a scope | +| Raw `` sql`...` `` | Not rewritten. Unqualified identifiers resolve against the connection's default schema, so qualify them by hand | +| `ctx.noorm` | Not affected. Pass `schema` to `db.describe*` explicitly | -#### run.dir(dirpath, options?) +There is no schema field on the config, and no per-dialect gating. The qualifier means whatever Kysely's `withSchema` means for the active dialect: a schema on postgres and mssql, a database on mysql, an attached database name on sqlite. -Execute all SQL files in a directory. -```typescript -await ctx.noorm.run.dir('seeds/'); -``` - -**Returns:** `Promise`. Per-file errors and skip reasons live on `result.files[]`. +## Stored procedures, functions and TVFs -#### Run result shape - -`FileResult` (returned by `run.file`, and by each entry in `BatchResult.files`): - -| Field | Type | When set | -|-------|------|----------| -| `filepath` | `string` | Always | -| `checksum` | `string` | Always (empty when the file could not be read) | -| `status` | `'success' \| 'failed' \| 'skipped'` | Always | -| `error` | `string` | Only when `status === 'failed'` | -| `skipReason` | `'unchanged' \| 'already-run'` | Only when `status === 'skipped'` | -| `durationMs` | `number` | Set for executed and failed files | -| `renderedSql` | `string` | Only in preview mode (`run.preview`) | -| `outputPath` | `string` | Only on a dry run that wrote rendered SQL to disk | - -Example handling a failed build: +`proc()`, `func()`, and `tvf()` call routines by name, with the SQL built for the active dialect and every value bound as a parameter. They are not gated by access policy. Declare each routine your database defines as an `[Args, Return]` tuple and pass the maps as generics: ```typescript -const result = await ctx.noorm.run.build(); - -if (result.status !== 'success') { - - for (const file of result.files) { - - if (file.status === 'failed') { - - console.error(`${file.filepath}: ${file.error}`); +// src/db.ts +import { createContext, type TvpValue } from '@noormdev/sdk'; - } - - } - - process.exit(1); +interface NewLine { product_id: number; qty: number } +interface Procs { + place_order: [{ party_id: number; lines: TvpValue }, { order_no: number }]; + set_order_status: [{ party_id: number; order_no: number; status?: string }, void]; + close_order: [[number, number], void]; + archive_closed_orders: void; } -``` - - -#### run.discover(dirpath?) - -Discover SQL files in a directory without executing. Works offline — no connection required. Defaults to the project's configured SQL path. - -```typescript -const files = await ctx.noorm.run.discover('sql/'); -console.log(`Found ${files.length} SQL files`); -``` - -**Returns:** `Promise` — absolute paths to discovered SQL files. - -#### run.preview(filepaths, output?) - -Render SQL files (including templates) without executing. Useful for reviewing what would run before committing. - -```typescript -const results = await ctx.noorm.run.preview(['sql/001.sql', 'sql/002.sql']); -for (const r of results) { - console.log(`${r.filepath}:\n${r.renderedSql}`); +interface Funcs { + party_balance: [{ party_id: number }, { balance: number }]; + line_total: [[number, number], { total: number }]; + schema_version: [void, { v: string }]; } -``` - -**Returns:** `Promise` — the rendered SQL for each file lands in `renderedSql`, not `sql`. A file that failed to load or render comes back with `status: 'failed'` and an `error` message instead of throwing. - -Pass a second argument to write the combined output to a file: `run.preview(paths, 'tmp/preview.sql')`. Rendering resolves every secret into the returned string, so treat preview output as plaintext secrets. - -### ctx.noorm.db — Database Operations - - -#### db.truncate(options?) - -Wipe all data, keeping the schema intact. - -```typescript -const result = await ctx.noorm.db.truncate(); -console.log(`Truncated ${result.truncated.length} tables`); - -// Keep a reference table standing. -await ctx.noorm.db.truncate({ preserve: ['countries'] }); - -// Or restrict the wipe to a few tables. -await ctx.noorm.db.truncate({ only: ['users', 'posts'] }); -``` - -**Options (`TruncateOptions`):** - -| Option | Type | Default | Description | -| ----------------- | ---------- | ------- | -------------------------------------------------------------------------- | -| `preserve` | `string[]` | — | Tables to leave alone. Falls back to `settings.teardown.preserveTables`. | -| `only` | `string[]` | — | Truncate only these tables. Inverse of `preserve`. | -| `restartIdentity` | `boolean` | `true` | Restart identity / auto-increment sequences. | -| `dryRun` | `boolean` | `false` | Return the SQL in `result.statements` without executing it. | - -**Returns:** `Promise` — `{ truncated, preserved, statements, durationMs }`. - -A `dryRun` is checked against the config's role but never against a confirmation gate, so a role that may truncate can always look first. A role that may not truncate cannot preview either. - -**Implementation notes.** FK constraints are disabled first, every targeted table is `DELETE`d (or `TRUNCATE`d, where the dialect supports it), then constraints are re-enabled. PG/MySQL/SQLite flip a session- or connection-level switch once. MSSQL, which has no session-level toggle, emits one `ALTER TABLE [name] NOCHECK CONSTRAINT ALL` per truncated table on a single connection — replacing the older `sp_MSforeachtable` call that spawned parallel workers and could deadlock. See [dev/teardown.md](../dev/teardown.md#mssql-truncate-strategy) for the full reasoning. - - -#### db.teardown(options?) - -Drop all database objects except noorm's own internal tables, which cover change tracking, locks, vault, and identities. - -```typescript -const result = await ctx.noorm.db.teardown(); - -// Compute the drop plan without executing it. -const plan = await ctx.noorm.db.teardown({ dryRun: true }); - -// Leave an application-owned schema untouched. -await ctx.noorm.db.teardown({ preserveSchemas: ['app_private'] }); -``` - -**Options (`TeardownOptions`):** the SDK reads two fields off what you pass, `dryRun` and `preserveSchemas`. It fills `preserveTables` and `postScript` from `settings.teardown` in `settings.yml`. The remaining `TeardownOptions` fields (`keepViews`, `keepFunctions`, `keepProcedures`, `keepTypes`, and a caller-supplied `preserveTables`) are dropped on the way through and have no effect from the SDK. - -`preserveSchemas` exists because `preserveTables` is a flat name list: excluding `app_private.secrets` by name would also spare `public.secrets`. - -**Returns:** `Promise` — `{ dropped: { tables, views, functions, procedures, types, foreignKeys }, preserved, statements, durationMs }`, plus `postScriptResult`, `staleCount`, and `resetRecordId` when they apply. - -As with `truncate`, a `dryRun` is gated on the role but not on confirmation. - - -#### db.previewTeardown() - -Preview what teardown would drop without executing. Useful for confirming destructive operations before running them. - -```typescript -const preview = await ctx.noorm.db.previewTeardown(); -for (const [kind, names] of Object.entries(preview.toDrop)) { - console.log(`Would drop ${names.length} ${kind}`); +interface Tvfs { + party_open_orders: [{ party_id: number }, { order_no: number; status: string }]; } -``` - -**Returns:** `Promise` — `{ toDrop, toPreserve, statements }`. `toDrop` groups names by object kind (`tables`, `views`, `functions`, `procedures`, `types`, `foreignKeys`); there is no flat `objects` array. - - -#### db.reset() -Full rebuild: teardown + build. - -```typescript -await ctx.noorm.db.reset(); +export const ctx = await createContext({ config: 'dev' }); ``` -**Returns:** `Promise`. Unlike `teardown()` and `truncate()`, `reset()` takes no options and deliberately ignores `settings.teardown.preserveTables`: it rebuilds the whole schema from `sql/`, so any preserved table would collide with the build's `CREATE TABLE` and abort the rebuild. - -**Implementation notes.** Objects are dropped in `FK → Procedures → Functions → Views → Tables → Types` order. Procs/funcs/views go before tables because MSSQL schema-bound objects (`WITH SCHEMABINDING`) hold dependency locks on the tables they reference — dropping the table first fails with `Cannot DROP TABLE ... because it is being referenced by object ...`. Types drop last because TVPs may still be referenced by procs/funcs earlier in the chain. See [dev/teardown.md](../dev/teardown.md#drop-order) for the full reasoning. +The routine names above are examples. A tuple's first element types the arguments, and the second types the rows returned: +| Entry | Arguments | Return | +|---|---|---| +| `[{ k: T }, R]` | object, named parameters | `R` | +| `[[A, B], R]` | array, positional parameters | `R` | +| `[Args, void]` | as above | `void` | +| `[void, R]` | none | `R` | +| `void` | none | `unknown` | -### ctx.noorm.changes — Change Management +A routine map you do not provide leaves the matching method uncallable at the type level. Each method takes a second generic to override the inferred return type, such as `ctx.proc<'set_order_status', MyRow>(...)`. Names are quoted per dialect (`"name"`, `[name]`, `` `name` ``), and a schema-qualified name splits on its first `.` and quotes each half. In the tables below, placeholders are written `$1`; the driver supplies its own markers. -#### Scaffold Operations (offline) - -These operations work without a database connection — they manage change directories on disk. - - -#### changes.create(options) - -Create a new change directory with `change/` and `revert/` folders. - -```typescript -const change = await ctx.noorm.changes.create({ description: 'add-user-roles' }); -console.log(change.name); // e.g. '2024-01-15-add-user-roles' -``` - -**Returns:** `Promise` — the parsed change object. - - -#### changes.addFile(change, folder, options) - -Add a file to a change's `change/` or `revert/` folder. - -```typescript -const updated = await ctx.noorm.changes.addFile(change, 'change', { - name: 'create-table', - type: 'sql', -}); -``` - -**Returns:** `Promise` — the updated change object. - - -#### changes.removeFile(change, folder, filename) - -Remove a file from a change. - -```typescript -await ctx.noorm.changes.removeFile(change, 'change', '001_create-table.sql'); -``` - -**Returns:** `Promise` — the updated change object. +### proc(name, params?) -#### changes.renameFile(change, folder, oldFilename, newDescription) +Calls a stored procedure and returns its result-set rows as `Promise`. -Rename a file in a change folder. Preserves the numeric prefix. +| Dialect | Named | Positional | No params | +|---|---|---|---| +| mssql | `EXEC name @k = $1` | `EXEC name $1` | `EXEC name` | +| postgres | `CALL name(k => $1)` | `CALL name($1)` | `CALL name()` | +| mysql | `CALL name($1)`, values in key order | `CALL name($1)` | `CALL name()` | +| sqlite | throws | throws | throws | ```typescript -await ctx.noorm.changes.renameFile(change, 'change', '001_old.sql', 'new-name'); +await ctx.proc('set_order_status', { party_id: 2, order_no: 1, status: 'paid' }); +await ctx.proc('close_order', [2, 1]); +await ctx.proc('archive_closed_orders'); ``` -**Returns:** `Promise` — the updated change object. +On postgres, `proc()` retries as `SELECT * FROM name(...)` when `CALL` fails with SQLSTATE `42809` or `42883`. One method therefore reaches both `CREATE PROCEDURE` and `CREATE FUNCTION` objects. -#### changes.reorderFiles(change, folder, newOrder) - -Reorder files in a change folder. Pass filenames in the desired order — numeric prefixes are reassigned. +### Parameter handling and NULL semantics -```typescript -await ctx.noorm.changes.reorderFiles(change, 'change', [ - '002_b.sql', - '001_a.sql', -]); -``` -**Returns:** `Promise` — the updated change object. +Every key present in a params object is sent. The SDK never drops a key, and a value of `undefined` or `null` is bound as SQL `NULL` (`sql.ts` builds one `@key = ` per `Object.entries` pair, and Kysely's mssql driver binds both as NVarChar `NULL`). +MSSQL applies a parameter's `DEFAULT` only when the parameter is omitted from the call. `status: undefined` against `@status text = 'open'` sends `@status = NULL`, overwrites the default, and fails a `NOT NULL` column. -#### changes.delete(change) +| Zod schema | A missing key parses to | The SDK sends | +|---|---|---| +| `z.string()` | rejects before the call | nothing | +| `z.string().nullable()` | `null` when the value is `null` | `@key = NULL` | +| `z.string().optional()` | `undefined` | `@key = NULL`, the key is still present | +| `z.string().default('open')` | `'open'` | `@key = 'open'` | -Delete a change directory from disk entirely. +`.optional()` reads like "leave the parameter alone" and means "send NULL". Two patterns keep the SQL default working: ```typescript -await ctx.noorm.changes.delete(change); -``` - - -#### changes.discover() - -Discover all changes on disk. Works offline — scans the configured changes directory. +// Pattern A: put the default in the schema. +const StatusChange = z.object({ + party_id: z.number(), + order_no: z.number(), + status: z.string().default('open'), +}); -```typescript -const changes = await ctx.noorm.changes.discover(); -console.log(`Found ${changes.length} changes`); +// Pattern B: leave the key out when the value is absent. +const params: Record = { party_id: 2, order_no: 1 }; +if (status !== undefined) { + params.status = status; +} +await ctx.proc('set_order_status', params); ``` -**Returns:** `Promise` — all parsed change objects. - +For a column that stores `NULL`, `{ note: undefined }` and `{ note: null }` are the same call. Use `undefined` for "the input did not carry this field" and `null` for "clear it". -#### changes.parse(name) -Parse a single change from disk by name. - -```typescript -const change = await ctx.noorm.changes.parse('2024-01-15-add-users'); -``` +### func(name, params?, column) -**Returns:** `Promise` — the parsed change object. +Calls a database function and returns the first row, `{ [column]: value }`, or `null` when there is none. The last argument is the result column alias. -#### changes.validate(change) +| Dialect | Named | Positional | No params | +|---|---|---|---| +| mssql | `DECLARE @__result sql_variant; EXEC @__result = name @k = $1; SELECT @__result AS col` | `SELECT name($1) AS col` | `SELECT name() AS col` | +| postgres | `SELECT name(k => $1) AS col` | `SELECT name($1) AS col` | `SELECT name() AS col` | +| mysql | throws | `SELECT name($1) AS col` | `SELECT name() AS col` | +| sqlite | throws | throws | throws | -Validate a change's structure. Throws `ChangeValidationError` if the change is malformed. +T-SQL has no named arguments in `SELECT`, which is why mssql uses `EXEC @var = name`. mysql throws `MySQL does not support named parameters in function calls. Use positional parameters (array) instead.` rather than reorder an object into positional form. ```typescript -ctx.noorm.changes.validate(change); +const { balance } = await ctx.func('party_balance', { party_id: 2 }, 'balance'); +const { total } = await ctx.func('line_total', [3, 49], 'total'); +const { v } = await ctx.func('schema_version', 'v'); ``` -**Returns:** `void` — throws on invalid structure. +### tvf(name, params?) -#### Execution Operations (connected) +Calls a table-valued function and returns its rows as `Promise`. mssql and postgres only; mysql and sqlite throw. -#### changes.apply(name, options?) +| Dialect | Named | Positional | No params | +|---|---|---|---| +| mssql | `SELECT * FROM name($1)`, values in key order | `SELECT * FROM name($1)` | `SELECT * FROM name()` | +| postgres | `SELECT * FROM name(k => $1)` | `SELECT * FROM name($1)` | `SELECT * FROM name()` | -Apply a specific change. Pass `{ dryRun: true }` to render the change to `tmp/` without touching the database, or `{ preview: true }` to emit rendered SQL without writing. `{ force: true }` bypasses the already-applied check. +A T-SQL `FROM` clause takes positional arguments only. On mssql, declare the parameters as an array when order matters, because an object is flattened in key order. ```typescript -const result = await ctx.noorm.changes.apply('2024-01-15-add-users'); - -// Dry run — does not touch the change tracking tables. -const dry = await ctx.noorm.changes.apply( - '2024-01-15-add-users', - { dryRun: true }, -); +const open = await ctx.tvf('party_open_orders', { party_id: 2 }); ``` -**Options:** `ChangeOptions` — `{ force?: boolean; dryRun?: boolean; preview?: boolean; output?: string | null }`. - -**Returns:** `Promise`. +### Table-valued parameters -#### changes.revert(name, options?) -Revert a specific change. Accepts the same `ChangeOptions` as `apply()`. +`tvp(typeName, rows)` passes an array of rows to an mssql routine as a table-valued parameter. Other dialects throw `Table-valued parameters (TVP) are only supported on MSSQL.` Kysely's driver cannot bind a TVP, so the SDK emits a batch that declares a table variable of your table type, inserts the rows into it, and passes the variable. Every value stays a bound parameter. ```typescript -const result = await ctx.noorm.changes.revert('2024-01-15-add-users'); - -// Dry-run a revert before pulling the trigger in production. -const dry = await ctx.noorm.changes.revert( - '2024-01-15-add-users', - { dryRun: true }, -); -``` - +import { tvp } from '@noormdev/sdk'; -#### changes.ff(options?) - -Apply all pending changes. Accepts `BatchChangeOptions` — the same fields as `ChangeOptions` plus `abortOnError`. Pass `{ dryRun: true }` to render every pending change to `tmp/` without writing to the database; the change tracking tables are left untouched. - -```typescript -const result = await ctx.noorm.changes.ff(); -console.log(`Applied ${result.executed} changes`); +const lines: NewLine[] = [ + { product_id: 10, qty: 1 }, + { product_id: 11, qty: 2 }, +]; -// Preview before a production deploy. -const dry = await ctx.noorm.changes.ff({ dryRun: true }); +const [{ order_no }] = await ctx.proc('place_order', { + party_id: 2, + lines: tvp('order_line_input', lines), +}); ``` +`TvpValue` keeps the row type through the signature, so the compiler checks the columns. A TVP works in `proc()`, `func()`, `tvf()`, and in `scope.proc()` and its siblings: -#### changes.next(count?, options?) +| Method | Statement shape | +|---|---| +| `proc` | `DECLARE @__tvp_ ; INSERT INTO @__tvp_ ([col], ...) VALUES (...), (...); EXEC [name] @k = $1, @ = @__tvp_` | +| `func` | the same preamble, then `DECLARE @__result sql_variant; EXEC @__result = [name] ...; SELECT @__result AS [col]` | +| `tvf` | the same preamble, then `SELECT * FROM [name]($1, @__tvp_)` | -Apply the next `count` pending changes (default `1`). Accepts the same `BatchChangeOptions` as `ff()`. +With positional parameters the variable is `@__tvp_`. -```typescript -const result = await ctx.noorm.changes.next(); -const three = await ctx.noorm.changes.next(3); - -// Render the next two without touching the database. -const dry = await ctx.noorm.changes.next(2, { dryRun: true }); -``` +| Rule | Behavior | +|---|---| +| Type name | A schema-qualified name works: `tvp('dbo.order_line_input', rows)`. An empty name throws `TVP type name is required.` | +| Row keys | Every row must have the first row's keys, or `tvp()` throws `TVP row has mismatched keys...`. Columns come from the first row's keys | +| Empty TVP | `tvp('order_line_input', [])` declares the variable and inserts nothing | +| Several TVPs | Allowed in one call | +| Parameter limit | mssql allows 2,100 bound parameters per batch. The SDK counts `rows x columns` per TVP plus scalar parameters, and throws `TVP parameter count (n) exceeds MSSQL limit of 2100...` before sending. Split the rows and call again | -#### changes.rewind(target, options?) +## Access policy -Revert applied changes in reverse order. Pass a change name to revert back to and including that change, or a number to revert that many of the most recent ones. Accepts the same `BatchChangeOptions` as `ff()`. -```typescript -// Back out everything applied after (and including) this change. -const result = await ctx.noorm.changes.rewind('2024-01-15-add-users'); +Each config grants a role to two channels: `access: { user, agent }`, with roles `viewer`, `operator`, and `admin`, or `agent: false` to hide the config from agents entirely. The default is `{ user: 'admin', agent: 'viewer' }`. A stage with `protected: true` caps its configs at `operator` for the user and `viewer` for agents. [Access roles](/guide/environments/configs#access-roles) covers setting them. -// Or back out the last three. -const three = await ctx.noorm.changes.rewind(3); +`createContext({ channel })` picks which half applies. Each permission resolves to `allow`, `deny`, or `confirm`, and the SDK has no prompt to answer a `confirm`: -// Render the revert files without touching the database. -const dry = await ctx.noorm.changes.rewind(3, { dryRun: true }); -``` +| Matrix cell | `user` channel | `agent` channel | +|---|---|---| +| `allow` | runs | runs | +| `deny` | throws | throws | +| `confirm` | throws unless `yes: true` or `NOORM_YES` is set | throws, and `yes` is never read | -**Returns:** `Promise`. A name matching no applied change comes back as a failed batch carrying `result.error` (`No applied change named "..." to rewind to`), not a throw. +A gated method checks before it does any work and throws `ProtectedConfigError` (`configName`, `operation`). Raw `ctx.kysely` queries, `proc`, `func`, `tvf`, `impersonate`, and `transaction` are not gated. These are the permissions that SDK methods check; the Gate column of each namespace table names the one each method uses: +| Permission | `viewer` | `operator` | `admin` | +|---|---|---|---| +| `run:build`, `run:file`, `run:dir` | deny | confirm | allow | +| `change:run`, `change:ff`, `change:revert` | deny | confirm | allow | +| `change:rm` | deny | confirm | confirm | +| `db:reset` | deny | confirm | allow | +| `db:truncate` | deny | confirm | confirm | +| `db:teardown` | deny | deny | confirm | +| `lock:force` | deny | confirm | confirm | +| `secret:read` | deny | allow | allow | +| `secret:write` | deny | confirm | allow | +| `vault:read` | deny | allow | allow | +| `vault:write` | deny | confirm | allow | +| `vault:propagate` | deny | confirm | confirm | +| `transfer:plan` | deny | allow | allow | -#### changes.status() +Three exceptions to the plain rule: -Get status of all changes. +- `db.truncate` and `db.teardown` with `dryRun: true` check for denial but skip confirmation, so a role that may run the operation can always look first. A role that may not run it cannot preview it either. +- Denials raised below the SDK seam throw a plain `Error` with the same message, not `ProtectedConfigError`. That covers `run.preview`, `transfer.plan`, and `vault.copy`. Match on the message there. +- `transfer.to` and `transfer.plan` check the destination config, not the source, because the destination is the write target. `vault.copy` checks `vault:read` on the source and `vault:write` on the destination. -```typescript -const changes = await ctx.noorm.changes.status(); -for (const cs of changes) { - console.log(`${cs.name}: ${cs.status}`); -} -``` +## ctx.noorm namespaces -#### changes.pending() -Get only pending changes. +| Namespace | Does | +|---|---| +| [`run`](#ctx-noorm-run) | Executes and previews SQL files, and builds the schema | +| [`changes`](#ctx-noorm-changes) | Scaffolds, applies, reverts, and reports changes | +| [`db`](#ctx-noorm-db) | Explores the schema, truncates, tears down, and resets | +| [`lock`](#ctx-noorm-lock) | Holds the per-config lock that serializes schema operations | +| [`secrets`](#ctx-noorm-secrets) | Reads and writes config-scoped secrets in local encrypted state | +| [`vault`](#ctx-noorm-vault) | Reads and writes team secrets stored in the database | +| [`templates`](#ctx-noorm-templates) | Renders a `.sql.tmpl` file without executing it | +| [`transfer`](#ctx-noorm-transfer) | Copies data between two configs | +| [`dt`](#ctx-noorm-dt) | Exports and imports `.dt` data files | +| [`utils`](#ctx-noorm-utils) | Checksums a file and probes a connection | -```typescript -const pending = await ctx.noorm.changes.pending(); -``` +### ctx.noorm.run -### ctx.noorm.db — Explore +Relative paths resolve against `projectRoot`. -#### db.listTables() +| Method | Returns | Gate | Notes | +|---|---|---|---| +| `discover(dirpath?)` | `Promise` | none | Absolute, sorted paths of `.sql` and `.sql.tmpl` files. Defaults to `settings.paths.sql`. Throws `Failed to read directory` for a missing one | +| `preview(filepaths, output?)` | `Promise` | `run:file`, deny only | Renders without executing. SQL lands in `renderedSql`. A file that fails to load or render returns `status: 'failed'` with `error`. `output` writes the combined text to a file; a write failure emits an `error` event | +| `file(filepath, options?)` | `Promise` | `run:file` | Skips an unchanged file with `skipReason: 'unchanged'` | +| `files(filepaths, options?)` | `Promise` | `run:dir` | Runs in the order given | +| `dir(dirpath, options?)` | `Promise` | `run:dir` | Runs every SQL file under the directory, sorted by path. A missing directory returns `status: 'failed'` with `error` | +| `build(options?)` | `Promise` | `run:build` | Runs the schema directory. Applies `settings.build.include`, `build.exclude`, and `settings.rules` | -List all tables in the database. +`preview` resolves every secret into the returned text, so treat its output as plaintext secrets. ```typescript -const tables = await ctx.noorm.db.listTables(); -for (const table of tables) { - console.log(`${table.name}: ${table.columnCount} columns`); -} -``` - - -#### db.describeTable(name, schema?) - -Get detailed information about a table. +const result = await ctx.noorm.run.build({ force: true }); -```typescript -const detail = await ctx.noorm.db.describeTable('users'); -if (detail) { - for (const col of detail.columns) { - console.log(`${col.name}: ${col.dataType}`); +if (result.status !== 'success') { + for (const file of result.files.filter((f) => f.status === 'failed')) { + console.error(`${file.filepath}: ${file.error}`); } + process.exit(1); } -``` - - -#### db.overview() - -Get database overview with counts of all object types. - -```typescript -const overview = await ctx.noorm.db.overview(); -console.log(`Tables: ${overview.tables}, Views: ${overview.views}`); -``` - -**Returns:** `Promise` — counts for `tables`, `views`, `procedures`, `functions`, `types`, `indexes`, `foreignKeys`, `triggers`, `locks`, and `connections`. - - -#### Other explore methods - -Tables are not the only object kind. Every `list*` method returns a summary array, and every `describe*` method takes `(name, schema?)` and returns the detail object or `null` when the object does not exist. - -| Method | Returns | -| ---------------------------------- | ----------------------------- | -| `db.listViews()` | `ViewSummary[]` | -| `db.describeView(name, schema?)` | `ViewDetail \| null` | -| `db.listProcedures()` | `ProcedureSummary[]` | -| `db.describeProcedure(name, schema?)` | `ProcedureDetail \| null` | -| `db.listFunctions()` | `FunctionSummary[]` | -| `db.describeFunction(name, schema?)` | `FunctionDetail \| null` | -| `db.listTypes()` | `TypeSummary[]` | -| `db.describeType(name, schema?)` | `TypeDetail \| null` | -| `db.listIndexes()` | `IndexSummary[]` | -| `db.listForeignKeys()` | `ForeignKeySummary[]` | - -### ctx.noorm.lock — Lock Management - - -#### lock.acquire(options?) - -Acquire a database lock. - -```typescript -const lock = await ctx.noorm.lock.acquire({ timeout: 60000 }); +await ctx.noorm.run.dir('db/sql/09_seeds'); ``` +| Option | Type | Default | Applies to | Effect | +|---|---|---|---|---| +| `force` | `boolean` | `false` | all | Re-run files even when unchanged | +| `dryRun` | `boolean` | `false` | all | Render to disk, execute nothing | +| `abortOnError` | `boolean` | `true` | `RunOptions` | Stop at the first failure | +| `concurrency` | `number` | `1` | `RunOptions` | Files in parallel. `1` keeps DDL sequential | +| `preview` | `boolean` | `false` | `RunOptions` | Render without executing or writing history | +| `output` | `string \| null` | `null` | `RunOptions` | File that receives preview output | -#### lock.release() - -Release the current lock. +`build()` takes `BuildOptions` (`force`, `dryRun`). The other run methods take `RunOptions` (all six). -```typescript -await ctx.noorm.lock.release(); -``` +#### Run result shape -#### lock.status() - -Get current lock status. - -```typescript -const status = await ctx.noorm.lock.status(); -if (status.isLocked) { - console.log(`Locked by ${status.lock.lockedBy}`); -} -``` - - -#### lock.withLock(fn, options?) - -Execute an operation with automatic lock acquisition and release. -```typescript -await ctx.noorm.lock.withLock(async () => { - await ctx.noorm.run.build(); - await ctx.noorm.changes.ff(); +`FileResult` is what `run.file` returns and what each entry of `BatchResult.files` holds: + +| Field | Type | Set when | +|---|---|---| +| `filepath` | `string` | always | +| `checksum` | `string` | always; empty when the file could not be read | +| `status` | `'success' \| 'failed' \| 'skipped'` | always | +| `error` | `string` | `status` is `'failed'` | +| `skipReason` | `'unchanged' \| 'already-run'` | `status` is `'skipped'` | +| `durationMs` | `number` | executed and failed files | +| `renderedSql` | `string` | preview mode | +| `outputPath` | `string` | a dry run that wrote rendered SQL to disk | + +`BatchResult`: + +| Field | Type | Meaning | +|---|---|---| +| `status` | `'success' \| 'failed' \| 'partial'` | Overall outcome | +| `files` | `FileResult[]` | One per file | +| `filesRun`, `filesSkipped`, `filesFailed` | `number` | Counts | +| `durationMs` | `number` | Total time | +| `changeId` | `number` | Tracking-table id, when recorded | +| `error` | `string` | The batch failed before any file ran | +| `unmatchedInclude` | `string[]` | `build()` only: `include` entries that matched no file | +| `unmatchedExclude` | `string[]` | `build()` only: `exclude` entries that matched no file | + +A build over zero files still succeeds, so `unmatchedInclude` is how a mistyped pattern shows up. A mistyped `exclude` fences off nothing and the held-back files run, so check `unmatchedExclude` too. + + +### ctx.noorm.changes + + +| Method | Returns | Gate | Notes | +|---|---|---|---| +| `create(options)` | `Promise` | none | Creates `-/` with `change/`, `revert/`, `changelog.md`, and a `001_.sql` stub in each folder. Throws `Change already exists: ` | +| `addFile(change, folder, options)` | `Promise` | none | Adds the next numbered file to `'change'` or `'revert'` | +| `removeFile(change, folder, filename)` | `Promise` | none | | +| `renameFile(change, folder, oldFilename, newDescription)` | `Promise` | none | Keeps the numeric prefix | +| `reorderFiles(change, folder, newOrder)` | `Promise` | none | Renumbers by the order of filenames given | +| `delete(change)` | `Promise` | `change:rm` | Removes the folder from disk | +| `discover()` | `Promise` | none | Every parseable change. Folders that fail to parse are dropped and reported as `error` events | +| `parse(name)` | `Promise` | none | Throws `ChangeNotFoundError`, `ChangeValidationError`, or `ManifestReferenceError` | +| `validate(change)` | `void` | none | Throws `ChangeValidationError` on duplicate filenames in a folder | +| `apply(name, options?)` | `Promise` | `change:run` | An applied, unchanged change returns `success` with no files | +| `revert(name, options?)` | `Promise` | `change:revert` | Throws `ChangeNotAppliedError` for a never-applied change | +| `ff(options?)` | `Promise` | `change:ff` | Applies every pending change in name order | +| `next(count = 1, options?)` | `Promise` | `change:run` | Applies the next `count` pending changes | +| `rewind(target, options?)` | `Promise` | `change:revert` | Reverts newest first. `target` is a change name (reverts it and everything after) or a number of changes | +| `status()` | `Promise` | none | Every change, merged from disk and database, sorted by name | +| `pending()` | `Promise` | none | The items a forward run would pick up | +| `history(limit?)` | `Promise` | none | Execution records | +| `historyForChange(name, limit?)` | `Promise` | none | One change's records, newest first | +| `getFileHistory(operationId)` | `Promise` | none | Per-file records behind one history `id` | + +`apply` and `revert` take the config's lock for the length of each change, with reason `Change: ` or `Revert: `. A lock held by another identity throws `LockAcquireError`, and an empty change throws `ChangeValidationError`, out of `ff`, `next`, and `rewind` as well. Per-file failures come back on the result as `status: 'failed'` or `'partial'`. + +```typescript +const change = await ctx.noorm.changes.create({ description: 'order-status' }); +// db/changes/2026-10-20-order-status/{change,revert}/001_order-status.sql + +await ctx.noorm.changes.addFile(change, 'change', { + name: 'rebuild-views', + type: 'txt', + paths: ['02_views/01_open_orders.sql'], }); -``` - -#### lock.forceRelease() - -Force release any database lock regardless of ownership. Use when a lock is stuck due to a crashed process or stale session. +const dry = await ctx.noorm.changes.ff({ dryRun: true }); +const result = await ctx.noorm.changes.ff(); -```typescript -const { released, holder } = await ctx.noorm.lock.forceRelease(); -if (released) { - console.log(`Stale lock cleared (was held by ${holder})`); +if (result.status !== 'success') { + console.error(result.error ?? result.changes.filter((c) => c.status === 'failed')); } ``` -**Returns:** `Promise` — `{ released: boolean; holder: string | null }`. `holder` is `null` when there was nothing to release. The result object is always truthy, so branch on `released`, not on the return value. - -This is the one lock method behind the access policy. Breaking someone else's lock interrupts their in-flight migration, so `viewer` is denied outright and `operator`/`admin` need `yes: true` on the context. Denial throws `ProtectedConfigError`. - +A `txt` file is a manifest: one path per line, relative to `paths.sql`, that runs those `db/sql/` files in place. [Forward and revert](/guide/changes/forward-revert) covers when to use one. -### ctx.noorm.templates — Template Operations +| Option | Type | Default | Applies to | Effect | +|---|---|---|---|---| +| `force` | `boolean` | `false` | all | Skip the already-applied check | +| `dryRun` | `boolean` | `false` | all | Render to `tmp/`, leave the tracking tables untouched | +| `preview` | `boolean` | `false` | all | Emit rendered SQL without writing | +| `output` | `string \| null` | `null` | all | File that receives preview output | +| `abortOnError` | `boolean` | `true` | `ff`, `next`, `rewind` | Stop at the first failure | +`ChangeOptions` holds the first four. `BatchChangeOptions` adds `abortOnError` and is not exported. `AddFileOptions` is `{ name: string; type: 'sql' \| 'txt'; paths?: string[]; content?: string }`; `CreateChangeOptions` is `{ description: string; date?: Date }`. -#### templates.render(filepath) +A change moves between five statuses. `pending()` returns `pending`, `reverted`, and `stale`, plus a folder that never ran and fails to parse: -Render a template file without executing. - -```typescript -const result = await ctx.noorm.templates.render('sql/001_users.sql.tmpl'); -console.log(result.sql); +```mermaid +stateDiagram-v2 + [*] --> pending: folder on disk, no record + pending --> success: apply, ff, next + pending --> failed: a file errors + failed --> success: apply + success --> reverted: revert, rewind + reverted --> success: apply, ff, next + success --> stale: db.teardown, db.reset + stale --> success: apply, ff, next ``` +On PostgreSQL a failed change rolls back its history rows, so it keeps its previous status. On mysql, mssql, and sqlite the failure is recorded as `failed`. -### ctx.noorm.transfer — Transfer Operations +An unparseable folder, such as a manifest that names a `db/sql/` file that no longer exists, comes back from `status()` and `pending()` as `status: 'failed'` and `isNew: true`, with `errorMessage` holding the parse error. `ff` and `next` fail at that change when they would run it, instead of skipping it. If the change already ran, it keeps its status and the batch adds a warning. A folder with neither `change/` nor `revert/` is not a change and is ignored. +`ChangeListItem`: -#### transfer.to(destConfig, options?) +| Field | Type | Meaning | +|---|---|---| +| `name` | `string` | Folder name | +| `path`, `date`, `description` | `string`, `Date \| null`, `string` | From disk. Absent when `orphaned` | +| `changeFiles`, `revertFiles`, `hasChangelog` | `ChangeFile[]`, `ChangeFile[]`, `boolean` | From disk. Absent when `orphaned` or unparseable | +| `status` | `'pending' \| 'success' \| 'failed' \| 'reverted' \| 'stale'` | Latest recorded status | +| `appliedAt`, `appliedBy`, `revertedAt` | `Date \| null`, `string \| null`, `Date \| null` | From the tracking tables | +| `errorMessage` | `string \| null` | Failure message or parse error | +| `appliedHistoryId` | `number \| null` | History id behind `appliedAt`; the true apply order | +| `isNew` | `boolean` | On disk with no record | +| `orphaned` | `boolean` | Recorded in the database, folder gone | -Transfer data from this context's database to a destination config. Both contexts must be connected. Throws on failure. +`ChangeResult`: `name`, `direction` (`'change' \| 'revert'`), `status`, `files` (each with `filepath`, `checksum`, `status`, `skipReason?`, `durationMs?`, `error?`, `renderedSql?`), `durationMs`, `error?`, `operationId?`. -```typescript -const source = await createContext({ config: 'staging' }); -const dest = await createContext({ config: 'dev' }); -await source.connect(); -await dest.connect(); +`BatchChangeResult`: `status` (`'success' \| 'failed' \| 'partial'`), `changes` (`ChangeResult[]`), `executed`, `skipped`, `failed`, `durationMs`, `error?`, `warnings?`. `error` is set when the batch failed before any change could report, such as a `rewind` name that matches no applied change. `warnings` carries a missing changes directory and applied changes whose files can no longer be read. -const result = await source.noorm.transfer.to(dest.noorm.config, { - tables: ['users', 'posts'], - onConflict: 'skip', - batchSize: 5000, -}); - -console.log(`Transferred ${result.totalRows} rows (${result.status})`); - -await source.disconnect(); -await dest.disconnect(); -``` +`ChangeHistoryRecord`: `id`, `name`, `direction`, `status`, `executedAt`, `executedBy`, `durationMs`, `errorMessage`, `checksum`. `FileHistoryRecord`: `id`, `changeId`, `filepath`, `fileType`, `checksum`, `status`, `skipReason`, `errorMessage`, `durationMs`; the type is not exported. -**Options (`TransferOptions`):** -| Option | Type | Default | Description | -| -------------------- | ------------------ | -------- | -------------------------------------------- | -| `tables` | `string[]` | all | Tables to transfer. Empty = all user tables. | -| `onConflict` | `ConflictStrategy` | `'fail'` | How to handle primary key conflicts. | -| `batchSize` | `number` | `1000` | Rows per batch for cross-server transfers. | -| `disableForeignKeys` | `boolean` | `true` | Disable FK checks during transfer. | -| `preserveIdentity` | `boolean` | `true` | Preserve identity/auto-increment values. | -| `truncateFirst` | `boolean` | `false` | Truncate destination tables before transfer. | -| `dryRun` | `boolean` | `false` | Validate only, don't execute. | -| `exportPath` | `string` | — | Export to .dt file instead of DB insert. | -| `passphrase` | `string` | — | Passphrase for .dtzx export encryption. | +### ctx.noorm.db -**Returns:** `Promise` — `{ status, tables, totalRows, durationMs, fkChecksRestored }`. Check `fkChecksRestored`: it is the only signal that referential integrity may still be off, because `status` does not flip when the FK re-enable fails. -The policy gate runs against the **destination** config, not the source, since the destination is the write target. `transfer.plan()` is gated separately on `transfer:plan`, which `operator` and `admin` both hold outright, so planning stays available to a role that may not execute the transfer. +Every `db` method needs a connection. The explore methods are not gated. +| Method | Returns | +|---|---| +| `overview()` | `ExploreOverview`: counts for `tables`, `views`, `procedures`, `functions`, `types`, `indexes`, `foreignKeys`, `triggers`, `locks`, `connections` | +| `listTables()` | `TableSummary[]`: `name`, `schema?`, `columnCount`, `rowCountEstimate?` | +| `describeTable(name, schema?)` | `TableDetail \| null`: `columns`, `indexes`, `foreignKeys`, and the summary fields | +| `listViews()` / `describeView(name, schema?)` | `ViewSummary[]` / `ViewDetail \| null` | +| `listProcedures()` / `describeProcedure(name, schema?)` | `ProcedureSummary[]` / `ProcedureDetail \| null` | +| `listFunctions()` / `describeFunction(name, schema?)` | `FunctionSummary[]` / `FunctionDetail \| null` | +| `listTypes()` / `describeType(name, schema?)` | `TypeSummary[]` / `TypeDetail \| null` | +| `listIndexes()` | `IndexSummary[]` | +| `listForeignKeys()` | `ForeignKeySummary[]` | +| `previewTeardown()` | `TeardownPreview`: `toDrop`, `toPreserve`, `statements` | -#### transfer.plan(destConfig, options?) - -Generate a transfer plan without executing. Inspects both databases and returns table ordering, row estimates, and warnings. Throws on failure. +Each `describe*` returns `null` when the object does not exist. `ColumnDetail` carries `name`, `dataType`, `isNullable`, `defaultValue?`, `isPrimaryKey`, `ordinalPosition`. ```typescript -const plan = await source.noorm.transfer.plan(dest.noorm.config); -console.log(`${plan.estimatedRows} rows across ${plan.tables.length} tables`); -for (const warning of plan.warnings) { - console.warn(warning); +const detail = await ctx.noorm.db.describeTable('order_line'); + +for (const col of detail?.columns ?? []) { + console.log(`${col.name}: ${col.dataType}`); } ``` +The destructive methods: -### ctx.noorm.dt — DT File Operations - - -#### dt.exportTable(tableName, filepath, options?) - -Export a table to a .dt file. The file extension determines the format: `.dt` (plain), `.dtz` (gzipped), `.dtzx` (encrypted). Throws on failure. +| Method | Returns | Gate | What it does | +|---|---|---|---| +| `truncate(options?)` | `Promise` | `db:truncate` | Wipes data, keeps the schema. Disables foreign keys, deletes or truncates each target, re-enables them | +| `teardown(options?)` | `Promise` | `db:teardown` | Drops every object except noorm's own tables (change tracking, locks, vault, identities), and marks applied changes `stale` | +| `reset()` | `Promise` | `db:reset` | Teardown, then `run.build({ force: true })`, which also passes the `run:build` gate | ```typescript -const result = await ctx.noorm.dt.exportTable('users', './exports/users.dtz'); -console.log(`Exported ${result.rowsWritten} rows (${result.bytesWritten} bytes)`); +await ctx.noorm.db.truncate({ preserve: ['party_type', 'product'] }); +await ctx.noorm.db.truncate({ only: ['order_line', 'sales_order'] }); -// Encrypted export -const encrypted = await ctx.noorm.dt.exportTable('users', './exports/users.dtzx', { - passphrase: 'my-secret', -}); +const plan = await ctx.noorm.db.teardown({ dryRun: true }); +await ctx.noorm.db.teardown({ preserveSchemas: ['app_private'] }); ``` -**Options (`ExportOptions`):** - -| Option | Type | Description | -| ------------ | -------- | ------------------------------------------------- | -| `passphrase` | `string` | Passphrase for .dtzx encryption. | -| `schema` | `string` | Schema/namespace (e.g., 'public' for PostgreSQL). | -| `batchSize` | `number` | Rows per batch. Default: 1000. | +`TruncateOptions`: +| Option | Type | Default | Effect | +|---|---|---|---| +| `preserve` | `string[]` | `settings.teardown.preserveTables` | Tables to leave alone | +| `only` | `string[]` | all | Truncate only these. The inverse of `preserve` | +| `restartIdentity` | `boolean` | `true` | Restart identity and auto-increment sequences | +| `dryRun` | `boolean` | `false` | Return the SQL in `statements`, execute nothing | +| `policy` | `DbPolicyContext` | none | Internal gate context; the SDK already gates, so leave it unset | -#### dt.importFile(filepath, options?) +`TeardownOptions` is not exported. From the SDK, two fields take effect and the rest come from settings or are dropped: -Import a .dt file into the connected database. Throws on failure. +| Field | From the SDK | +|---|---| +| `dryRun` | read from your options | +| `preserveSchemas` | read from your options. Leaves whole schemas alone, which `preserveTables` cannot express: excluding `app_private.secrets` by name would also spare `public.secrets` | +| `preserveTables`, `postScript` | filled from `settings.teardown`; a value you pass is ignored | +| `configName`, `executedBy` | set by the SDK, so teardown marks applied changes `stale` and records a reset | +| `keepViews`, `keepFunctions`, `keepProcedures`, `keepTypes`, `policy` | ignored | -```typescript -const result = await ctx.noorm.dt.importFile('./exports/users.dtz', { - onConflict: 'skip', -}); -console.log(`Imported ${result.rowsImported} rows, skipped ${result.rowsSkipped}`); -``` +`reset()` takes no options and ignores `settings.teardown.preserveTables`: it rebuilds from `db/sql/`, so a surviving table would collide with its `CREATE TABLE`. -**Options (`ImportOptions`):** +`TruncateResult`: `truncated`, `preserved`, `statements`, `durationMs`. `TeardownResult`: `dropped` (`tables`, `views`, `functions`, `procedures`, `types`, `foreignKeys`), `preserved`, `statements`, `durationMs`, and when they apply `postScriptResult`, `staleCount`, `resetRecordId`. `TeardownPreview.toDrop` groups names by the same six kinds; there is no flat list. -| Option | Type | Default | Description | -| ------------ | ------------------ | -------- | ------------------------------------ | -| `passphrase` | `string` | — | Passphrase for .dtzx decryption. | -| `batchSize` | `number` | `1000` | Rows per batch. | -| `onConflict` | `ConflictStrategy` | `'fail'` | Conflict strategy. | -| `truncate` | `boolean` | `false` | Truncate target table before import. | +Drop order is foreign keys, procedures, functions, views, tables, then types, and mssql truncate disables constraints one table at a time. [Teardown internals](/dev/teardown#drop-order) gives the reasons, and [the mssql strategy](/dev/teardown#mssql-truncate-strategy) covers truncate. -`importFile` writes to the database, so it is gated on `db:reset`, which requires confirmation for `operator`. `exportTable` only reads and is not gated. +### ctx.noorm.lock -### ctx.noorm.changes — History +One lock row per config serializes schema operations across every noorm user of that database. A lock expires on its own. -#### changes.history(limit?) +| Method | Returns | Gate | Notes | +|---|---|---|---| +| `acquire(options?)` | `Promise` | none | The same identity can acquire again and extends the lock. Another holder throws `LockAcquireError` unless `wait` is set | +| `release()` | `Promise` | none | Throws when no lock exists or another identity holds it | +| `status()` | `Promise` | none | `{ isLocked, lock }`. Clears an expired lock first | +| `withLock(fn, options?)` | `Promise` | none | Acquires, runs `fn`, releases in a `finally`. A failed release emits an `error` event instead of throwing | +| `forceRelease()` | `Promise` | `lock:force` | Deletes the lock whoever holds it. `{ released, holder }`; `holder` is `null` when nothing was held | -Get execution history. +`ForceReleaseResult` is not exported. The result object is always truthy, so branch on `released`. ```typescript -const history = await ctx.noorm.changes.history(10); -for (const record of history) { - console.log(`${record.name}: ${record.status} at ${record.executedAt}`); -} -``` - -**Returns:** `Promise` — one record per operation, each with `id`, `name`, `direction`, `status`, `executedAt`, `executedBy`, `durationMs`, `errorMessage`, and `checksum`. - - -#### changes.historyForChange(name, limit?) - -Get execution history for one change, most recent first. +await ctx.noorm.lock.withLock( + () => ctx.noorm.run.dir('db/sql/09_seeds'), + { timeout: 120_000, wait: true, reason: 'seed products' }, +); -```typescript -const records = await ctx.noorm.changes.historyForChange('2024-01-15-add-users'); +const { released, holder } = await ctx.noorm.lock.forceRelease(); ``` -**Returns:** `Promise`. - - -#### changes.getFileHistory(operationId) - -Drill into the per-file records behind a single operation. Pass the `id` from a `ChangeHistoryRecord`. - -```typescript -const records = await ctx.noorm.changes.historyForChange('2024-01-15-add-users'); -const files = await ctx.noorm.changes.getFileHistory(records[0].id); - -for (const file of files) { - console.log(`${file.filepath}: ${file.status}`); -} -``` +| `LockOptions` | Type | Default | Meaning | +|---|---|---|---| +| `timeout` | `number` | `300000` | Milliseconds until the lock expires | +| `wait` | `boolean` | `false` | Poll until the lock is free | +| `waitTimeout` | `number` | `30000` | Most milliseconds to wait. Used only with `wait` | +| `pollInterval` | `number` | `1000` | Milliseconds between polls. Used only with `wait` | +| `reason` | `string` | none | Shown to whoever the lock blocks | +| `dialect` | `Dialect` | the config's | Set by the SDK; a value you pass is replaced | -**Returns:** `Promise` — `id`, `changeId`, `filepath`, `fileType`, `checksum`, `status`, `skipReason`, and the error message when the file failed. +`Lock` is `{ lockedBy, lockedAt, expiresAt, reason? }`. `release()` throws `LockNotFoundError` or `LockOwnershipError`, neither exported; match on `err.name`. -### ctx.noorm.secrets — Secrets +### ctx.noorm.secrets -#### secrets.get(key) +Config-scoped secrets live in local encrypted state, not the database. No connection is needed. Global secrets are not reachable here. -Get a config-scoped secret. Synchronous, and needs no connection: these live in local encrypted state, not the database. +| Method | Returns | Gate | Notes | +|---|---|---|---| +| `get(key)` | `string \| undefined` | `secret:read` | Synchronous. `undefined`, not `null`, for an unset key | +| `list()` | `string[]` | `secret:read` | Synchronous. Key names only; values are never returned | +| `set(key, value)` | `Promise` | `secret:write` | | +| `delete(key)` | `Promise` | `secret:write` | | ```typescript -const apiKey = ctx.noorm.secrets.get('API_KEY'); +await ctx.noorm.secrets.set('PAYMENT_GATEWAY_KEY', process.env.GATEWAY_KEY!); +const key = ctx.noorm.secrets.get('PAYMENT_GATEWAY_KEY'); ``` -**Returns:** `string | undefined` — not a promise, and `undefined` rather than `null` when the key is unset. - -#### secrets.list() - -List the config's secret key names. Values are never returned, so a caller that only needs to know which secrets exist never handles them. - -```typescript -const keys = ctx.noorm.secrets.list(); -``` +### ctx.noorm.vault -**Returns:** `string[]` — synchronous. +The vault holds team secrets in the database, encrypted with identity keypairs. `init`, `status`, `set`, `get`, `getAll`, `propagate`, and `copy` need a cryptographic identity at `~/.noorm/identity.json` or in `NOORM_IDENTITY_*`. `privateKey` is the identity's private key as hex, the same value as `NOORM_IDENTITY_PRIVATE_KEY`. -#### secrets.set(key, value) / secrets.delete(key) +| Method | Returns | Gate | Notes | +|---|---|---|---| +| `init()` | `Promise` | `vault:write` | The vault key on first init, `null` when already initialized. Safe to call at startup | +| `status()` | `Promise` | none | `isInitialized`, `hasAccess`, `secretCount`, `usersWithAccess`, `usersWithoutAccess` | +| `set(key, value, privateKey)` | `Promise` | `vault:write` | Throws `VaultAccessError` when the key yields no vault key. The two causes, never propagated and wrong key, are indistinguishable by design | +| `get(key, privateKey)` | `Promise` | `vault:read` | `null` when absent or without access | +| `getAll(privateKey)` | `Promise>` | `vault:read` | `{}` without access | +| `list()` | `Promise` | `vault:read` | Key names; no private key needed | +| `exists(key)` | `Promise` | `vault:read` | Does not decrypt | +| `delete(key)` | `Promise` | `vault:write` | | +| `propagate(privateKey)` | `Promise` | `vault:propagate` | Grants the vault key to identities without it. Run it after adding a teammate | +| `copy(destConfig, keys, privateKey, options?)` | `Promise` | `vault:read` source, `vault:write` destination | `keys` is a list or `'all'`. Initializes the destination vault when needed | -Write and remove config-scoped secrets. Both are async and both need `secret:write`, which is `confirm` for `operator`. +`vault:initialized` fires on first init only. Neither read throws on missing access: `get` returns `null` and `getAll` returns `{}`. ```typescript -await ctx.noorm.secrets.set('API_KEY', 'sk-live-...'); -await ctx.noorm.secrets.delete('OLD_KEY'); -``` - -**Returns:** `Promise` for both. - - -### ctx.noorm.vault — Vault (Encrypted Team Secrets) - -Database-stored encrypted secrets shared across team members. Unlike config-scoped secrets, vault secrets live in the database and are encrypted with identity keypairs. - - -#### vault.init() - -Initialize the vault for this database. Creates the vault key and stores it encrypted for the current identity. +const privateKey = process.env.NOORM_IDENTITY_PRIVATE_KEY!; -Idempotent. Calling `init()` a second time against an already-initialized vault returns `null` — no state change, no error. Callers can `init()` defensively at startup without special-casing an error string. Real failures (DB errors, encryption errors) throw. +await ctx.noorm.vault.init(); +await ctx.noorm.vault.set('PAYMENT_GATEWAY_KEY', 'sk-live-...', privateKey); -```typescript -const vaultKey = await ctx.noorm.vault.init(); - -if (vaultKey) { - // First-time init — seed initial team secrets, etc. +const grant = await ctx.noorm.vault.propagate(privateKey); +if (grant.failed.length > 0) { + throw new Error(`Vault access not granted to ${grant.failed.map((f) => f.email).join(', ')}`); } -else { - // Already initialized — proceed normally with vault.get / vault.set. -} -``` - -The `vault:initialized` observer event fires only on first init, never on repeat calls. Cross-reference with `vault.status()` if you need to distinguish "just initialized" from "was already there" alongside other status fields. - -**Returns:** `Promise` — the vault key buffer on first init, `null` if already initialized. Throws on failure. - - -#### vault.status() -Get vault status for the current identity. Useful alongside `vault.init()` when callers need to know whether a vault existed before they called `init()`: a `null` from `init()` plus `status.isInitialized === true` confirms idempotent no-op. - -```typescript -const status = await ctx.noorm.vault.status(); -console.log(`Initialized: ${status.isInitialized}, Has access: ${status.hasAccess}`); +await ctx.noorm.vault.copy(stagingConfig, ['PAYMENT_GATEWAY_KEY'], privateKey, { dryRun: true }); ``` -**Returns:** `Promise` - - -#### vault.set(key, value, privateKey) +| Result | Fields | +|---|---| +| `VaultSecret` | `key`, `value`, `setBy`, `createdAt`, `updatedAt` | +| `VaultPropagationResult` | `propagatedTo` (identity hashes), `alreadyHadAccess` (count), `failed` (`identityHash`, `email`, `error`) | +| `VaultCopyResult` | `copied`, `skipped` (exists and not forced), `errors` (`{ key, error }`); all key lists, not counts | -Set a vault secret. Requires the caller's private key for vault key decryption. Throws on failure. +`VaultCopyOptions` is `{ force?: boolean; dryRun?: boolean; channel?: Channel }`. `force` overwrites existing destination secrets. `dryRun` runs the full preflight and reports what would happen without writing. The SDK sets `channel` from the context. -```typescript -await ctx.noorm.vault.set('API_KEY', 'sk-live-...', privateKey); -``` -**Returns:** `Promise`. Throws `VaultAccessError` when the private key yields no usable vault key: either the vault was never propagated to this identity, or the key does not match. The two causes are deliberately indistinguishable. +### ctx.noorm.templates -#### vault.get(key, privateKey) +`render(filepath)` renders a `.sql.tmpl` file and returns it without executing. No connection is needed, but the vault tier of `$.secrets` resolves only when the context is connected. -Get a single vault secret by key. +| Method | Returns | Gate | +|---|---|---| +| `render(filepath)` | `Promise` | `run:file`, deny only, throws `ProtectedConfigError` | ```typescript -const value = await ctx.noorm.vault.get('API_KEY', privateKey); +const { sql, isTemplate, durationMs } = await ctx.noorm.templates.render( + 'db/sql/09_seeds/01_products.sql.tmpl', +); ``` -**Returns:** `Promise` — the decrypted value, or `null` if not found or no access. +`TemplateResult` is `{ sql: string; isTemplate: boolean; durationMs?: number }`. A plain `.sql` file comes back unchanged with `isTemplate: false`. Rendering resolves every secret tier into `sql` and runs the template's helpers, so the output is plaintext secrets. -#### vault.getAll(privateKey) +### ctx.noorm.transfer -Get all vault secrets as a key-value map with metadata. - -```typescript -const all = await ctx.noorm.vault.getAll(privateKey); -for (const [key, secret] of Object.entries(all)) { - console.log(`${key}: set by ${secret.setBy}`); -} -``` - -**Returns:** `Promise>` — an empty object when the caller has no vault access, mirroring `get()` returning `null`. Neither read throws on missing access. +Both methods take the destination `Config` and open their own connections, so neither context needs `connect()`. Source and destination must be postgres, mysql, or mssql; sqlite throws. -#### vault.list() - -List all vault secret keys without decrypting values. Does not require a private key. +| Method | Returns | Gate | Notes | +|---|---|---|---| +| `to(destConfig, options?)` | `Promise` | `db:reset` on the destination | Throws on failure. A dry run is gated the same way | +| `plan(destConfig, options?)` | `Promise` | `transfer:plan` on the destination | Table order, row estimates, warnings. Throws on failure | ```typescript -const keys = await ctx.noorm.vault.list(); -console.log('Available secrets:', keys.join(', ')); -``` - -**Returns:** `Promise` +const staging = await createContext({ config: 'staging' }); +const qa = await createContext({ config: 'qa' }); +const result = await staging.noorm.transfer.to(qa.noorm.config, { + tables: ['party', 'product', 'sales_order', 'order_line'], + onConflict: 'skip', +}); -#### vault.delete(key) - -Delete a vault secret. Throws on failure. - -```typescript -const deleted = await ctx.noorm.vault.delete('OLD_KEY'); -if (deleted) { - console.log('Secret removed'); +if (!result.fkChecksRestored) { + console.error('Foreign key checks are still off on qa'); } ``` -**Returns:** `Promise` — throws on failure. - - -#### vault.exists(key) - -Check if a vault secret exists without decrypting. - -```typescript -const exists = await ctx.noorm.vault.exists('API_KEY'); -``` - -**Returns:** `Promise` +| `TransferOptions` | Type | Default | Effect | +|---|---|---|---| +| `tables` | `string[]` | all user tables | Tables to copy | +| `onConflict` | `'fail' \| 'skip' \| 'update' \| 'replace'` | `'fail'` | Primary key conflict strategy | +| `batchSize` | `number` | `1000` | Rows per batch for cross-server transfers | +| `disableForeignKeys` | `boolean` | `true` | Disable FK checks during the copy | +| `preserveIdentity` | `boolean` | `true` | Keep identity and auto-increment values | +| `truncateFirst` | `boolean` | `false` | Truncate destination tables first | +| `dryRun` | `boolean` | `false` | Validate only | +| `exportPath` | `string` | none | Write a `.dt` file instead of inserting | +| `passphrase` | `string` | none | Encrypts a `.dtzx` export | +| `channel` | `Channel` | the context's | Set by the SDK | +`TransferResult`: `status` (`'success' \| 'partial' \| 'failed'`), `tables` (`table`, `status`, `rowsTransferred`, `rowsSkipped`, `durationMs`, `error?`), `totalRows`, `durationMs`, `fkChecksRestored`. Check `fkChecksRestored`: `status` does not change when re-enabling foreign keys fails, so it is the only signal that referential integrity may still be off. -#### vault.propagate(privateKey) +`TransferPlan`: `tables` in dependency order (`name`, `schema?`, `rowCount`, `hasIdentity`, `identityColumn?`, `primaryKey`, `columns`, `dependsOn`, `columnTypes?`), `sameServer`, `estimatedRows`, `warnings`, `crossDialect`, `sourceDialect`, `destinationDialect`. -Propagate vault key to all team identities that don't yet have access. Run after adding new team members. -```typescript -const result = await ctx.noorm.vault.propagate(privateKey); -console.log(`Propagated to ${result.propagatedTo.length} new users`); -``` +### ctx.noorm.dt -**Returns:** `Promise` — `{ propagatedTo, alreadyHadAccess, failed }`. A non-empty `failed` means the operation partially succeeded: a teammate believes they have access and does not. Treat it as an error, not a warning. +A `.dt` file holds one table in noorm's portable format. The extension picks the form: `.dt` plain, `.dtz` compressed, `.dtzx` encrypted with a passphrase. -#### vault.copy(destConfig, keys, privateKey, options?) +| Method | Returns | Gate | +|---|---|---| +| `exportTable(tableName, filepath, options?)` | `Promise<{ rowsWritten, bytesWritten }>` | none | +| `importFile(filepath, options?)` | `Promise<{ rowsImported, rowsSkipped }>` | `db:reset` | -Copy vault secrets to another config's database. Useful for seeding a new environment with secrets from an existing one. Throws on failure. - -`keys` takes either an explicit list or the literal `'all'`. +Both throw on failure. ```typescript -const result = await ctx.noorm.vault.copy( - destConfig, - ['API_KEY', 'DB_TOKEN'], - privateKey, -); -console.log(`Copied ${result.copied.length} secrets`); - -// Or copy everything. -await ctx.noorm.vault.copy(destConfig, 'all', privateKey); +await ctx.noorm.dt.exportTable('product', './exports/product.dtz'); +await ctx.noorm.dt.exportTable('product', './exports/product.dtzx', { passphrase }); +await ctx.noorm.dt.importFile('./exports/product.dtz', { onConflict: 'skip' }); ``` -**Options (`VaultCopyOptions`):** `force` (overwrite existing destination secrets, default `false`) and `dryRun` (run the full preflight and report what would happen without writing). - -**Returns:** `Promise` — `{ copied: string[]; skipped: string[]; errors: Array<{ key, error }> }`. All three are key lists, not counts. `skipped` holds keys that already exist on the destination and were not forced. This is the one vault method that checks the access policy on both the source and the destination config. +| Option | Applies to | Type | Default | Effect | +|---|---|---|---|---| +| `passphrase` | both | `string` | none | `.dtzx` encryption or decryption | +| `batchSize` | both | `number` | `1000` | Rows per batch | +| `schema` | export | `string` | none | Schema or namespace, such as `public` | +| `onConflict` | import | `ConflictStrategy` | `'fail'` | Primary key conflict strategy | +| `truncate` | import | `boolean` | `false` | Truncate the target table first | -### ctx.noorm.utils — Utilities +### ctx.noorm.utils -#### utils.testConnection() - -Tests if the connection can be established without actually connecting. +| Method | Returns | Notes | +|---|---|---| +| `checksum(filepath)` | `Promise` | SHA-256 of the file. Works offline | +| `testConnection()` | `Promise<{ ok: boolean; error?: string }>` | Probes the config without connecting the context. Reports failure as data, not an exception | ```typescript -const result = await ctx.noorm.utils.testConnection(); -if (!result.ok) { - console.error('Connection failed:', result.error); -} +const probe = await ctx.noorm.utils.testConnection(); +if (!probe.ok) console.error(probe.error); ``` -#### utils.checksum(filepath) - -Compute SHA-256 checksum for a file. - -```typescript -const checksum = await ctx.noorm.utils.checksum('sql/001_users.sql'); -``` - +## Events -### Event Subscriptions -Subscribe to core events via the process-global `noormObserver` bus: +Core operations emit events on `noormObserver`, an `ObserverEngine`. It is one process-global bus, not one per context, so a process that creates several contexts sees all of their events on the same stream. ```typescript import { noormObserver } from '@noormdev/sdk'; -noormObserver.on('file:after', (event) => { - console.log(`Executed ${event.filepath} in ${event.durationMs}ms`); -}); - -noormObserver.on('change:complete', (event) => { - console.log(`Change ${event.name}: ${event.status}`); +const stop = noormObserver.on('file:after', (event) => { + console.log(`${event.filepath}: ${event.status} in ${event.durationMs}ms`); }); -``` - -`noormObserver` is a process-global singleton, not a per-context bus. A process that calls `createContext()` more than once (a server juggling several tenant databases, say) sees every context's events on this one stream. Config-scoped events carry `configName` so you can filter, but not every event does: `file:after` and `change:complete` do not, while `file:before`, `run:file`, `lock:*`, `secret:*`, and `db:*` do. Check the payload type before relying on it. - - -## Environment Variables - -The SDK supports environment variable overrides for CI/CD. +noormObserver.on(/^change:/, ({ event, data }) => console.log(event, data)); -### Override Stored Configs - -```bash -export NOORM_CONNECTION_HOST=db.ci.internal -export NOORM_CONFIG=staging +stop(); ``` -Priority (highest to lowest): -1. `NOORM_*` env vars -2. Stored config -3. Stage defaults -4. Defaults - - -### Env-Only Mode +`on` returns a function that removes the listener, and a regular expression subscribes to every matching event. Start the process with `NOORM_DEBUG=1` to log each event name to stderr. -Run without stored configs by setting minimum required env vars: +Context-scoped events carry `configName`, so filter on it when you run more than one context. Several do not, so check the payload type first: -```bash -export NOORM_CONNECTION_DIALECT=postgres -export NOORM_CONNECTION_DATABASE=mydb -export NOORM_CONNECTION_HOST=localhost -export NOORM_CONNECTION_USER=postgres -export NOORM_CONNECTION_PASSWORD=secret -``` +| Family | Events | `configName` | +|---|---|---| +| File | `file:before`, `file:after`, `file:skip`, `file:progress`, `file:dry-run` | `file:before` only | +| Build and run | `build:start`, `build:complete`, `run:file`, `run:dir`, `run:files` | `run:*` only | +| Change | `change:created`, `change:start`, `change:file`, `change:complete`, `change:skip` | no | +| Lock | `lock:acquiring`, `lock:acquired`, `lock:released`, `lock:blocked`, `lock:expired` | yes | +| Connection | `connection:open`, `connection:close`, `connection:error` | yes | +| Secret | `secret:set`, `secret:deleted` | yes | +| Teardown | `teardown:start`, `teardown:progress`, `teardown:error`, `teardown:complete` | no | +| Template | `template:render`, `template:load`, `template:helpers` | no | +| Vault | `vault:initialized`, `vault:secret:created`, `vault:secret:updated`, `vault:secret:deleted`, `vault:propagated`, `vault:copy:starting`, `vault:copy:completed` | no | +| Transfer | `transfer:planning`, `transfer:plan:ready`, `transfer:starting`, `transfer:table:before`, `transfer:table:progress`, `transfer:table:after`, `transfer:complete`, `db:dual:connecting`, `db:dual:connected`, `db:dual:disconnecting`, `db:dual:disconnected`, `db:dual:cleanup-warning` | no | +| DT | `dt:export:start`, `dt:export:progress`, `dt:export:complete`, `dt:import:start`, `dt:import:schema`, `dt:import:progress`, `dt:import:complete`, `dt:stream:start`, `dt:stream:progress`, `dt:stream:complete`, `dt:validate:result`, `dt:modify:start`, `dt:modify:progress`, `dt:modify:complete`, and per-stage `dt:export:loaded`, `dt:export:processed`, `dt:export:saved`, `dt:import:loaded`, `dt:import:processed`, `dt:import:saved` | no | +| Errors | `error`: `{ source, error, context? }` | no | -```typescript -// No config name needed—uses env vars directly -const ctx = await createContext(); -``` +`error` reports problems the SDK absorbs instead of throwing: an unparseable change folder during `discover()`, a failed lock release in `withLock`, an unwritable preview output path. `NoormEvents` also types families the CLI and TUI emit: `state`, `config`, `identity`, `app`, `router`, `version`, `logger`, `sql-terminal`, `settings`, `update`, and `init`. [Teardown events](/dev/teardown#observer-events) give the teardown payloads. -## Error Handling +## Error handling -SDK methods throw named, `instanceof`-matchable errors and let them propagate — a call either -returns its value or throws. There are no `[value, error]` tuples on the `ctx.noorm.*` surface. -Wrap a call in `attempt()` (from `@logosdx/utils`) only when you're going to do something with the -error: translate it, recover, observe, or knowingly ignore it. If you'd just re-throw it unchanged, -skip `attempt` and let it propagate. Never wrap an SDK call in try-catch. -Carve-outs that don't follow that plain throw-and-`attempt` pattern: -`ctx.noorm.utils.testConnection()` returns `{ ok, error? }` by design — a probe reporting failure as -data, not an exception. `ctx.noorm.vault.get()` and `.getAll()` return `null` and `{}` on missing -vault access rather than throwing, and `ctx.noorm.vault.init()` returns `null` when the vault already -exists. `ctx.noorm.run.*` and `ctx.noorm.changes.ff/next/rewind` report per-item failures on the -result (`result.files[].error`, `result.error`) instead of throwing, so a non-throwing call is not -proof of success; check `status`. `ctx.transaction(...)` callbacks must throw to trigger Kysely's -rollback — throwing is the rollback signal, so don't swallow it inside the callback. Raw `ctx.kysely` -queries throw whatever the driver throws. +A call either returns its value or throws a named, `instanceof`-matchable error. There are no `[value, error]` tuples on this surface. Wrap a call in `attempt()` from `@logosdx/utils` only to translate, recover, or observe the error; otherwise let it propagate. These calls report differently: -One gap in the "named errors" rule: a policy denial caught at the SDK seam throws `ProtectedConfigError`, -but a denial caught deeper, inside the runner or the transfer module, throws a plain `Error` carrying the -same message. Match on `ProtectedConfigError` where you can, and fall back to the message otherwise. +| Call | Reports | +|---|---| +| `run.*`, `changes.ff`, `changes.next`, `changes.rewind` | Per-item failures on the result (`result.files[].error`, `result.error`). A call that returns is not proof of success, so check `status` | +| `utils.testConnection()` | `{ ok, error? }`, by design | +| `vault.get`, `vault.getAll`, `vault.init` | `null`, `{}`, and `null` for missing access or an existing vault | +| `transaction(fn)` | `fn` must throw to roll back | +| `ctx.kysely` queries | Whatever the driver throws | ```typescript import { attempt } from '@logosdx/utils'; -import { - createContext, - tvp, - RequireTestError, - ProtectedConfigError, - NotConnectedError, - VaultAccessError, - ImpersonationError, - LockAcquireError, -} from '@noormdev/sdk'; - -// attempt() is used deliberately here — the caller inspects the named error and reacts to it. +import { createContext, RequireTestError, ProtectedConfigError } from '@noormdev/sdk'; + const [ctx, err] = await attempt(() => createContext({ config: 'prod', requireTest: true })); + if (err instanceof RequireTestError) { - console.error('Cannot use production config in tests'); + console.error(`${err.configName} is not a test database`); } -const [, truncateErr] = await attempt(() => ctx.noorm.db.truncate()); -if (truncateErr instanceof ProtectedConfigError) { - console.error('Denied by the config\'s access role, or needs confirmation the SDK can\'t give'); -} +const [, wipeErr] = await attempt(() => ctx!.noorm.db.truncate()); -const [, lockErr] = await attempt(() => ctx.noorm.lock.acquire()); -if (lockErr instanceof LockAcquireError) { - console.error(`Lock held by ${lockErr.holder}`); +if (wipeErr instanceof ProtectedConfigError) { + console.error(`Denied ${wipeErr.operation} on ${wipeErr.configName}`); } ``` -The full set of `instanceof`-matchable errors exported from `@noormdev/sdk`: - -| Error | Thrown when | -| --- | --- | -| `RequireTestError` | `requireTest: true` and the config lacks `isTest: true`. Carries `configName`. | -| `NotConnectedError` | A namespace method needing a live connection ran before `connect()` or after `disconnect()`. | -| `ProtectedConfigError` | The config's access policy denied the operation, or requires a confirmation `yes: true` did not supply. Carries `configName` and `operation`. | -| `VaultAccessError` | `vault.set()` got a private key that yields no usable vault key. Carries `configName`. | -| `ImpersonationError` | `impersonate()` on MySQL or SQLite, or with an invalid username. | -| `LockAcquireError` | The lock is held by someone else. Carries `holder`, `heldSince`, `expiresAt`, `configName`, `reason`. | -| `LockExpiredError` | The lock expired mid-operation. Carries `configName`, `identity`, `expiredAt`. | -| `ChangeValidationError`, `ChangeNotFoundError`, `ChangeAlreadyAppliedError`, `ChangeNotAppliedError`, `ChangeOrphanedError`, `ManifestReferenceError` | Change parsing and execution problems. | - - -## TypeScript Support - -Use generics for type-safe Kysely access: - -```typescript -interface Database { - users: { - id: number; - name: string; - email: string; - }; - posts: { - id: number; - user_id: number; - title: string; - }; -} - -const ctx = await createContext({ config: 'dev' }); -await ctx.connect(); - -// ctx.kysely is now Kysely—full type safety -const users = await ctx.kysely - .selectFrom('users') - .select(['id', 'name']) - .where('email', '=', email) - .execute(); -``` - - -## Exported Types - -```typescript -import type { - // Core - Context, - NoormOps, - CreateContextOptions, - Config, - Settings, - Identity, - Dialect, - - // Access policy - Channel, - ConfigAccess, - Role, - - // Results - BatchResult, - FileResult, - RunOptions, - BuildOptions, - - // Changes - Change, - CreateChangeOptions, - AddFileOptions, - ChangeResult, - BatchChangeResult, - ChangeListItem, - ChangeOptions, - ChangeHistoryRecord, - - // Explore - TableSummary, - TableDetail, - ExploreOverview, - - // Operations - TruncateOptions, - TruncateResult, - TeardownResult, - TeardownPreview, - - // Locks - Lock, - LockStatus, - LockOptions, - - // Templates - TemplateResult, - - // Vault - VaultSecret, - VaultStatus, - VaultCopyResult, - VaultCopyOptions, - VaultPropagationResult, - - // Transfer - TransferOptions, - TransferPlan, - TransferTablePlan, - TransferResult, - TransferTableResult, - ConflictStrategy, - - // DT - ExportOptions, - ImportOptions, - - // TVP - TvpValue, - - // Impersonation - ImpersonatedScope, - - // Proc/func/tvf tuple helpers - ExtractArgs, - ExtractReturn, - - // Events - NoormEvents, - NoormEventNames, -} from '@noormdev/sdk'; -``` - -Four shapes named by public method signatures are not re-exported from the package: `BatchChangeOptions` -(`changes.ff` / `next` / `rewind`), `TeardownOptions` (`db.teardown`), `ForceReleaseResult` -(`lock.forceRelease`'s return), and `ConnectionRetryOptions` (`connect`'s parameter). Declare those -inline until they are exported. +Every class below is exported from `@noormdev/sdk`: + +| Error | Thrown when | Carries | +|---|---|---| +| `RequireTestError` | `requireTest: true` and the config lacks `isTest: true` | `configName` | +| `NotConnectedError` | A call that needs the pool ran before `connect()` or after `disconnect()` | | +| `ProtectedConfigError` | The access policy denied the call, or it needs a confirmation `yes` did not supply | `configName`, `operation` | +| `VaultAccessError` | `vault.set()` got a key that yields no vault key | `configName` | +| `ImpersonationError` | `impersonate()` on mysql or sqlite, or with an invalid username | | +| `LockAcquireError` | Another identity holds the lock | `configName`, `holder`, `heldSince`, `expiresAt`, `reason` | +| `LockExpiredError` | The lock expired during an operation | `configName`, `identity`, `expiredAt` | +| `ChangeValidationError` | `validate()` found duplicates, a change has no files, or its files are empty or only template placeholders | `changeName`, `issue` | +| `ChangeNotFoundError` | `parse`, `apply`, or `revert` named a change that is not on disk | `changeName` | +| `ChangeOrphanedError` | `apply` or `revert` named a change recorded in the database whose folder is gone | `changeName` | +| `ChangeNotAppliedError` | `revert` of a change that was never applied | `changeName` | +| `ChangeAlreadyAppliedError` | Exported; no SDK method throws it today | `changeName`, `appliedAt` | +| `ManifestReferenceError` | A `.txt` manifest names a file that is not under `paths.sql` | `manifestPath`, `missingPath` | + + +## Exported types and values + + +| Group | Names | +|---|---| +| Factory and classes | `createContext`, `Context`, `NoormOps`, `ChangesNamespace`, `RunNamespace`, `DbNamespace`, `LockNamespace`, `VaultNamespace`, `SecretsNamespace`, `TemplatesNamespace`, `TransferNamespace`, `DtNamespace`, `UtilsNamespace` | +| Core types | `CreateContextOptions`, `Config`, `Settings`, `Identity`, `Dialect`, `Channel`, `ConfigAccess`, `Role` | +| Run | `RunOptions`, `BuildOptions`, `FileResult`, `BatchResult` | +| Changes | `Change`, `ChangeFile`, `ChangeFileType`, `CreateChangeOptions`, `AddFileOptions`, `ChangeOptions`, `ChangeResult`, `BatchChangeResult`, `ChangeListItem`, `ChangeHistoryRecord` | +| Explore | `ExploreOverview`, `TableSummary`, `TableDetail`, `ViewSummary`, `ViewDetail`, `ProcedureSummary`, `ProcedureDetail`, `FunctionSummary`, `FunctionDetail`, `TypeSummary`, `TypeDetail`, `IndexSummary`, `ForeignKeySummary`, `ColumnDetail`, `ParameterDetail` | +| Database operations | `TruncateOptions`, `TruncateResult`, `TeardownResult`, `TeardownPreview` | +| Locks | `Lock`, `LockStatus`, `LockOptions` | +| Templates | `TemplateResult` | +| Vault | `VaultSecret`, `VaultStatus`, `VaultCopyOptions`, `VaultCopyResult`, `VaultPropagationResult` | +| Transfer and DT | `TransferOptions`, `TransferPlan`, `TransferTablePlan`, `TransferResult`, `TransferTableResult`, `ConflictStrategy`, `ExportOptions`, `ImportOptions` | +| Routines | `tvp`, `TvpValue`, `ImpersonatedScope`, `ExtractArgs`, `ExtractReturn` | +| Events | `noormObserver`, `NoormEvents`, `NoormEventNames` | +| Errors | `RequireTestError`, `ProtectedConfigError`, `NotConnectedError`, `VaultAccessError`, `ImpersonationError`, `LockAcquireError`, `LockExpiredError`, `ChangeValidationError`, `ChangeNotFoundError`, `ChangeAlreadyAppliedError`, `ChangeNotAppliedError`, `ChangeOrphanedError`, `ManifestReferenceError` | + +Five shapes that public signatures name are not exported: `BatchChangeOptions` (`changes.ff`, `next`, `rewind`), `FileHistoryRecord` (`changes.getFileHistory`), `TeardownOptions` (`db.teardown`), `ForceReleaseResult` (`lock.forceRelease`), and `ConnectionRetryOptions` (`connect`). Declare them inline, or derive them with `Parameters<>` and `Awaited>`. + + +## What bites + + +| Symptom | Cause | Fix | +|---|---|---| +| `ProtectedConfigError: requires confirmation` on `db.truncate()` against your own dev config | The default `admin` role still confirms `db:truncate`, `db:teardown`, `change:rm`, `vault:propagate`, and `lock:force`, and the SDK cannot prompt | Create the context with `yes: true`, or set `NOORM_YES=1`. On the `agent` channel nothing unblocks it | +| `NotConnectedError` | A query, `db`, `lock`, `vault`, or `dt` call ran before `connect()` | Call `await ctx.connect()` first; see the table in [How is a context laid out?](#how-is-a-context-laid-out) | +| `ff()` returned and nothing applied | A failed change comes back as `status: 'failed'`, and a missing changes directory is only a warning | Check `result.status`, `result.error`, and `result.warnings` | +| `ChangeValidationError: Files are empty or contain only template placeholders` | `changes.create` scaffolds stubs and `apply` refuses them | Write the SQL into the `change/` file first | +| `build()` succeeds and ran nothing | A mistyped `build.include` matched no file | Read `result.unmatchedInclude` and `result.unmatchedExclude` | +| `pending()` shows an item with `status: 'failed'` and `isNew: true` | The folder cannot be parsed, such as a manifest naming a deleted `db/sql/` file | Read `errorMessage`. `discover()` omits the same folder | +| `{ note: undefined }` overwrote a procedure's `DEFAULT` | `undefined` is sent as `NULL`, and mssql applies a default only to an omitted parameter | See [Parameter handling and NULL semantics](#parameter-handling-and-null-semantics) | +| `func()` throws on mysql | Named parameters are not supported in function calls | Pass an array | +| A connection stays out of the pool | An explicit `impersonate()` scope was never reverted | Call `scope.revert()`, or prefer callback mode. `disconnect()` also releases it | +| `withSchema` did not qualify a raw fragment | `` sql`...` `` bypasses the query-builder plugins | Qualify identifiers by hand inside raw SQL | +| One context's handler fires for another tenant's events | `noormObserver` is process-global | Filter on `event.configName`, where the payload has it | +| `Vault operations require a cryptographic identity` | No `~/.noorm/identity.json` and no `NOORM_IDENTITY_*` | Run `noorm identity init`, or set the three CI variables | +| `Private key required to decrypt state` in CI | A `state.enc` exists and no key is available | Set `NOORM_IDENTITY_PRIVATE_KEY`, `_NAME`, and `_EMAIL`, or use env-only mode | +| A viewer's context cannot read secrets or the vault | `secret:read` and `vault:read` are denied to `viewer` | Use a context on a role that holds them | + + +## Next + + +- [Building your SDK](/getting-started/building-your-sdk): wire a database package into an application +- [Deploying an application](/guide/deployment): env-only mode, one context per process +- [Access roles](/guide/environments/configs#access-roles): set `access` per config +- [Forward and revert](/guide/changes/forward-revert): what `ff`, `next`, and `rewind` do +- [Teardown internals](/dev/teardown): drop order and the mssql truncate strategy diff --git a/docs/tui.md b/docs/tui.md index 42fabf82..12e1ee5c 100644 --- a/docs/tui.md +++ b/docs/tui.md @@ -1,140 +1,92 @@ # Terminal UI -Launch the interactive terminal interface with: +Run builds, apply changes, and explore the database from the keyboard. ```bash noorm ui ``` -![A full pass through the TUI: adding a config, creating the database, building the schema, fast-forwarding changes, and browsing the result in the explorer](/image/tui.gif) +![Adding a config, creating the database, building the schema, fast-forwarding changes, and browsing the result](/image/tui.gif) -Everything in noorm is accessible through keyboard shortcuts. The mouse works -too, in lists and grids — see [Mouse](#mouse) if you would rather it did not. +`noorm ui` takes no flags and opens on Home. Every other `noorm` command is a non-interactive CLI, and bare `noorm` prints the command list; see the [CLI reference](/headless). -The TUI is a dedicated subcommand — every other `noorm` command runs as a non-interactive CLI. Running `noorm` on its own prints the command list (citty's `--help`) instead of opening the wizard, so the entry into the TUI is always explicit. See the [CLI Reference](/headless) for the headless surface. +## How do I get around? -## Home Screen -![The noorm TUI home screen: status, quick actions, and recent activity](/image/tui/home.png) +Home opens four working screens with one key. Secrets hang off a config, and the schema explorer off the database menu: - -## Navigation Map - -``` - ┌─────────┐ - │ Home │ - └────┬────┘ - ┌──────────┬──────────┬────┴────┬──────────┐ - │ │ │ │ │ - [c]│ [g]│ [r]│ [d]│ [s]│ - ▼ ▼ ▼ ▼ ▼ - ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ - │ Config │ │ Change │ │ Run │ │Database│ │Settings│ - │ List │ │ List │ │ Menu │ │ Menu │ │ Menu │ - └────────┘ └────────┘ └────────┘ └────────┘ └────────┘ - │ │ │ │ - │ │ │ ├── Explore (tables, views...) - │ │ │ └── Terminal (SQL REPL) - │ │ │ - │ │ └── Build, File, Directory, Exec, Inspect - │ │ - │ └── FF, Next, Run, Revert, Rewind, History - │ - ├── Add, Edit, Delete, Copy, Use - │ - └──[k]── Secrets List +```mermaid +flowchart TD + Home["Home"] -->|c| Config["Config list"] + Home -->|g| Changes["Changes list"] + Home -->|r| Run["Run menu"] + Home -->|d| Db["Database menu"] + Config -->|k| Secrets["Secrets of the
highlighted config"] + Db -->|x| Explore["Schema explorer"] ``` -Secrets hang off a config, so `k` opens them from the Config List, not from -Home. Export, import, and validate sit behind `[+] More` on the Config List. - - -## Keyboard Shortcuts +![Home: status, quick actions, and recent activity](/image/tui/home.png) + +| Key on Home | Opens | +|-------------|-------| +| `r` | Run menu | +| `c` | Config list | +| `g` | Changes list | +| `d` | Database menu | +| `+` | More: Settings, Vault, Identity, Lock | +| `s`, `v`, `i`, `l` | Settings, Vault, Identity, Lock, without going through More | +| `u` | Update | +| `1` | Run build | +| `2` | Fast-forward changes | +| `3` | Lock status | +| `q` | Quit | +| Task | Keys | +|------|------| +| Switch config | `c`, `↓` to the config, `Enter` | +| Fast-forward all changes | `g`, `f` | +| Run the build | `r`, `b` | +| Validate a config | `c`, `+`, `v` | -### Home Navigation -| Key | Screen | Description | -|-----|--------|-------------| -| `r` | Run | Execute schema files | -| `c` | Config | Manage database connections | -| `g` | Changes | View and apply changes | -| `d` | Database | Explore schema, run queries | -| `+` | More | Settings, vault, identity, lock | -| `s` | Settings | Project configuration | -| `v` | Vault | Team-shared encrypted secrets | -| `i` | Identity | View/edit your identity | -| `l` | Lock | View/manage database locks | -| `u` | Update | Check for a newer noorm | -| `q` | — | Quit noorm | +## Which keys work on every screen? -`s`, `v`, `i`, and `l` work from Home directly as well as from `[+] More`. -Three number keys run the quick actions listed on the home screen: | Key | Action | |-----|--------| -| `1` | Run build | -| `2` | Apply changes (fast-forward) | -| `3` | View lock status | - -Per-config **secrets** are not on this list — they hang off a config rather -than the project, so you reach them with `k` from the config list. - +| `Esc` | Go back, or cancel the operation in flight | +| `Ctrl+C` | Quit | +| `?` | Help overlay. Any key closes it | +| `Shift+L` | Toggle the log viewer | +| `Shift+Q` | Open the SQL terminal | +| `D` | Toggle dry-run mode. The status bar shows `DRY` | +| `F` | Toggle force mode. The status bar shows `FORCE` | -### Common Actions (in sub-screens) +`D` applies to the run screens, `change ff`, and the database wipe, teardown, and transfer screens. `F` applies to the run screens and `change ff`. -| Key | Action | Available In | -|-----|--------|--------------| -| `a` | Add new | Config, Changes, Secrets | -| `e` | Edit | Config, Changes, Secrets | -| `d` | Delete | Config, Changes, Secrets | -| `c` | Copy | Config | -| `k` | Secrets | Config (secrets for the highlighted config) | -| `+` | More | Config (export, import, validate) | -| `Enter` | Use/Activate | Config (set as active) | +While a text field takes input (a form field in edit mode, a search box, the SQL editor), every key above except `Ctrl+C` types its character instead. `?`, `D`, and `F` also stand down inside dialogs. -Export, import, and validate live behind `[+] More` on the config list rather -than on the list itself, which keeps the destructive and the routine actions -apart. +### Lists -### List Navigation | Key | Action | |-----|--------| -| `↑` | Move up | -| `↓` | Move down | +| `↑` `↓` | Move the cursor. Wraps at both ends, except on the changes list | | `Enter` | Select | -| `Escape` | Go back | -| `1`-`9` | Quick select by number, on lists that show numbers | +| `Esc` | Back | +| `1`-`9` | Select by number, on lists that show numbers: Settings and the explorer | +| `Space` | Toggle an item, on multi-select lists. `Enter` then submits | -Numbered selection is enabled per list — if a list renders numbers down its -left edge (Settings and the schema explorer do), the digits work there. +A list fits the window, so a taller terminal shows more rows. It keeps its cursor when you open an item and come back. -Every list sizes itself to the window, so a taller terminal shows more rows -rather than the same fixed count with the rest behind a scroll marker. A list -also keeps its cursor when you open an item and come back, so walking a long -list one entry at a time does not restart at the top. +### Forms -### Form Navigation -Forms have two modes and `Enter` switches between them. - -| Mode | Key | Effect | -|------|-----|--------| -| Browse | `↑` `↓` | Move between fields, every field type included | -| Browse | `Tab` / `Shift+Tab` | Same, forwards and backwards | -| Browse | `Enter` | Open the active field for editing | -| Browse | `Escape` | Cancel the form | -| Edit | `Enter` | Commit and return to browse | -| Edit | `Escape` | Restore the value the field held when edit opened | -| Edit | `Tab` | Commit and move to the next field | - -Enter is the mode switch, so submitting is a place you navigate to rather than -a key you press: +A form has two modes. `Enter` switches between them, so submitting is a place you navigate to: ```mermaid stateDiagram-v2 @@ -147,105 +99,67 @@ stateDiagram-v2 Actions --> [*]: Enter on the submit button ``` -Past the last field the cursor lands on the action row, where `←` `→` move -between the buttons and `Enter` activates one. The submit button carries the -screen's own label, `[ Create Config ]` on the add-config form. - -Two field types never open into edit mode: - -- A **checkbox** toggles in place on `Enter` or `Space`. -- A **select** expands on `Enter`, moves with `↑` `↓`, and takes the highlighted - option on a second `Enter`. Collapsed, it shows its current value on one line - like every other field. +| Mode | Key | Effect | +|------|-----|--------| +| Browse | `↑` `↓`, `Tab`, `Shift+Tab` | Move between fields, wrapping through the action row | +| Browse | `Enter` | Open the field for editing | +| Browse | `Esc` | Cancel the form | +| Edit | `Enter` | Commit and return to browse | +| Edit | `Esc` | Restore the value the field held when edit opened | +| Edit | `Tab`, `Shift+Tab` | Commit and move to the next or previous field | +| Actions | `←` `→`, `Enter` | Pick a button and activate it | -A red `*` after a label marks a required field. Submitting with one empty puts -the cursor on it and shows the error beside the value. +A checkbox toggles in place on `Enter` or `Space`. A select expands on `Enter`, moves with `↑` `↓`, and takes the highlighted option on a second `Enter`. A red `*` marks a required field. Submitting with one empty puts the cursor on it, with the error beside the value. ### Mouse -On by default, in lists and result grids only: + +On by default, in lists and result grids: | Action | Effect | |--------|--------| | Click a row | Move the cursor to it | | Double-click a row | Same as `Enter` on that row | -| Wheel up / down | Move the cursor one row | +| Wheel | Move the cursor one row, or scroll a document | -A click acts on whatever already has focus. Clicking a list that is not focused -does nothing — it does not move focus there, so the keyboard stays where you -left it. +A click acts on whatever already has focus. Clicking an unfocused list does nothing. - -#### Text selection stopped working - -That is this feature, and it is the one thing it costs you. Any terminal -mouse-tracking mode hands the mouse to the application, so click-drag selection -needs a modifier held down: Option in macOS Terminal and iTerm2, Shift in most -others. - -If you would rather have plain selection back, turn the mouse off in -`.noorm/settings.yml`: +::: tip +Mouse tracking takes click-drag text selection away from the terminal. Hold Option in macOS Terminal and iTerm2, or Shift in most others. To turn the mouse off, set `ui.mouse` in `.noorm/settings.yml`; `?` prints the same hint. ```yaml +# .noorm/settings.yml ui: mouse: false ``` +::: -`?` inside the TUI prints the same line, so you do not have to remember which -file it lives in. Nothing else changes: every screen is fully keyboard-driven -either way. - - -### Global Shortcuts - -| Key | Action | -|-----|--------| -| `Shift+L` | Toggle log viewer overlay | -| `Shift+Q` | Open the SQL terminal | -| `?` | Show help | -| `Escape` | Go back / Cancel | -| `Ctrl+C` | Quit | - -While a text field is taking input (a form field in edit mode, a search box, -the SQL editor), `Shift+L`, `Shift+Q`, `?`, `D`, and `F` type their character -instead. They work again once you leave the field. +## What does Esc do while the database is busy? -### Cancelling a Database Operation -A screen waiting on a database says so, and says that `Escape` will stop it: +A screen waiting on the database says so and names the key: ``` Testing connection... [Esc] Cancel ``` -That appears while a config is being tested (add and edit), while the -force-release screen checks lock status, and while the SQL terminal connects or -runs a query. `Escape` returns the screen to a usable state; nothing is saved. - -Two things are worth knowing about what "cancel" means here: +That covers testing a config (add and edit), the force-release lock check, and the SQL terminal connecting or running a query. `Esc` returns the screen to a usable state and saves nothing. The client always stops waiting. Whether the server stops depends on the dialect: -- **The client always stops waiting. The server usually does not stop working.** - A cancelled query keeps running on PostgreSQL and MySQL only until noorm's - cancel request reaches it, and on SQL Server and SQLite it is never told at - all. The message names which happened, and - [the SQL terminal guide](./guide/database/terminal.md) has the per-dialect - table. -- **A connect that answers after you cancelled is closed, not adopted.** The - screen keeps the state you cancelled into, and the connection is destroyed - rather than left half-open. +| Dialect | Effect on the server | +|---------|----------------------| +| PostgreSQL | `pg_cancel_backend` asks it to stop the statement | +| MySQL | `KILL QUERY` stops the statement | +| SQL Server, SQLite | Nothing. The statement finishes. The message reads `Stopped waiting. The query may still be running on the server.` | -A connection attempt also ends on its own after 15 seconds, with or without a -keypress. Raise `connection.connectTimeoutMs` for a link that is slow but -working. +A connect that answers after you cancelled is closed, not adopted. A connection attempt also ends on its own after 15 seconds. Raise `connection.connectTimeoutMs` for a link that is slow but working. [The SQL terminal guide](/guide/database/terminal) has the full table. ### Long-Running Files -A file that runs longer than 10 seconds gets a status block under its name, -refreshed every 10 seconds, on the run screens (Build, Exec, Dir, File) and the -change screens (Run, Revert, Fast Forward, Next, Rewind): + +A file that runs past 10 seconds gets a status block under its name, refreshed every 10 seconds. It appears on the run screens (Build, Exec, Dir, File) and the change screens (Run, Revert, Fast Forward, Next, Rewind): ``` Current: 06_items.sql @@ -254,286 +168,276 @@ CREATE INDEX on items · building index: loading tuples · 4210/9800 (42%) 2 pa Waiting on pid 4411 (CREATE INDEX items_embedding ..., 17m04s) · Lock: transactionid ``` -The third line appears when the server reports progress for the running -command (index builds, `VACUUM`, `CLUSTER`, `COPY`, `ANALYZE` on PostgreSQL; -`percent_complete` on SQL Server; stage counters on MySQL). The fourth appears when the file is waiting -on another session's lock. SQLite shows elapsed time only. +| Line | Appears when | +|------|--------------| +| `Running for … pid …` | Always. SQLite shows elapsed time only | +| Operation progress | The server reports it: index builds, `VACUUM`, `CLUSTER`, `COPY`, `ANALYZE` on PostgreSQL; `percent_complete` on SQL Server; stage counters on MySQL | +| `Waiting on pid …` | The file waits on another session's lock | -On Run Build and Run Exec, `Escape` pressed once shows a warning; pressed again -within 2 seconds it cancels the run. No further file starts. On PostgreSQL and -MySQL the running statement is cancelled on the server; on SQL Server and -SQLite it finishes first, and the screen says so. Leaving the screen does not -cancel the run. +On Run Build and Run Exec, press `Esc` once for a warning and again within 2 seconds to cancel. No further file starts. PostgreSQL and MySQL cancel the running statement. SQL Server and SQLite let it finish, and the screen says `Stopping after the current file finishes.` Leaving the screen does not cancel the run. -## Screen Reference +## What does each screen do? ### Config List -![The configuration list screen](/image/tui/config-list.png) -- `●` indicates active config -- `○` indicates inactive config -- `>` indicates cursor position -- `[user: agent:]` tag shows access roles for any config whose access differs from the default (`user: admin`, `agent: viewer`) — omitted entirely for configs still on the default -- `[test]` tag shows test configs -- Press `Enter` on a config to activate it +![The configuration list](/image/tui/config-list.png) + +A config shows `●` when active and `○` when not, with `[user: agent:]` when its access differs from the default (`admin` for you, `viewer` for the agent) and `[test]` on a test config. + +| Key | Action | +|-----|--------| +| `Enter` | Make the highlighted config active. On the config that is already active, open its edit form | +| `a` | Add | +| `e` | Edit | +| `d` | Delete | +| `c` | Copy | +| `k` | Secrets of the highlighted config | +| `+` | More: `x` export, `i` import, `v` validate | -Press `a` to add one. Adding a config is the one operation that is interactive only — `noorm config add` on the CLI directs you here: +Adding a config is interactive only. `noorm config add` points here, and `noorm config import ` is the headless route. -![The add-config form: name, dialect, connection details, and per-channel access roles](/image/tui/config-add.png) +![The add-config form](/image/tui/config-add.png) -The two role fields set access per **channel** — who is *driving*. `User Role` covers a human on the CLI, TUI, or SDK; `Agent Role` covers an AI agent, over MCP and the CLI alike. They are independent, so a config can be wide open at your terminal and read-only — or invisible — to an agent. New configs default to `admin` for you and `viewer` for the agent. See [Configs](/guide/environments/configs#access-roles) for what each role permits. +`User Role` covers a human on the CLI, TUI, or SDK. `Agent Role` covers an AI agent over MCP and the CLI. They are independent. [Configs](/guide/environments/configs#access-roles) lists what each role permits. ### Changes List -![The changes list, showing applied and pending changes](/image/tui/changes-list.png) -- `✓` = Applied -- `○` = Pending -- `✗` = Failed +![The changes list](/image/tui/changes-list.png) -When no changes exist: +| Symbol | Meaning | +|--------|---------| +| `✓` | Applied | +| `○` | Pending | +| `✗` | Failed | +| `↩` | Reverted | +| `!` | Orphaned: recorded in the database, folder gone from disk | -``` -No changes found. Press [a] to create one. -``` +| Key | Action | +|-----|--------| +| `Enter` | Run a pending, failed, or reverted change. Revert an applied one | +| `a` | Add | +| `e` | Edit. Not for an orphaned change | +| `d` | Delete | +| `r` | Run. Pending, failed, or reverted only | +| `v` | Revert. Applied only | +| `n` | Next: apply the next N pending changes, one by default | +| `f` | Fast-forward all pending changes | +| `w` | Rewind | +| `h` | History | -Press `h` for the execution history — what ran, when, and who ran it: +With no changes, the list reads `No changes found. Press [a] to create one.` -![Change execution history, with per-change status and the identity that applied it](/image/tui/change-history.png) +![Change history](/image/tui/change-history.png) + +History lists what ran, when, and under which identity. `Enter` shows the files, and `e` shows the full error of a failed change. ### Run Menu + ![The run menu](/image/tui/run-menu.png) -Picking a file that already ran with the same content opens **File Previously Run** and asks "Run this file again?". A directory with such files opens **Files Previously Run**. +`b` or `1` Build, `e` or `2` Exec, `f` or `3` File, `d` or `4` Dir, `i` or `5` Inspect. + +A file that already ran with the same content opens **File Previously Run** and asks `Run this file again?`. A directory with such files opens **Files Previously Run**. ### Database Menu -![The database operations menu](/image/tui/database-menu.png) + +![The database menu](/image/tui/database-menu.png) + +| Key | Action | Needs a connection | +|-----|--------|--------------------| +| `c` | Create the database. Shows a toast if it is already initialized | No | +| `d` | Destroy | No | +| `x` | Explore | Yes | +| `w` | Wipe: truncate | Yes | +| `t` | Teardown | Yes | +| `r` | Transfer | Yes | + +The SQL terminal is not on this menu. Open it with `Shift+Q`. ### Schema Explorer -![The schema explorer overview, with object counts by category](/image/tui/explore-overview.png) -Press a number to drill into a category: +![The explorer overview, with object counts by category](/image/tui/explore-overview.png) + +Each category opens from the overview by its letter or its number. A category with no objects does nothing. + +| Category | Keys | +|----------|------| +| Tables | `t`, `1` | +| Views | `v`, `2` | +| Procedures | `p`, `3` | +| Functions | `f`, `4` | +| Types | `y`, `5` | +| Indexes | `i`, `6` | +| Foreign Keys | `k`, `7` | -![The explorer table list](/image/tui/explore-tables.png) +![The table list](/image/tui/explore-tables.png) -Select a table to see its full schema: +`Enter` on an object opens its detail. Indexes and foreign keys have none. ![Table detail: columns, indexes, and foreign keys](/image/tui/explore-table-detail.png) -A detail longer than the window scrolls rather than running off the bottom: +A detail longer than the window scrolls. The footer lists only the keys the screen answers to. | Key | Action | |-----|--------| | `↑` `↓` | Scroll one line | -| `Ctrl+U` / `Ctrl+D` | Half a page | -| `PageUp` / `PageDown` | A full page. On macOS these are `fn ↑` and `fn ↓`, which is what the footer says there | -| `Home` / `End` | Top, bottom | -| `v` | Redraw the same rows with nothing truncated | -| `r` | Read rows from the object | - -The footer lists only the keys the current screen answers to, so a detail that -fits shows no scroll hints and a view shows no `[r] Rows`. +| `Ctrl+U`, `Ctrl+D` | Half a page | +| `PageUp`, `PageDown` | A full page. On macOS the footer shows `fn ↑↓` | +| `Home`, `End` | Top, bottom | +| `v` | Full text: the same rows with nothing truncated | +| `r` | Read rows. Tables only | ### Reading Rows -The explorer describes structure. `r` on a detail screen reads the rows -themselves, without leaving the explorer for the SQL terminal: + +`r` on a table detail reads its rows without leaving the explorer for the SQL terminal. ![Peeking at both ends of a table, then opening one row as a document](/image/tui-rows.gif) -The peek reads both ends of the table rather than a page from the top, so the -last rows by primary key are as reachable as the first: +The peek reads both ends of the table, so the last rows by primary key are as reachable as the first: -![The row peek: first rows and last rows, side by side](/image/tui/explore-row-peek.png) +![First and last rows, side by side](/image/tui/explore-row-peek.png) | Key | Action | |-----|--------| | `↑` `↓` | Move the cursor within a set | | `Tab` | Swap between the first set and the last | | `Enter` | Open the highlighted row as a document | -| `/` `s` `c` | Filter, sort, clear, on the focused set | -| `Escape` | Close the peek | - -Three things decide what comes back: +| `/` | Filter. `Tab` cycles the column, `Enter` applies, `Esc` clears | +| `s` | Sort. `←` `→` pick the column, `a` ascending, `d` descending | +| `c` | Clear filter and sort | +| `Esc` | Close the peek | -- **Reading rows needs `sql:read`**, not the `explore` permission the rest of - these screens use. A config an agent may inspect the schema of is not one it - may read data from. See [Configs](/guide/environments/configs#access-roles). -- **The tail needs a primary key.** With one, the last rows ride the index. - Without one, only the first rows appear. -- **A short table is one set.** When both ends meet, the peek says - `All N rows` and draws a single table. +| Condition | Result | +|-----------|--------| +| The role lacks `sql:read` | Refused. Reading rows needs `sql:read`, not the `explore` permission the other explorer screens use | +| The table has no primary key | Only the first rows appear: `No primary key, so there is no last set — rows are in storage order.` | +| Both ends meet | One table, headed `All N rows` | -The table fits as many whole columns as the terminal holds and marks the rest -`… N more columns`. `Enter` on a row is what shows them all: +The grid fits as many whole columns as the terminal holds and marks the rest `… N more columns`. `Enter` on a row shows them all: -![One row as a YAML document, with the keys to walk to the next](/image/tui/explore-row-view.png) +![One row as a YAML document](/image/tui/explore-row-view.png) | Key | Action | |-----|--------| -| `←` `→` | Previous row, next row, in the order the table was showing | +| `←` `→` | Previous row, next row, in the order the table showed | | `↑` `↓` | Scroll a document taller than the window | -| `f` | Swap YAML for JSON. The choice holds for the rest of the session | -| `Escape` | Back to the peek, on the row you were reading | +| `f` | Swap YAML and JSON. The choice holds for the session | +| `Esc` | Back to the peek, on the row you were reading | ### SQL Terminal -Press `Shift+Q` anywhere to open the SQL terminal against the active config: -![The built-in SQL terminal](/image/tui/sql-terminal.png) +`Shift+Q` opens the SQL terminal against the active config from any screen. -- `↑` `↓` walk your query history, and `h` opens it as a list -- A result wider than the terminal keeps whole columns and marks the remainder - `… N more columns`, rather than cramming every column into a few characters -- `Enter` on a result row opens it as a document, the same YAML or JSON view the - row peek uses, which is how you read the columns the grid left out +![The SQL terminal](/image/tui/sql-terminal.png) + +| Key | Action | +|-----|--------| +| `Enter` | Run the query | +| `Shift+Enter` | New line | +| `Shift+Tab` | Toggle edit mode, where `Enter` inserts a new line | +| `↑` `↓` | Walk query history, when the box is empty or already browsing | +| `h` | Open the history list, when the box is empty | +| `Tab` | Switch between the query and the results, once there are rows | +| `Esc` | In results: back to the query. With text in the box: clear it. Empty: go back. While a query runs: cancel it | + +In the results grid, `/`, `s`, and `c` filter, sort, and clear as in the row peek, and `Enter` opens a row as a document. A result wider than the terminal keeps whole columns and marks the remainder `… N more columns`. + +The history list takes `↑` `↓`, `Enter` to view a result, `r` to re-run a query, and `c` to clear. ### Log Viewer -Press `Shift+L` anywhere to toggle the log overlay: -![The log viewer overlay, opened with Shift+L](/image/tui/log-viewer.png) +`Shift+L` toggles an overlay that tails events as they happen, newest first. + +![The log viewer overlay](/image/tui/log-viewer.png) -The overlay tails events as they happen, so you can watch an operation run. -`[/]` searches, `[g]` and `[G]` jump to the top and bottom, `[Space]` pauses the -tail, `[Enter]` opens a single entry in full, and `Shift+L` again closes it. +| Key | Action | +|-----|--------| +| `/` | Search. `Enter` leaves search, `Esc` clears it | +| `g`, `G` | Newest, oldest | +| `↑` `↓`, `PageUp`, `PageDown` | Move | +| `Space` | Pause or resume the tail | +| `r` | Refresh | +| `Enter` | Open one entry in full. `Esc` returns to the list | +| `Shift+L`, `Esc` | Close | -### More Options +### More -Press `+` from home for the screens that aren't part of the day-to-day loop: -![The More Options menu: settings, vault, identity, and lock](/image/tui/more-menu.png) +![The More menu](/image/tui/more-menu.png) -Each of these also has a direct key from home — `+` just groups them. +`+` from Home opens Settings (`s`), Vault (`v`), Identity (`i`), and Lock (`l`). Each also has its direct key on Home. ### Settings -![The settings screen, listing the seven setting groups](/image/tui/settings.png) -Edits here write to `.noorm/settings.yml`. Press `i` to create that file if the -project doesn't have one yet. See [Configs](/guide/environments/configs) and -[SQL File Organization](/guide/sql-files/organization) for what each group -controls. +![The settings screen](/image/tui/settings.png) + +Seven groups: Build, Paths, Stages, Rules, Strict, Logging, Secrets. `Enter` or a number opens one, and edits write to `.noorm/settings.yml`. `i` creates that file when the project has none. See [SQL file organization](/guide/sql-files/organization) and [Configs](/guide/environments/configs). ### Identity -![The identity screen, showing the current keypair and fingerprint](/image/tui/identity.png) -Your identity signs your name to every change execution, which is what makes -[change history](/guide/changes/history) attributable across a team. It lives -in `~/.noorm/`, not in the project, so it follows you between repositories. +![The identity screen](/image/tui/identity.png) + +`e` edit, `x` export your public key, `r` regenerate, `u` known users. The identity lives in `~/.noorm/` and names who ran each change in the [history](/guide/changes/history). The CLI equivalent is [`noorm identity`](/cli/identity). ### Vault + ![The vault screen before initialization](/image/tui/vault.png) -The vault holds team-shared encrypted secrets in the database itself, so -teammates get them by connecting rather than by copying a `.env` around. It -starts uninitialized — press `i` to create it. See [Vault](/guide/environments/vault). +The vault holds team-shared encrypted secrets in the database. It starts uninitialized: `i` creates it. After that, `a` adds a secret, `Enter` edits the highlighted one, and `p` propagates access to teammates. See [Vault](/guide/environments/vault). ### Secrets -Per-config secrets are reached with `k` from the config list, not from home — -they belong to a config rather than to the project: ![The secrets screen for a config](/image/tui/secrets.png) -These are values a config needs at connection or render time. See -[Secrets](/guide/environments/secrets) for how they resolve against stages. +`k` on the config list opens the highlighted config's secrets: `a` add, `e` or `Enter` edit, `d` delete. See [Secrets](/guide/environments/secrets). ### Lock -![The lock screen, showing current lock state](/image/tui/lock.png) - -noorm takes a lock around operations that write to the database, so two people -running a build against the same environment don't interleave. `[s]` shows -status, `[a]` acquires, `[r]` releases, and `[f]` force-breaks a stale lock. -See [Locking](/dev/lock). +![The lock screen](/image/tui/lock.png) -### Update - -Press `u` from Home to check GitHub releases for a newer noorm. `[i]` -installs an update if one is available, `[r]` re-checks, and `esc` goes -back. +noorm locks the database around writes, so two people building one environment do not interleave. On the lock menu, `s` or `1` shows status, `a` or `2` acquires, `r` or `3` releases, and `f` or `4` force-breaks a stale lock. The status screen also has `x` to extend. See [Locking](/dev/lock). -The binary runs to around 70MB, so the install reports where it has got to -rather than leaving you watching a spinner. Below the spinner you get the -byte count as `42.0 / 73.0 MB (57%)`, and a progress bar tracking the same -percent. When the server sends no `Content-Length` there's nothing to -measure against, so the count drops to a bare `9.0 MB received` and the bar -is omitted rather than sitting at zero. -If a download stalls and resumes, a yellow line names why and which attempt -it's on, for example `download stalled, no data for 30s — resuming (attempt -1/5)...`. Progress isn't reset by the retry, since it resumes from the bytes -already on disk rather than starting the file over. The npm install path -(`npm install -g`) owns its own output, so none of this applies there. - - -## Tips - - -### Quick Config Switching - -From home, press `c`, arrow to the config you want, then `Enter` to activate it. -The config list is not numbered, so the digits do nothing there: - -``` -c → ↓ → Enter -``` - -`Enter` on the config that is *already* active opens its edit form instead. - - -### Fast Forward All Changes - -``` -g → f -``` - - -### Run Build - -``` -r → b -``` +### Update -### Check Connection +`u` from Home checks GitHub releases for a newer noorm. `i` installs it, `r` checks again, and `Esc` goes back. -Validate lives behind `[+] More` on the config list: +A binary install reports its download as `42.0 / 73.0 MB (57%)` with a progress bar. Without a `Content-Length` it shows `9.0 MB received` and no bar. A stalled download resumes from the bytes on disk and shows `download stalled — no data for 30s — resuming (attempt N/5)...`. An npm install runs the package manager and shows none of this. -``` -c → + → v (validate highlighted config) -``` +## Next -## Color Coding -| Color | Meaning | -|-------|---------| -| Green | Success, applied, active | -| Yellow | Warning, pending, in progress | -| Red | Error, failed | -| Gray | Skipped, unchanged | -| Cyan | Info, links | +- [CLI reference](/headless): the non-interactive surface +- [The SQL terminal guide](/guide/database/terminal): history, result browsing, per-dialect cancel +- [Execution](/guide/sql-files/execution): what the run screens do with each file +- [Configs](/guide/environments/configs): access roles diff --git a/docs/why-noorm.md b/docs/why-noorm.md index 36482ad5..11989fb3 100644 --- a/docs/why-noorm.md +++ b/docs/why-noorm.md @@ -1,109 +1,94 @@ --- title: Why noorm -description: noorm is the fifth attempt at one problem. Here is the problem, the CLIs that came before, and the questions that kept making the tool bigger. +description: noorm is the fifth attempt at one problem. The questions that grew it, the CLI libraries that came first, and what LLMs changed. --- # Why noorm -I stopped using ORMs and needed something to run the SQL. +I stopped using ORMs and needed something to run the SQL. Every other part of noorm answers a question I hit after that. -That is the whole origin. Everything else in noorm arrived as an answer to a question I hit while working without one. +## Which questions grew the tool? -## The tools that came first - -noorm is the fifth attempt. The first three were about the interface, and each one replaced the last because I needed something more robust: **minimist** first, then **cmd-ts**, then **oclif**. The bar kept rising. I wanted a CLI that was discoverable by someone being onboarded, easy to use without a manual, and easy to keep building on. - -Then I scrapped all of it and moved to **citty** and **@clack/prompts**, which fit what noorm actually does far better than any of the three. - -Around the same time I was experimenting with terminal interfaces and tried **Ink**. Testing it against the CLI settled something. Running commands means remembering commands. A screen that shows you your configs, your pending changes, and your schema while you decide is a better workflow than typing a verb and hoping you got the flags right, because the decision needs data in front of it and a CLI gives you none. - -That produced the split noorm still has. Anything interactive lives in the [TUI](/tui): setup wizards, config editing, secrets, browsing the schema. Anything automatable lives in the [CLI](/headless), where it belongs, so the same operations run headless in CI with JSON output. - - -## The questions that kept arriving - - -None of this was designed up front. It was excavated. Each row is a question I could not answer with what I had at the time. +Each row is a question I hit. | Question | What it became | |---|---| -| How do I run SQL? | A directory of files, executed in order | -| How do I undo one? | Forward and revert pairs, tracked per change | -| How do I test against it? | A real database with safety guards, not mocks | -| How do I express what DDL makes painful? | Templates that render SQL from YAML | -| How do I seed data? | Templates again, plus transfer between databases | -| Where does connection configuration live? | Configs you can export and hand to a teammate | -| How do I onboard someone onto four environments? | Stages, with required and optional variables | -| Where do team secrets live? | An encrypted vault table inside the database | -| What ran, by whom, and why did it break? | History that records the operator and the error | -| How do I re-run the idempotent objects? | Manifests | - -Three of those are worth more than a table row. - - -### Templates are not just per-environment strings - - -Rendering one file differently for dev and production is the obvious use. It was not the reason. - -Templates exist so seed data can be expressed and iterated on, and so YAML can become DDL that is miserable to write by hand. SQL Server's scheduled jobs are the clearest case: expressing a schedule semantically and rendering it into the DDL that creates the job beats hand-writing that DDL every time. The same goes for roles and permissions across stored procedures and views. A YAML list of who gets what is readable and diffable. The forty `GRANT` statements it renders into are neither. +| How do I run SQL? | [A folder of files](/getting-started/concepts), run in path order | +| How do I undo one? | [Forward and revert](/guide/changes/forward-revert) pairs | +| How do I test against it? | A real database behind a [`requireTest`](/reference/sdk) guard | +| How do I write painful DDL and seed data? | [Templates](/guide/sql-files/templates) that render SQL from YAML, and [transfer](/guide/database/transfer) | +| Where does connection configuration live? | [Configs](/guide/environments/configs) you export to a teammate | +| How do I onboard someone onto four environments? | [Stages](/guide/environments/stages) with required secrets | +| Where do team secrets live? | A [vault](/guide/environments/vault) table inside the database | +| What ran, who ran it, and why did it break? | [History](/guide/changes/history) with the operator and the error | +| How do I re-run objects meant to be re-run? | `.txt` [manifests](/guide/changes/overview) that run `db/sql/` files | -So the templating layer is where iteration happens, and SQL is the output rather than the source you maintain. +## Why is the interface split in two? -### History records why, not just what +noorm is the fifth attempt. I replaced each of the first three because I needed something more robust, one a new teammate could use without a manual. Then I moved to citty and @clack/prompts, and tried Ink. -Every applied change is recorded with its status, when it ran, **who ran it**, and the **error message** if it failed. That last one is the part even ORMs skip. +| Library, in the order I used it | What it is | In noorm today | +|---|---|---| +| minimist | argument parser | no | +| cmd-ts | type-driven argument parser | no | +| oclif | CLI framework | no | +| citty | CLI builder | the commands in `src/cli/` | +| @clack/prompts | interactive prompts | `noorm init`, `noorm change add` | +| Ink | React for the terminal | the TUI in `src/tui/` | -It matters when a change fails, you fix it, and you run it again. Something that broke during the failed attempt may not surface for days. A history with the actual error lets you trace back to which run went wrong instead of guessing. Recording the operator means a pattern of failures has a name attached to it. +Running commands means remembering commands. A screen that shows configs, pending changes, and schema while you decide beats typing a verb and hoping the flags are right. -The reason for a change almost never gets written down anywhere. That is what `changelog.md` in the change folder is for: the justification, next to the SQL it justifies. +`noorm ui` opens the [TUI](/tui). The other commands take flags, so they run in scripts and CI ([CLI reference](/headless)). -**Manifests** handle the objects that are meant to be re-run. Views, functions, stored procedures, and the templates that generate crons or permissions are all idempotent, so a manifest lists them and the change re-runs them. The point is that fixing a stored procedure stops being something that happens in a SQL editor at 11pm and starts being a tracked change like any other. +## Why templates? -### Configuration is something you hand to someone +Templates let seed data be written and iterated on, and let YAML become DDL that is miserable to write by hand. SQL Server scheduled jobs are the clearest case: a schedule written semantically renders into the DDL that creates the job. A YAML list of who gets what is readable and diffable, and the `GRANT` statements it renders into are not. -A config is exportable. Sharing one goes through the TUI, where export encrypts to the recipient's public key with X25519 key exchange and AES-256-GCM, and leaves credentials out, so the recipient supplies their own username and password on import. You can export local dev, local test, staging, and production, send all four, and the other person fills in only the parts that are theirs. (The CLI's `config export` is a different tool: plain JSON, credentials included, for your own backups.) -**Stages** make that prescriptive. Rather than describing your environments in a README, you define them once, and everyone gets the same shape with the details left blank. A stage declares its variables as required or optional, and a build fails with a clear error when a required one is missing instead of rendering something half-formed. Configs are typed `local` or `remote`, which is the distinction that decides how careful the tool should be. Losing a local dev database costs nothing. Production is a different question, which is what [access roles](/guide/environments/configs#access-roles) gate. +## Why does history record the operator and the error? -**The vault** covers what stages cannot: secrets the whole team needs. Those live encrypted in the database itself, in `noorm.vault` where the dialect has schemas and a `__noorm_vault__` table where it does not, so they distribute automatically rather than over Slack. They sit lowest in the precedence chain, so a local override always wins. +Every applied change records its status, when it ran, who ran it, and the error message when it failed. A failure you fixed and re-ran can surface days later. The recorded error shows which run to trace back to, and the operator puts a name on a pattern of failures. -## The Arrival of LLMs +`changelog.md` in the change folder holds the reason for the change, next to the SQL it justifies. A `.txt` manifest lists the views, functions, and stored procedures a change re-runs, so fixing a stored procedure is a tracked change. -The constraint changed. Writing SQL stopped being the slow part, so the slow part became reviewing SQL that an agent wrote quickly and confidently and sometimes wrongly. +## How do teams share environments? -That pushed the tool in three directions at once. -**Safeguards**, so a confident mistake cannot reach production: per-channel [access roles](/guide/environments/configs#access-roles), dry runs, checksums, and a typed confirmation on anything destructive. +A config exports from the TUI encrypted to the recipient's public key, without `user` and `password`, so the recipient supplies their own. [Configs](/guide/environments/configs#how-do-i-share-a-config-with-a-teammate) covers both export paths. -**Guidance**, so an agent writes code that matches the conventions here instead of guessing at them. That is what the [agent skill](/getting-started/installation) is for. +A template that reads an unset secret fails with `Secret "KEY" not found (searched: config-local, global-local, vault)`. The vault keeps team secrets encrypted in the database (`noorm.vault` on PostgreSQL and SQL Server, `__noorm_vault__` on MySQL and SQLite) at the lowest precedence, so a local value wins. -**Planning**, because the cheapest moment to catch a bad schema is before it exists. -That last one turned into a separate tool. See [Information modeling](/modeling/). +## What changed when LLMs arrived? -## Where to start +Writing SQL stopped being the slow part. Reviewing SQL that an agent wrote quickly, confidently, and sometimes wrongly became the slow part. Three additions act before the agent writes, while it writes, and before its SQL runs. +```mermaid +flowchart TD + M["ignatius model"] -->|plan| A[AI agent] + K["noorm skill"] -->|conventions| A + A -->|SQL| G["access role, dry run,
typed confirmation"] + G --> DB[(database)] +``` -
+- **ignatius model**: the cheapest moment to catch a bad schema is before it exists, which became a separate tool, [ignatius](/modeling/). +- **noorm skill**: the [agent skill](/getting-started/installation#install-the-agent-skill-optional) teaches an agent this project's conventions. +- **access role, dry run, typed confirmation**: per-channel [access roles](/guide/environments/configs#access-roles), `--dry-run`, and a typed confirmation on destructive operations. -[**Installation**](/getting-started/installation) -Get noorm installed and running. -[**Concepts**](/getting-started/concepts) -How SQL files and changes divide the work. +## Next -[**Information modeling**](/modeling/) -Design the data model before you write the schema. -
+- [Installation](/getting-started/installation) +- [Concepts](/getting-started/concepts) +- [Information modeling](/modeling/) diff --git a/examples/llm-memory-db-mssql/CHANGELOG.md b/examples/llm-memory-db-mssql/CHANGELOG.md index 71485e30..03a41957 100644 --- a/examples/llm-memory-db-mssql/CHANGELOG.md +++ b/examples/llm-memory-db-mssql/CHANGELOG.md @@ -1,5 +1,12 @@ # @noormdev/example-llm-memory-db-mssql +## 0.0.11 + +### Patch Changes + +- Updated dependencies [0fcc257] + - @noormdev/sdk@1.5.1 + ## 0.0.10 ### Patch Changes diff --git a/examples/llm-memory-db-mssql/package.json b/examples/llm-memory-db-mssql/package.json index 97f73eec..fa936653 100644 --- a/examples/llm-memory-db-mssql/package.json +++ b/examples/llm-memory-db-mssql/package.json @@ -1,6 +1,6 @@ { "name": "@noormdev/example-llm-memory-db-mssql", - "version": "0.0.10", + "version": "0.0.11", "private": true, "type": "module", "description": "LLM-memory schema + SDK example exercising NoORM against MSSQL with TVPs", diff --git a/examples/llm-memory-db-pg/CHANGELOG.md b/examples/llm-memory-db-pg/CHANGELOG.md index fdf51886..9b026644 100644 --- a/examples/llm-memory-db-pg/CHANGELOG.md +++ b/examples/llm-memory-db-pg/CHANGELOG.md @@ -1,5 +1,12 @@ # @noormdev/example-llm-memory-db-pg +## 0.0.11 + +### Patch Changes + +- Updated dependencies [0fcc257] + - @noormdev/sdk@1.5.1 + ## 0.0.10 ### Patch Changes diff --git a/examples/llm-memory-db-pg/package.json b/examples/llm-memory-db-pg/package.json index ee4afa3a..64c37e13 100644 --- a/examples/llm-memory-db-pg/package.json +++ b/examples/llm-memory-db-pg/package.json @@ -1,6 +1,6 @@ { "name": "@noormdev/example-llm-memory-db-pg", - "version": "0.0.10", + "version": "0.0.11", "private": true, "type": "module", "description": "Full-surface noorm example: LLM memory + task tracker schema, SDK, and tests against PostgreSQL", diff --git a/examples/todo-db/CHANGELOG.md b/examples/todo-db/CHANGELOG.md index 9899ae75..f2bf738f 100644 --- a/examples/todo-db/CHANGELOG.md +++ b/examples/todo-db/CHANGELOG.md @@ -1,5 +1,12 @@ # @noormdev/example-todo-db +## 0.0.11 + +### Patch Changes + +- Updated dependencies [0fcc257] + - @noormdev/sdk@1.5.1 + ## 0.0.10 ### Patch Changes diff --git a/examples/todo-db/package.json b/examples/todo-db/package.json index f963b3ac..02835bc2 100644 --- a/examples/todo-db/package.json +++ b/examples/todo-db/package.json @@ -1,6 +1,6 @@ { "name": "@noormdev/example-todo-db", - "version": "0.0.10", + "version": "0.0.11", "private": true, "type": "module", "description": "SDK integration tests exercising the todo-db example schema", diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 5a4de9e0..244943ef 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,11 @@ # @noormdev/cli +## 1.5.1 + +### Patch Changes + +- 0fcc257: Decide which files to skip with one or two tracking queries per batch instead of one or two per file. Skipped files no longer write execution rows, and a build where nothing changed writes no operation record. The SDK's `run.file(path, { preview: true })` now renders without executing. The TUI's re-run confirmation now finds earlier runs. + ## 1.5.0 ### Minor Changes diff --git a/packages/cli/package.json b/packages/cli/package.json index bd7f791e..922fbfb9 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "@noormdev/cli", - "version": "1.5.0", + "version": "1.5.1", "description": "Database schema & changeset manager CLI", "type": "module", "bin": { diff --git a/packages/sdk/CHANGELOG.md b/packages/sdk/CHANGELOG.md index 589a5db7..0e2d300a 100644 --- a/packages/sdk/CHANGELOG.md +++ b/packages/sdk/CHANGELOG.md @@ -1,5 +1,11 @@ # @noormdev/sdk +## 1.5.1 + +### Patch Changes + +- 0fcc257: Decide which files to skip with one or two tracking queries per batch instead of one or two per file. Skipped files no longer write execution rows, and a build where nothing changed writes no operation record. The SDK's `run.file(path, { preview: true })` now renders without executing. The TUI's re-run confirmation now finds earlier runs. + ## 1.5.0 ### Minor Changes diff --git a/packages/sdk/package.json b/packages/sdk/package.json index 3c9c1e98..1de9afbc 100644 --- a/packages/sdk/package.json +++ b/packages/sdk/package.json @@ -1,6 +1,6 @@ { "name": "@noormdev/sdk", - "version": "1.5.0", + "version": "1.5.1", "description": "Database schema & changeset manager SDK", "type": "module", "main": "./dist/index.js", diff --git a/skills/noorm/SKILL.md b/skills/noorm/SKILL.md index 4c527245..51e48999 100644 --- a/skills/noorm/SKILL.md +++ b/skills/noorm/SKILL.md @@ -1,6 +1,6 @@ --- name: noorm -description: Guide for building applications and tests with the NoORM database SDK (@noormdev/sdk) and composing CLI headless commands for CI/CD pipelines. Use this skill whenever the user imports @noormdev/sdk, calls createContext(), writes Kysely queries in a noorm-managed project, writes database integration tests, sets up CI/CD with noorm commands, manages database schemas or migrations, runs SQL files through noorm, transfers data between databases, manages vault secrets, or discusses noorm in any development context. Also trigger when you see noorm environment variables (NOORM_*), .sql.tmpl template files, or change/revert directory structures. Trigger for any database task in a noorm-managed project — the SDK conventions (SDK methods throw named errors, `attempt()` wraps deliberately, no try-catch, test safety guards, Kysely patterns) and CLI flags are non-obvious and critical to get right. +description: Guide for building applications and tests with the NoORM database SDK (@noormdev/sdk) and composing CLI headless commands for CI/CD pipelines. Use this skill whenever the user imports @noormdev/sdk, calls createContext(), writes Kysely queries in a noorm-managed project, writes database integration tests, sets up CI/CD with noorm commands, manages database schemas or migrations, runs SQL files through noorm, transfers data between databases, manages vault secrets, or discusses noorm in any development context. Also trigger on noorm environment variables (NOORM_*), .sql.tmpl templates, .txt change manifests, a `.noorm/` directory, `db/sql` or `db/changes` folders, change/revert directories, and any task that adds, alters, or drops a table, column, view, function, or seed data in a noorm project. Trigger for any database task in a noorm-managed project: the SDK conventions (named errors, no try-catch, test safety guards) and CLI flags are non-obvious. --- # NoORM @@ -23,9 +23,34 @@ Database schema and change manager with two interfaces: | Managing noorm locks (tool-level coordination, not DB engine locks), vault secrets, or data transfers via CLI | `references/cli.md` | | Configuring `.noorm/settings.yml` or defining stages | `references/config.md` | | Setting up a new noorm project or understanding config structure | `references/config.md` | -| Writing `.sql.tmpl` template files | `references/templates.md` | +| Adding or changing a table, column, view, function, or seed data | `references/schema-files.md` | +| Deciding whether SQL goes in `sql/` or `changes/`, writing a change, or writing a `.txt` manifest | `references/schema-files.md` | +| Writing `.sql.tmpl` template files | `references/templates.md` (and **Templates** in `references/schema-files.md` for when to use one) | | Combining SDK code with CLI operations | Read SDK + CLI references | +## Project Layout + + +`noorm init` creates this: + + + db/ + ├── sql/ what the schema IS. `noorm run build` creates a database from nothing with these files + ├── changes/ how an EXISTING database reaches that state. `noorm change ff` applies them + ├── AGENTS.md the user's own steering for this directory. Read it first + └── CLAUDE.md imports AGENTS.md + + +The real locations are `paths.sql` and `paths.changes` in `.noorm/settings.yml`. Projects created before the `db/` layout have `./sql` and `./changes` written in their `paths`; a settings file with no `paths` key means `./db/sql` and `./db/changes`. Read the settings before creating a file, and never add a second `sql/` or `changes/` beside the configured ones. + +Four rules that every schema edit follows (details and examples in `references/schema-files.md`): + +1. **`sql/` holds the current state and nothing else.** `CREATE` statements and seeds. Each column and constraint is declared once, inline in its `CREATE TABLE`. No `ALTER`, `DROP`, backfill, `IF NOT EXISTS` guard, or history comment. +2. **`changes/` holds every step that moves an existing database to that state.** `ALTER`, `DROP`, backfills, data fixes, written to be safe to run twice (a built database's first `ff` replays every change), each with a `revert/`. A change that has been applied is never edited; add a new one. +3. **Every edit to `sql/` ships with its change.** Edit `sql/` first, then write the change that brings an already-built database to the same state. Existing databases move with `noorm change ff`, never `noorm run build`. +4. **Reuse, do not copy.** A change re-creates a view, function, procedure, or trigger by listing its `sql/` file in a `.txt` manifest (paths relative to `paths.sql`, one per line). Repeated or data-driven SQL (seed rows, grants, per-table triggers, per-environment values, secrets) is a `.sql.tmpl` plus a data file. + + ## Shared Conventions These apply to both SDK and CLI usage. @@ -80,3 +105,14 @@ These are the patterns LLMs get wrong most often with noorm: | `Procs = { name: ArgsType }` | `Procs = { name: [ArgsType, ReturnType] }` | Tuple format enables return type inference | | Assume every `ctx.noorm.*` call returns a tuple | SDK methods throw named errors — wrap with `attempt()` only when you'll do something with the error | Unhandled throws stop the caller immediately; that's the contract | | Use raw Kysely without noorm | Use `ctx.kysely` from a noorm Context | Misses config resolution, checksums, change tracking | +| `ALTER TABLE`, `DROP`, or a backfill in a `sql/` file | Put it in `changes//change/` and write the end state in `sql/` | `sql/` builds an empty database. History belongs to changes | +| `IF NOT EXISTS` on a `sql/` `CREATE TABLE` to make it re-runnable | Remove the guard. Guards go in changes | A changed file re-runs against the existing table and the guard makes the edit silently do nothing | +| Edit `sql/` and stop | Add the change that brings an existing database to the same state | A built database never receives the edit | +| Run `noorm run build` on a database that already has data | `noorm change ff` | Build only re-runs changed files, and a changed `CREATE TABLE` cannot alter an existing table | +| Copy a view or function into `change/001_*.sql` | List its `sql/` file in a `.txt` manifest | The copy drifts from `sql/`. Manifest paths are relative to `paths.sql` | +| Reference a `CREATE TABLE` file from a manifest | Copy the `CREATE TABLE IF NOT EXISTS` into the change | A manifest runs the file's current content, so a pending or replayed change executes today's `CREATE TABLE` | +| Edit a change that was already applied | Add a new change | Databases that already ran it keep the old body and the rest run the new one | +| Hand-write a list of `INSERT`s or `GRANT`s | `.sql.tmpl` with a `roles.yml`-style data file beside it | One data file, one loop, `$.quote()` on every value | +| Put a secret or a per-environment value in plain `.sql` | `.sql.tmpl` with `$.secrets` / `$.config` | Plain files are identical in every environment | +| Create `sql/` at the repo root when `paths.sql` is `./db/sql` | Read `.noorm/settings.yml` first | noorm never reads the stray folder, so its files never run | +| `build.include: [sql/01_tables]` | `build.include: [01_tables]` | Entries are relative to `paths.sql`. A wrong entry matches nothing and still exits 0 | diff --git a/skills/noorm/references/cli.md b/skills/noorm/references/cli.md index 8e7456a7..4fff1feb 100644 --- a/skills/noorm/references/cli.md +++ b/skills/noorm/references/cli.md @@ -62,8 +62,8 @@ When all three identity vars are set at process startup, every `noorm` command i ### Paths ```bash -NOORM_PATHS_SQL # Schema directory (default: ./sql) -NOORM_PATHS_CHANGES # Changes directory (default: ./changes) +NOORM_PATHS_SQL # Schema directory (default: ./db/sql) +NOORM_PATHS_CHANGES # Changes directory (default: ./db/changes) ``` ### Behavior @@ -83,7 +83,7 @@ NOORM_DEBUG # Enable debug logging ### init -Bootstrap a new noorm project interactively. Creates identity (if needed), project structure (`sql/`, `changes/`, `.noorm/`), `settings.yml`, and encrypted `state.enc`. +Bootstrap a new noorm project interactively. Creates identity (if needed), project structure (`db/sql/`, `db/changes/`, `.noorm/`), `settings.yml`, and encrypted `state.enc`. ```bash noorm init # Interactive — requires TTY @@ -310,9 +310,9 @@ noorm run build --dry-run # Preview without executing Execute a single SQL or `.sql.tmpl` file. ```bash -noorm run file sql/01_tables/001_users.sql +noorm run file db/sql/01_tables/001_users.sql noorm run file seed.sql.tmpl -noorm run file sql/init.sql --json +noorm run file db/sql/init.sql --json ``` **JSON:** `{ "filepath": "seed.sql", "status": "success", "durationMs": 45 }` @@ -324,7 +324,7 @@ Status is `"success"` or `"skipped"` (unchanged checksum). Execute all SQL files in a directory in alphabetical order. ```bash -noorm run dir sql/01_tables/ +noorm run dir db/sql/01_tables/ noorm run dir seeds/ ``` @@ -333,15 +333,15 @@ noorm run dir seeds/ Show the template context for a `.sql.tmpl` file without rendering it. Lists available data files, helpers, built-in functions, config keys, and secret counts. ```bash -noorm run inspect sql/users/001_create.sql.tmpl -noorm run inspect sql/core/Crons.sql.tmpl --json +noorm run inspect db/sql/users/001_create.sql.tmpl +noorm run inspect db/sql/core/Crons.sql.tmpl --json ``` **JSON:** ```json { - "filepath": "sql/users/001_create.sql.tmpl", + "filepath": "db/sql/users/001_create.sql.tmpl", "context": { "dataFiles": [{ "key": "roles", "type": "Array [3]" }], "helpers": [{ "key": "padId", "type": "Function" }], @@ -359,10 +359,10 @@ noorm run inspect sql/core/Crons.sql.tmpl --json Render a `.sql.tmpl` file and output the resulting SQL. Does **not** execute against the database. ```bash -noorm run preview sql/schema.sql.tmpl # Raw SQL to stdout -noorm run preview sql/schema.sql.tmpl > rendered.sql # Pipe to file -noorm run preview sql/seed.sql.tmpl --json # JSON envelope -noorm run preview sql/migrations/002.sql.tmpl --config staging +noorm run preview db/sql/schema.sql.tmpl # Raw SQL to stdout +noorm run preview db/sql/schema.sql.tmpl > rendered.sql # Pipe to file +noorm run preview db/sql/seed.sql.tmpl --json # JSON envelope +noorm run preview db/sql/migrations/002.sql.tmpl --config staging ``` **JSON:** `{ "filepath": "...", "sql": "CREATE TABLE ...", "durationMs": 12 }` @@ -379,9 +379,9 @@ Pickers filter their list to whatever the command can act on: | Command | TTY picker lists | |---------|------------------| -| `change edit` | every folder in `changes/` | +| `change edit` | every folder in `db/changes/` | | `change add` | — (prompts for free-form description via `p.text`) | -| `change rm` | every folder in `changes/` | +| `change rm` | every folder in `db/changes/` | | `change run` | pending / reverted changes | | `change revert` | successfully applied changes | | `change rewind` | successfully applied changes | @@ -490,7 +490,7 @@ noorm change add add-users-table noorm change add create-audit-log --json ``` -**JSON:** `{ "name": "2024-04-17-add-users-table", "path": "/project/changes/2024-04-17-add-users-table" }` +**JSON:** `{ "name": "2024-04-17-add-users-table", "path": "/project/db/changes/2024-04-17-add-users-table" }` ### change edit @@ -982,13 +982,13 @@ fi ### Render template and pipe to external tool ```bash -noorm run preview sql/hotfix.sql.tmpl | psql -h localhost -d myapp +noorm run preview db/sql/hotfix.sql.tmpl | psql -h localhost -d myapp ``` ### Check template context before rendering ```bash -files=$(noorm run inspect sql/seed.sql.tmpl --json | jq '.context.dataFiles | length') +files=$(noorm run inspect db/sql/seed.sql.tmpl --json | jq '.context.dataFiles | length') echo "Template has $files data files available" ``` diff --git a/skills/noorm/references/config.md b/skills/noorm/references/config.md index 54d37343..0822af39 100644 --- a/skills/noorm/references/config.md +++ b/skills/noorm/references/config.md @@ -10,7 +10,7 @@ Project settings live in `.noorm/settings.yml`. Configs (database connections) a build: # Folders to include. Paths are relative to `paths.sql`, NOT the project - # root — with `paths.sql: ./sql`, `tables` means `./sql/tables`. + # root — with `paths.sql: ./db/sql`, `tables` means `./db/sql/tables`. # Empty list means all files under `paths.sql`. include: - tables @@ -26,8 +26,8 @@ build: # Override default directory locations (relative to project root). paths: - sql: ./sql # Schema SQL files (default: ./sql) - changes: ./changes # Change directories (default: ./changes) + sql: ./db/sql # Schema SQL files (default: ./db/sql) + changes: ./db/changes # Change directories (default: ./db/changes) # ── Rules ──────────────────────────────────────────────────── # Conditional include/exclude based on the active config's properties. @@ -128,16 +128,16 @@ secrets: This is the single most common misconfiguration, and it fails silently. -Entries in `build.include`, `build.exclude`, and `rules[].include` / `rules[].exclude` are matched against each file's path **relative to `paths.sql`** — never the project root. With the default `paths.sql: ./sql`: +Entries in `build.include`, `build.exclude`, and `rules[].include` / `rules[].exclude` are matched against each file's path **relative to `paths.sql`** — never the project root. With the `paths.sql: ./db/sql` that `noorm init` writes: ```yaml paths: - sql: ./sql + sql: ./db/sql build: include: - - 01_tables # correct -> ./sql/01_tables - - sql/01_tables # WRONG -> ./sql/sql/01_tables, matches nothing + - 01_tables # correct -> ./db/sql/01_tables + - sql/01_tables # WRONG -> ./db/sql/sql/01_tables, matches nothing ``` A non-matching entry is not an error. The build reports `"status": "success"` with `"filesRun": 0` and exits `0`, so a misconfigured `include` looks exactly like a successful build — the schema is simply never created, and the failure surfaces later as a missing table. @@ -155,8 +155,8 @@ Use bare, unprefixed segment names. Nested paths (`06_seeds/cron`) are supported | Field | Type | Default | Description | |---|---|---|---| -| `sql` | `string` | `./sql` | Schema SQL directory | -| `changes` | `string` | `./changes` | Change (migration) directory | +| `sql` | `string` | `./db/sql` | Schema SQL directory | +| `changes` | `string` | `./db/changes` | Change directory | ### `rules[]` @@ -275,8 +275,8 @@ For a simple project with just dev and test: ```yaml paths: - sql: ./sql - changes: ./changes + sql: ./db/sql + changes: ./db/changes stages: dev: diff --git a/skills/noorm/references/schema-files.md b/skills/noorm/references/schema-files.md new file mode 100644 index 00000000..56675f09 --- /dev/null +++ b/skills/noorm/references/schema-files.md @@ -0,0 +1,281 @@ +# Schema Files and Changes + + +Where SQL goes in a noorm project and why. Read this before adding a table, altering a column, writing seed data, or creating a change. + + +## Table of Contents + +1. [Layout](#layout) +2. [What each directory is for](#what-each-directory-is-for) +3. [Which directory gets this edit](#which-directory-gets-this-edit) +4. [Anatomy of a change](#anatomy-of-a-change) +5. [Manifests](#manifests) +6. [Templates](#templates) +7. [Workflow for a schema edit](#workflow-for-a-schema-edit) +8. [Worked example](#worked-example) +9. [Checks before you finish](#checks-before-you-finish) + + +--- + + +## Layout + + + project/ + ├── .noorm/ + │ └── settings.yml # paths.sql, paths.changes, build.include/exclude + └── db/ + ├── sql/ # what the schema IS: the current state + ├── changes/ # how an EXISTING database gets to that state + ├── AGENTS.md # the user's steering for this directory; read it first + └── CLAUDE.md # imports AGENTS.md + + +`noorm init` writes `paths.sql: ./db/sql` and `paths.changes: ./db/changes` into `.noorm/settings.yml`. Projects created before that have `./sql` and `./changes` written in their `paths`, and a settings file with no `paths` key means the `db/` paths. + +Read `.noorm/settings.yml` before creating or moving a file. Never create a second `sql/` or `changes/` beside the configured ones: noorm does not read them, so their files never run. + + +--- + + +## What each directory is for + + +| | `sql/` | `changes/` | +|---|---|---| +| Describes | What the schema is now | How a database that already exists gets there | +| Run by | `noorm run build` | `noorm change ff`, `change run`, `change revert` | +| Target | An empty database | A database in the state before the change | +| Executes | Every file whose rendered checksum changed, in sorted path order | Each change once, recorded in the database | +| Keeps history | No | Yes, that is its job | + + +### Rules for `sql/` + +- Only statements that create the current state: `CREATE TABLE`, `CREATE OR REPLACE` views, functions, procedures and triggers (`CREATE OR ALTER` on mssql), `CREATE INDEX`, seed `INSERT`s, grants. +- Declare each column and constraint once, inline in its `CREATE TABLE`. Adding a column means editing that statement. +- Never put in `sql/`: + - `ALTER` of a table this directory creates + - `DROP`, or a file that exists only to retire an object + - `UPDATE` or backfills of existing rows + - `IF NOT EXISTS` / `DROP IF EXISTS` guards added to survive a second run + - comments about history ("added in v2") + +A table whose definition is spread over a `CREATE` and later `ALTER`s cannot be read without replaying them. A guard is worse: when the `CREATE TABLE` file changes, `build` re-runs it, and `CREATE TABLE IF NOT EXISTS` against the existing table does nothing, so the edit silently never lands. + +`sql/` is for fresh databases: CI, tests, a new developer's machine. Do not run `noorm run build` against a database that already holds data. A changed `CREATE TABLE` file re-executes against the existing table, where it either errors or no-ops and never alters anything. Existing databases move with `noorm change ff`. + + +### Rules for `changes/` + +- Everything that turns an existing database into the current state: `ALTER`, `DROP`, backfills, data fixes, type changes, index and constraint additions on live tables. +- Make every statement safe to run twice (see [Guards by dialect](#guards-by-dialect); the examples below are PostgreSQL). `noorm run build` records no change history, so the first `change ff` on a freshly built database replays every change in the folder, and a statement that is not safe to run twice fails there. +- Write a `revert/` for each change. +- Never edit a change that has been applied anywhere. Databases that already ran it keep the old body and the rest run the new one, so environments drift apart. Add a new change instead. `change ff` skips applied changes; `noorm change run ` re-executes an edited one on the one database you point it at. + + +### Guards by dialect + + +| Statement | PostgreSQL | MySQL 8 | SQL Server | SQLite | +|---|---|---|---|---| +| Create table | `CREATE TABLE IF NOT EXISTS` | `CREATE TABLE IF NOT EXISTS` | `IF OBJECT_ID('dbo.t', 'U') IS NULL CREATE TABLE ...` | `CREATE TABLE IF NOT EXISTS` | +| Add column | `ADD COLUMN IF NOT EXISTS` | Check `information_schema.columns` first | `IF COL_LENGTH('dbo.t', 'c') IS NULL ALTER TABLE ... ADD ...` | No guard | +| Drop column | `DROP COLUMN IF EXISTS` | Check `information_schema.columns` first | `DROP COLUMN IF EXISTS` | No guard | +| Create index | `CREATE INDEX IF NOT EXISTS` | Check `information_schema.statistics` first | `IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE ...)` | `CREATE INDEX IF NOT EXISTS` | +| Drop table or view | `DROP ... IF EXISTS` | `DROP ... IF EXISTS` | `DROP ... IF EXISTS` | `DROP ... IF EXISTS` | + +MySQL and SQL Server reject the PostgreSQL forms of `ADD COLUMN IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` as syntax errors. SQLite has no guard for `ADD COLUMN`, so a replayed change that adds one fails there. + + +--- + + +## Which directory gets this edit + + +Every edit to `sql/` that changes what a built database contains ships with a change that brings an already-built database to the same state. Edit `sql/` first (the target), then write the change. + + +| Edit | `sql/` | `changes//change/` | +|---|---|---| +| New table | `CREATE TABLE` | The same `CREATE TABLE IF NOT EXISTS`, inline. Do not reference the table's file from a manifest | +| New column | Add it to that table's `CREATE TABLE` | `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`, plus a backfill if rows need a value | +| Drop or rename a column | Remove or rename it in the `CREATE TABLE` | `ALTER TABLE ... DROP / RENAME COLUMN` | +| New or edited view, function, procedure, trigger | Edit its `CREATE OR REPLACE` file | A `.txt` manifest naming that file | +| New index | `CREATE INDEX` | `CREATE INDEX IF NOT EXISTS` | +| New seed row | Add it to the seed file or its data file | An idempotent `INSERT` for the rows existing databases lack | +| Bad data in existing rows | Nothing | `UPDATE` | +| Remove an object | Delete its file | `DROP ... IF EXISTS` | + + +--- + + +## Anatomy of a change + + + db/changes/2026-10-19-add-user-bio/ + ├── change/ + │ ├── 001_add-bio-column.sql + │ └── 002_rebuild-views.txt + ├── revert/ + │ └── 001_drop-bio-column.sql + └── changelog.md + + +- Scaffold with `noorm change add `. It creates the dated folder under `paths.changes` with comment-only stubs. Replace the comments with SQL. +- `change/` and `revert/` hold `.sql`, `.sql.tmpl` and `.txt` files. They run in filename order, so keep the `NNN_` prefix zero-padded to three digits. +- `revert/` also runs in forward order (`001`, then `002`). Number the revert files in the order the undo has to happen. +- `changelog.md` says why the change exists. The diff already shows what it does. +- A change whose files contain only comments or placeholders is rejected. +- A folder under `changes/` with neither `change/` nor `revert/` is not a change and is ignored, so shared fragments and archives can live beside the real ones. + + +--- + + +## Manifests + + +A `.txt` file in `change/` or `revert/` runs existing `sql/` files in place instead of copying their SQL into the change. + + + # db/changes/2026-10-19-widen-name/change/003_rebuild-views.txt + views/active_users.sql + functions/get_user_stats.sql + + +- One path per line, relative to `paths.sql` (`views/active_users.sql`, not `db/sql/views/active_users.sql`). Blank lines and lines starting with `#` are ignored. +- Lines run in file order. The manifest sorts among the other change files by filename, so `001_rebuild.txt` runs before `002_alter.sql`. +- A missing file makes the change unreadable. `noorm change ff` stops at that change and exits non-zero (1, or 3 when earlier changes in the batch applied) with the missing path in the message, `change list` shows it as `failed`, and `change run ` reports the same error. If the change already ran, it keeps its status and `ff` only warns. A manifest with no path lines is treated the same way. +- A manifest runs the referenced file's current content, not its content when the change was written. A change still pending on a lagging database, or replayed after a build, executes today's definition. +- Because of that, reference only objects that are safe to run repeatedly: `CREATE OR REPLACE` views, functions, procedures, triggers. Never reference a `CREATE TABLE` file: replayed, it hits a table that already exists. +- A path listed more than once across a change's manifests runs once, at its first position. +- A manifest in `revert/` rebuilds the current `sql/` definition. It cannot restore an older definition of a view or function; write that older `CREATE OR REPLACE` inline in the revert file. + + +Use a manifest whenever a change has to re-create an object that `sql/` already defines. Copying the DDL into the change leaves two definitions that drift apart. + + +A table change that a dependent view blocks: + + + change/ + ├── 001_drop-dependent-views.sql # DROP VIEW IF EXISTS active_users; + ├── 002_widen-name-column.sql # ALTER TABLE users ALTER COLUMN name TYPE VARCHAR(200); + └── 003_rebuild-views.txt # views/active_users.sql + + +--- + + +## Templates + + +A `.sql.tmpl` file is rendered by Eta before it runs, with any `.json`, `.yml`, `.csv` or `.dt` file in the same directory loaded onto `$`. Syntax, helpers and context are in `references/templates.md`. + +Reach for a template when any of these holds: + +| Situation | Template shape | +|---|---| +| Rows come from data: lookups, enums, roles, feature flags, cron schedules | Keep the data in `roles.yml` beside the template and loop over `$.roles` | +| Three or more statements differ only by a name or value: grants, audit triggers, per-table indexes | Loop over a list in a data file, or call a function from `$helpers.ts` | +| A value differs by environment: stage, test flag, host | Branch on `$.config.name` (a config named `prod` is the `prod` stage) or `$.config.isTest`, or read `$.env` | +| A value is secret | Read `$.secrets.KEY`. Never write a secret into a plain `.sql` file | +| The same fragment appears in several files | `$.include()` it | + +Use plain `.sql` when none applies. A template with no variables is noise. + +- Quote every interpolated value with `$.quote()`. Data files are not trusted input. +- Templates work in `sql/` and in `change/` and `revert/`. In a change folder a template reads the data files beside it, and those `.json`, `.yml` and `.csv` files are not run as change files. +- In `sql/` the checksum covers the rendered output, so editing a data file re-runs the template on the next build. `$.now()` and `$.uuid()` change the output every build, so a file that calls them runs every build. A change's data files sit outside the change's checksum. +- Check the output with `noorm run preview ` before running it. + + +A hand-written list like this: + + + INSERT INTO role (name) VALUES ('admin'); + INSERT INTO role (name) VALUES ('editor'); + INSERT INTO role (name) VALUES ('viewer'); + + +becomes data plus a template: + + + # db/sql/10_seeds/roles.yml + - admin + - editor + - viewer + + -- db/sql/10_seeds/001_roles.sql.tmpl + -- {% for (const name of $.roles) { %} + INSERT INTO role (name) VALUES ({%~ $.quote(name) %}); + -- {% } %} + + +--- + + +## Workflow for a schema edit + + +1. Read `.noorm/settings.yml` for `paths.sql` and `paths.changes`. Read `db/AGENTS.md` for the user's own rules. +2. Edit the target state in `sql/`. Columns and constraints go inline in `CREATE TABLE`. +3. `noorm change add ` to scaffold the change folder. +4. Write `change/NNN_*.sql`: the `ALTER`s, backfills and guards that move an existing database to the new `sql/` state. +5. Re-create dependent views, functions, procedures and triggers with a `.txt` manifest, not a copy. +6. Write `revert/NNN_*.sql` that undoes the change, numbered in the order the undo runs. +7. Fill in `changelog.md`. +8. Verify (see below), then commit `sql/` and the change together. + + +--- + + +## Worked example + + +Add a `bio` column to `users` (PostgreSQL). + + + -- db/sql/01_tables/001_users.sql + CREATE TABLE users ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL, + bio TEXT NOT NULL DEFAULT '' + ); + + + -- db/changes/2026-10-19-add-user-bio/change/001_add-bio-column.sql + ALTER TABLE users ADD COLUMN IF NOT EXISTS bio TEXT NOT NULL DEFAULT ''; + + + -- db/changes/2026-10-19-add-user-bio/revert/001_drop-bio-column.sql + ALTER TABLE users DROP COLUMN IF EXISTS bio; + + + # db/changes/2026-10-19-add-user-bio/changelog.md + Profiles need a short bio. Default is empty so existing rows stay valid. + + +The `ALTER` is in the change and the column is declared once, inline, in `sql/`. The reverse placements are the usual mistake: the `ALTER` in `sql/`, or the column only in the change. + + +--- + + +## Checks before you finish + + +- A fresh database built from `sql/` and a database built from the previous commit then moved with `noorm change ff` end up with the same schema. This is the test of the pairing rule. +- `noorm run build --dry-run` and `noorm change ff --dry-run` render every file, templates and manifests included, to `tmp/` and execute nothing. `change ff --dry-run` still reads the change tracking tables to decide what is pending. +- `sql/` contains no `ALTER`, `DROP`, backfill or guard that you added for this edit. +- No `CREATE OR REPLACE` object was copied into a change when a manifest would do. +- Seed lists, grants and repeated statements went through a template. +- No file landed outside the configured `paths`. +- No change shows `failed` in `noorm change list` because of a manifest that names a file `sql/` no longer has. Deleting or renaming a `sql/` file breaks every unapplied change that lists it. diff --git a/skills/noorm/references/sdk.md b/skills/noorm/references/sdk.md index faf416c9..ee0d06cb 100644 --- a/skills/noorm/references/sdk.md +++ b/skills/noorm/references/sdk.md @@ -309,10 +309,10 @@ const history = await ctx.noorm.changes.history(20); // Execution log ```typescript // Discovery -const files = await ctx.noorm.run.discover('sql/'); +const files = await ctx.noorm.run.discover('db/sql/'); // Preview (render templates without executing) -const previews = await ctx.noorm.run.preview(['sql/001.sql', 'sql/002.sql']); +const previews = await ctx.noorm.run.preview(['db/sql/001.sql', 'db/sql/002.sql']); // Execution await ctx.noorm.run.file('seeds/test-data.sql'); @@ -404,7 +404,7 @@ const apiKey = ctx.noorm.secrets.get('API_KEY'); // Returns undefined if not set Render `.sql.tmpl` files without executing — useful for previewing generated SQL. ```typescript -const result = await ctx.noorm.templates.render('sql/001_users.sql.tmpl'); +const result = await ctx.noorm.templates.render('db/sql/001_users.sql.tmpl'); // result: { name, content, error?, durationMs? } ``` @@ -449,7 +449,7 @@ const result = await ctx.noorm.dt.importFile('./exports/users.dtz', { ### utils ```typescript -const checksum = await ctx.noorm.utils.checksum('sql/001.sql'); +const checksum = await ctx.noorm.utils.checksum('db/sql/001.sql'); const { ok, error } = await ctx.noorm.utils.testConnection(); ``` diff --git a/skills/noorm/references/templates.md b/skills/noorm/references/templates.md index a944ab5d..34177c2c 100644 --- a/skills/noorm/references/templates.md +++ b/skills/noorm/references/templates.md @@ -1,6 +1,6 @@ # NoORM Templates Reference -Writing `.sql.tmpl` files — syntax, context API, secrets, environment variables, data files, and helpers. +Writing `.sql.tmpl` files — syntax, context API, secrets, environment variables, data files, and helpers. For when a template is the right choice over plain SQL, and where it goes (`sql/` or a change), see **Templates** in `schema-files.md`. ## Table of Contents @@ -64,8 +64,7 @@ The active database configuration object: | Key | Example | Description | |---|---|---| -| `$.config.name` | `'dev'` | Config name | -| `$.config.stage` | `'prod'` | Stage name | +| `$.config.name` | `'dev'` | Config name. A config links to the stage of the same name | | `$.config.isTest` | `true` | Whether config is marked as test | | `$.config.connection.database` | `'myapp'` | Database name | | `$.config.connection.host` | `'localhost'` | Database host | @@ -116,7 +115,7 @@ Combine with config for environment-aware templates: ```sql -- monitoring.sql.tmpl -{% if ($.config.stage === 'prod') { %} +{% if ($.config.name === 'prod') { %} -- Production: use real monitoring endpoint INSERT INTO settings (key, value) VALUES ('monitor_url', {%~ $.quote($.env.MONITOR_URL) %}); @@ -184,7 +183,7 @@ Include another SQL file. Path is relative to the current template. **Async — If the included file is `.sql.tmpl`, it is recursively rendered with the same context. ```sql --- changes/2025-01-15-setup/change/001_schema.sql.tmpl +-- db/changes/2025-01-15-setup/change/001_schema.sql.tmpl {%~ await $.include('../lib/core_tables.sql') %} {%~ await $.include('../lib/audit_triggers.sql.tmpl') %} ``` @@ -198,7 +197,7 @@ Path traversal outside the project root is blocked. Any supported file in the same directory as the template is automatically loaded onto `$`: ``` -sql/users/ +db/sql/users/ ├── 001_seed.sql.tmpl # Template ├── users.json # → $.users ├── roles.yml # → $.roles @@ -236,7 +235,7 @@ Filenames are converted to camelCase: Files named `$helpers.ts` (or `.js`, `.mjs`) are auto-discovered by walking UP the directory tree to the project root. Child helpers override parent helpers with the same name. ``` -sql/ +db/sql/ ├── $helpers.ts # Project-wide helpers ├── users/ │ ├── $helpers.ts # Overrides parent for this subtree @@ -248,7 +247,7 @@ sql/ ### Writing Helpers ```typescript -// sql/$helpers.ts +// db/sql/$helpers.ts export function padId(id: number, length = 6): string { return String(id).padStart(length, '0') @@ -301,7 +300,7 @@ INSERT INTO roles (name, permissions) VALUES ```sql -- setup.sql.tmpl -{% if ($.config.stage === 'prod') { %} +{% if ($.config.name === 'prod') { %} ALTER TABLE orders ADD CONSTRAINT chk_amount CHECK (amount > 0); {% } %} @@ -347,7 +346,7 @@ GRANT EXECUTE ON {%~ proc %} TO {%~ role %}; ### Composing from Fragments ```sql --- changes/2025-01-15-setup/change/001_schema.sql.tmpl +-- db/changes/2025-01-15-setup/change/001_schema.sql.tmpl {%~ await $.include('../lib/core_tables.sql') %} {%~ await $.include('../lib/content_tables.sql') %} {%~ await $.include('../lib/seed_defaults.sql') %} @@ -373,14 +372,14 @@ Before executing, preview or inspect templates: **CLI:** ```bash -noorm run preview sql/schema.sql.tmpl # Rendered SQL to stdout -noorm run preview sql/schema.sql.tmpl > rendered.sql -noorm run inspect sql/seed.sql.tmpl # Show available context +noorm run preview db/sql/schema.sql.tmpl # Rendered SQL to stdout +noorm run preview db/sql/schema.sql.tmpl > rendered.sql +noorm run inspect db/sql/seed.sql.tmpl # Show available context ``` **SDK:** ```typescript -const result = await ctx.noorm.templates.render('sql/001_users.sql.tmpl'); +const result = await ctx.noorm.templates.render('db/sql/001_users.sql.tmpl'); // result: { name, content, error?, durationMs? } ``` diff --git a/src/cli/change/add.ts b/src/cli/change/add.ts index df8f29af..167dd3b8 100644 --- a/src/cli/change/add.ts +++ b/src/cli/change/add.ts @@ -18,6 +18,7 @@ import { createChange } from '../../core/change/index.js'; import { getSettingsManager } from '../../core/settings/index.js'; import { outputResult, outputError, sharedArgs } from '../_utils.js'; import { requireTty } from './_prompt.js'; +import { DEFAULT_CHANGES_PATH } from '../../core/settings/defaults.js'; const addCommand = defineCommand({ meta: { name: 'add', description: 'Create a new change directory' }, @@ -45,7 +46,7 @@ const addCommand = defineCommand({ const changesDir = join( projectRoot, - settingsManager.settings.paths?.changes ?? 'changes', + settingsManager.settings.paths?.changes ?? DEFAULT_CHANGES_PATH, ); let description = args.name; diff --git a/src/cli/change/edit.ts b/src/cli/change/edit.ts index a09c676e..89450e69 100644 --- a/src/cli/change/edit.ts +++ b/src/cli/change/edit.ts @@ -24,6 +24,7 @@ import { getSettingsManager } from '../../core/settings/index.js'; import { outputError } from '../_utils.js'; import { selectChangeFromFs, requireTty } from './_prompt.js'; import { EXIT } from '../_exit.js'; +import { DEFAULT_CHANGES_PATH } from '../../core/settings/defaults.js'; /** * Spawn an editor against a target path and wait for it to exit. @@ -77,7 +78,7 @@ const editCommand = defineCommand({ const changesDir = join( projectRoot, - settingsManager.settings.paths?.changes ?? 'changes', + settingsManager.settings.paths?.changes ?? DEFAULT_CHANGES_PATH, ); let changeName = args.name; diff --git a/src/cli/change/list.ts b/src/cli/change/list.ts index c6d0dba4..3fe8543a 100644 --- a/src/cli/change/list.ts +++ b/src/cli/change/list.ts @@ -35,7 +35,7 @@ const listCommand = defineCommand({ : [ // `orphaned` is computed but was never rendered, so a change // deleted from disk printed identically to a live applied one. - ...changes.map((cs) => `${cs.name} (${cs.status}${cs.orphaned ? ', orphaned' : ''})`), + ...changes.map((cs) => `${cs.name} (${cs.status}${cs.orphaned ? ', orphaned' : ''})${cs.status === 'failed' && cs.errorMessage ? `: ${cs.errorMessage.split('\n')[0]}` : ''}`), ...(pending > 0 ? [`${pending} pending change(s)`] : []), ].join('\n'); diff --git a/src/cli/change/rm.ts b/src/cli/change/rm.ts index fbf17cb9..a39be2f9 100644 --- a/src/cli/change/rm.ts +++ b/src/cli/change/rm.ts @@ -30,6 +30,7 @@ import { checkConfigPolicy, resolveChannel } from '../../core/policy/index.js'; import { outputResult, outputError, sharedArgs, isYesMode } from '../_utils.js'; import { selectChangeFromFs, requireTty } from './_prompt.js'; import { EXIT } from '../_exit.js'; +import { DEFAULT_CHANGES_PATH } from '../../core/settings/defaults.js'; const rmCommand = defineCommand({ meta: { name: 'rm', description: 'Delete a change directory' }, @@ -82,7 +83,7 @@ const rmCommand = defineCommand({ const changesDir = join( projectRoot, - settingsManager.settings.paths?.changes ?? 'changes', + settingsManager.settings.paths?.changes ?? DEFAULT_CHANGES_PATH, ); let changeName = args.name; diff --git a/src/cli/dev/test-helpers.ts b/src/cli/dev/test-helpers.ts index c8b9b6a2..f2374851 100644 --- a/src/cli/dev/test-helpers.ts +++ b/src/cli/dev/test-helpers.ts @@ -7,8 +7,8 @@ * * @example * ```bash - * noorm dev test-helpers sql/core/05_Cron/Crons.sql.tmpl - * noorm dev test-helpers --json sql/seed.sql.tmpl + * noorm dev test-helpers db/sql/core/05_Cron/Crons.sql.tmpl + * noorm dev test-helpers --json db/sql/seed.sql.tmpl * ``` */ import { dirname, join, relative } from 'node:path'; @@ -160,8 +160,8 @@ const testHelpersCommand = defineCommand({ }); (testHelpersCommand as typeof testHelpersCommand & { examples: string[] }).examples = [ - 'noorm dev test-helpers sql/core/05_Cron/Crons.sql.tmpl', - 'noorm dev test-helpers --json sql/seed.sql.tmpl', + 'noorm dev test-helpers db/sql/core/05_Cron/Crons.sql.tmpl', + 'noorm dev test-helpers --json db/sql/seed.sql.tmpl', ]; export default testHelpersCommand; diff --git a/src/cli/run/dir.ts b/src/cli/run/dir.ts index 26296741..4685a583 100644 --- a/src/cli/run/dir.ts +++ b/src/cli/run/dir.ts @@ -123,7 +123,7 @@ const dirCommand = defineCommand({ (dirCommand as typeof dirCommand & { examples: string[] }).examples = [ 'noorm run dir migrations/', 'noorm run dir seeds/', - 'noorm run dir sql/01_tables/ --json', + 'noorm run dir db/sql/01_tables/ --json', ]; export default dirCommand; diff --git a/src/cli/run/exec.ts b/src/cli/run/exec.ts index c6be6c66..88b0e552 100644 --- a/src/cli/run/exec.ts +++ b/src/cli/run/exec.ts @@ -56,7 +56,7 @@ async function expandGlob(pattern: string, cwd: string): Promise { * discovery. Otherwise treats the argument as a glob pattern and expands it. * * @example - * const files = await resolveInputPaths('sql/', cwd); + * const files = await resolveInputPaths('db/sql/', cwd); * const files = await resolveInputPaths('migrations/*.sql', cwd); */ async function resolveInputPaths(input: string, cwd: string): Promise { @@ -188,11 +188,11 @@ const execCommand = defineCommand({ }); (execCommand as typeof execCommand & { examples: string[] }).examples = [ - 'noorm run exec sql/', + 'noorm run exec db/sql/', 'noorm run exec migrations/', - 'noorm run exec "sql/**/*.sql"', + 'noorm run exec "db/sql/**/*.sql"', 'noorm run exec "seeds/*.sql" --force', - 'noorm run exec sql/ --json', + 'noorm run exec db/sql/ --json', 'noorm run exec "migrations/*.sql" --dry-run', ]; diff --git a/src/cli/run/file.ts b/src/cli/run/file.ts index 046dabae..cddfa766 100644 --- a/src/cli/run/file.ts +++ b/src/cli/run/file.ts @@ -74,7 +74,7 @@ const fileCommand = defineCommand({ (fileCommand as typeof fileCommand & { examples: string[] }).examples = [ 'noorm run file seed.sql', 'noorm run file migrations/001_init.sql', - 'noorm run file sql/init.sql.tmpl --json', + 'noorm run file db/sql/init.sql.tmpl --json', ]; export default fileCommand; diff --git a/src/cli/run/files.ts b/src/cli/run/files.ts index 7bfa7213..f575f176 100644 --- a/src/cli/run/files.ts +++ b/src/cli/run/files.ts @@ -104,7 +104,7 @@ const filesCommand = defineCommand({ (filesCommand as typeof filesCommand & { examples: string[] }).examples = [ 'noorm run files --paths seed.sql,fixtures.sql', - 'noorm run files --paths sql/001_tables.sql,sql/002_indexes.sql', + 'noorm run files --paths db/sql/001_tables.sql,db/sql/002_indexes.sql', 'noorm run files --paths migrations/001.sql,migrations/002.sql --json', 'noorm run files --paths triggers/audit.sql --force', ]; diff --git a/src/cli/run/inspect.ts b/src/cli/run/inspect.ts index e7d755ed..8a95e4b5 100644 --- a/src/cli/run/inspect.ts +++ b/src/cli/run/inspect.ts @@ -6,8 +6,8 @@ * * @example * ```bash - * noorm run inspect sql/users/001_create.sql.tmpl - * noorm run inspect sql/core/05_Cron/Crons.sql.tmpl --json + * noorm run inspect db/sql/users/001_create.sql.tmpl + * noorm run inspect db/sql/core/05_Cron/Crons.sql.tmpl --json * ``` */ import { stat } from 'node:fs/promises'; @@ -272,8 +272,8 @@ const inspectCommand = defineCommand({ }); (inspectCommand as typeof inspectCommand & { examples: string[] }).examples = [ - 'noorm run inspect sql/users/001_create.sql.tmpl', - 'noorm run inspect sql/core/05_Cron/Crons.sql.tmpl --json', + 'noorm run inspect db/sql/users/001_create.sql.tmpl', + 'noorm run inspect db/sql/core/05_Cron/Crons.sql.tmpl --json', ]; export default inspectCommand; diff --git a/src/cli/run/preview.ts b/src/cli/run/preview.ts index 24bf425f..2c7ecf61 100644 --- a/src/cli/run/preview.ts +++ b/src/cli/run/preview.ts @@ -6,9 +6,9 @@ * * @example * ```bash - * noorm run preview sql/users/001_create.sql.tmpl - * noorm run preview sql/core/05_Cron/Crons.sql.tmpl --json - * noorm run preview sql/migrations/002.sql.tmpl > rendered.sql + * noorm run preview db/sql/users/001_create.sql.tmpl + * noorm run preview db/sql/core/05_Cron/Crons.sql.tmpl --json + * noorm run preview db/sql/migrations/002.sql.tmpl > rendered.sql * ``` */ import { stat } from 'node:fs/promises'; @@ -147,9 +147,9 @@ const previewCommand = defineCommand({ }); (previewCommand as typeof previewCommand & { examples: string[] }).examples = [ - 'noorm run preview sql/users/001_create.sql.tmpl', - 'noorm run preview sql/core/05_Cron/Crons.sql.tmpl --json', - 'noorm run preview sql/migrations/002.sql.tmpl > rendered.sql', + 'noorm run preview db/sql/users/001_create.sql.tmpl', + 'noorm run preview db/sql/core/05_Cron/Crons.sql.tmpl --json', + 'noorm run preview db/sql/migrations/002.sql.tmpl > rendered.sql', ]; export default previewCommand; diff --git a/src/cli/settings/edit.ts b/src/cli/settings/edit.ts index 4b64a904..e3c11cc7 100644 --- a/src/cli/settings/edit.ts +++ b/src/cli/settings/edit.ts @@ -22,6 +22,7 @@ import type { } from '../../core/settings/types.js'; import { isYesMode, sharedArgs } from '../_utils.js'; import { EXIT } from '../_exit.js'; +import { DEFAULT_CHANGES_PATH, DEFAULT_SQL_PATH } from '../../core/settings/defaults.js'; type Manager = ReturnType; @@ -77,7 +78,7 @@ async function editPaths(manager: Manager): Promise { const sql = await p.text({ message: 'SQL files path', - initialValue: current.sql ?? './sql', + initialValue: current.sql ?? DEFAULT_SQL_PATH, validate: (v) => { if (v && v.trim()) return undefined; @@ -91,7 +92,7 @@ async function editPaths(manager: Manager): Promise { const changes = await p.text({ message: 'Changes path', - initialValue: current.changes ?? './changes', + initialValue: current.changes ?? DEFAULT_CHANGES_PATH, validate: (v) => { if (v && v.trim()) return undefined; diff --git a/src/cli/update.ts b/src/cli/update.ts index 36ef4d38..499144f6 100644 --- a/src/cli/update.ts +++ b/src/cli/update.ts @@ -4,15 +4,27 @@ * Downloads the platform-appropriate binary from GitHub releases * and replaces the running binary in-place. */ +import * as p from '@clack/prompts'; import { defineCommand } from 'citty'; import { attempt } from '@logosdx/utils'; import { observer } from '../core/observer.js'; import { checkForUpdate, getCurrentVersion } from '../core/update/checker.js'; import { installUpdate } from '../core/update/updater.js'; +import { + SKILL_INSTALLERS, + canPromptForSkill, + findSkillInstaller, + installSkill, + skillCommand, +} from '../core/update/skill.js'; import { isInsecureMode, outputError, outputResult, sharedArgs } from './_utils.js'; import { EXIT } from './_exit.js'; +// Wider than any line rewritten in place, so each `\r` rewrite erases the +// previous one by construction. +const STATUS_LINE_WIDTH = 40; + /** Render a byte count as MB with one decimal (e.g. 41.2). */ function toMb(bytes: number): string { @@ -20,6 +32,177 @@ function toMb(bytes: number): string { } +/** + * Status output for `noorm update`. On a terminal, `pending` leaves its line + * open and the next `say` rewrites it in place; off one, every line is final. + * Under `--json` neither prints, so stdout holds only the JSON document. + * + * @example + * const status = updateStatus((text) => process.stdout.write(text), { isTty: true, json: false }); + * status.pending('Checking for updates...'); + * status.say('Already on the latest version (1.5.1)'); + */ +export function updateStatus(write: (text: string) => void, opts: { isTty: boolean; json: boolean }) { + + let lineOpen = false; + + const raw = (text: string) => { + + lineOpen = !text.endsWith('\n'); + write(text); + + }; + + return { + /** Writes text as given; the download renderer draws through this. */ + raw, + + say(line: string) { + + if (opts.json) return; + + raw(lineOpen ? `\r${line.padEnd(STATUS_LINE_WIDTH)}\n` : `${line}\n`); + + }, + + pending(line: string) { + + if (opts.json) return; + + raw(opts.isTty ? line : `${line}\n`); + + }, + + /** Ends a line left open, so what prints next starts on a line of its own. */ + endLine() { + + if (lineOpen) raw('\n'); + + }, + }; + +} + +/** + * Builds the `\r`-rewritten download line for a terminal, or null off one: + * piped output has no cursor to rewind, so every tick would land on its own + * line. The tick that reaches the total writes `Downloaded X MB (100%)` and + * ends the line; ticks after it write nothing. + * + * @example + * const render = downloadProgressRenderer((text) => process.stdout.write(text), true); + * observer.on('update:progress', (progress) => render?.(progress)); + */ +export function downloadProgressRenderer( + write: (text: string) => void, + isTty: boolean, +): ((progress: { received: number; total: number }) => void) | null { + + let done = false; + + if (!isTty) return null; + + return ({ received, total }) => { + + if (done) return; + + if (total > 0 && received >= total) { + + done = true; + write(`\r${`Downloaded ${toMb(total)} MB (100%)`.padEnd(STATUS_LINE_WIDTH)}\n`); + + return; + + } + + const line = total > 0 + ? `Downloading ${toMb(received)} / ${toMb(total)} MB (${Math.floor((received * 100) / total)}%)` + : `Downloading ${toMb(received)} MB`; + + write(`\r${line.padEnd(STATUS_LINE_WIDTH)}`); + + }; + +} + +/** Every way to install the skill, one per line, with what each needs. */ +function skillInstallOptions(): string { + + const width = Math.max(...SKILL_INSTALLERS.map((installer) => skillCommand(installer).length)); + + return SKILL_INSTALLERS + .map((installer) => ` ${skillCommand(installer).padEnd(width)} (${installer.bin}: ${installer.provider})`) + .join('\n'); + +} + +/** + * Offer to install or update the noorm agent skill once the binary is + * updated, through the first installer on PATH. With none, or when the + * installer fails, it prints every way to install the skill. The update + * already succeeded, so nothing here changes the exit code. + * + * @example + * if (result.success) await offerSkill(args.json); + */ +export async function offerSkill( + json: boolean | undefined, + interactive = canPromptForSkill({ stdinTty: process.stdin.isTTY, stdoutTty: process.stdout.isTTY }), +): Promise { + + if (json) return; + + const installer = findSkillInstaller(); + + if (!installer) { + + process.stdout.write(`Agent skill: install npx or gh, then run its command:\n${skillInstallOptions()}\n`); + + return; + + } + + const command = skillCommand(installer); + + if (!interactive) { + + process.stdout.write(`Agent skill: run \`${command}\` to install or update it.\n`); + + return; + + } + + const answer = await p.confirm({ + message: `Install or update the noorm agent skill? (${command}, run in ${process.cwd()})`, + initialValue: true, + }); + + if (p.isCancel(answer) || !answer) { + + process.stdout.write(`Skipped. Run \`${command}\` any time.\n`); + + return; + + } + + const [code, spawnErr] = await attempt(() => installSkill(installer)); + + if (spawnErr) { + + process.stderr.write(`Could not run ${installer.bin} (${spawnErr.message}). Install the skill with either:\n${skillInstallOptions()}\n`); + + return; + + } + + if (code !== 0) { + + process.stderr.write(`Skill install exited with code ${code}. Install the skill with either:\n${skillInstallOptions()}\n`); + + } + +} + const updateCommand = defineCommand({ meta: { name: 'update', @@ -35,9 +218,10 @@ const updateCommand = defineCommand({ async run({ args }) { const currentVersion = getCurrentVersion(); + const isTty = Boolean(process.stdout.isTTY) && !args.json; + const status = updateStatus((text) => process.stdout.write(text), { isTty, json: Boolean(args.json) }); - process.stdout.write(`Current version: ${currentVersion}\n`); - process.stdout.write('Checking for updates...\n'); + status.pending('Checking for updates...'); const [checkResult, checkErr] = await attempt(() => checkForUpdate()); @@ -63,6 +247,7 @@ const updateCommand = defineCommand({ } else { + status.endLine(); outputError(args, errorMsg); } @@ -73,7 +258,7 @@ const updateCommand = defineCommand({ if (!checkResult.updateAvailable) { - process.stdout.write(`Already on the latest version (${currentVersion})\n`); + status.say(`Already on the latest version (${currentVersion})`); if (args.json) { @@ -90,64 +275,40 @@ const updateCommand = defineCommand({ } - process.stdout.write(`Update available: ${currentVersion} → ${checkResult.latestVersion}\n`); + const latestVersion = checkResult.latestVersion; - // Render a live progress line for the binary download, TTY only. Piped - // output (CI, JSON mode) has no cursor to rewind, so every tick would - // land on its own line and bury the log under a few hundred of them. - // The static "Installing" line below is that case's only progress note. - const isTty = Boolean(process.stdout.isTTY) && !args.json; + status.say(`Update available: ${currentVersion} → ${latestVersion}`); - const onProgress = ({ received, total }: { received: number; total: number }) => { + const insecure = isInsecureMode(args); - if (!isTty) return; + if (insecure) { - const pct = total > 0 ? ` (${Math.floor((received / total) * 100)}%)` : ''; - const of = total > 0 ? ` / ${toMb(total)}` : ''; - const line = `Downloading ${toMb(received)}${of} MB${pct}`; + process.stderr.write('Warning: checksum verification will be skipped if checksums.txt is unreachable (--insecure).\n'); - process.stdout.write(`\r${line} `); + } - }; + const renderProgress = downloadProgressRenderer(status.raw, isTty); - const onRetry = ({ attempt: n, maxAttempts, error }: { attempt: number; maxAttempts: number; error: string }) => { + status.pending(isTty ? 'Downloading...' : 'Installing (downloading binary)...'); - if (args.json) return; + const onProgress = (progress: { received: number; total: number }) => renderProgress?.(progress); - // End the in-place progress line before printing the notice. - const prefix = isTty ? '\n' : ''; + const onRetry = ({ attempt: n, maxAttempts, error }: { attempt: number; maxAttempts: number; error: string }) => { - process.stdout.write(`${prefix}${error} — resuming (attempt ${n + 1}/${maxAttempts})...\n`); + status.endLine(); + status.say(`${error} — resuming (attempt ${n + 1}/${maxAttempts})...`); }; - if (!args.json) { - - process.stdout.write(isTty ? 'Installing...\n' : 'Installing (downloading binary)...\n'); - - } - - const insecure = isInsecureMode(args); - - if (insecure) { - - process.stderr.write('Warning: checksum verification will be skipped if checksums.txt is unreachable (--insecure).\n'); - - } - observer.on('update:progress', onProgress); observer.on('update:retry', onRetry); - const [result, installErr] = await attempt(() => installUpdate(checkResult.latestVersion, { insecure })); + const [result, installErr] = await attempt(() => installUpdate(latestVersion, { insecure })); observer.off('update:progress', onProgress); observer.off('update:retry', onRetry); - if (isTty) { - - process.stdout.write('\n'); - - } + status.endLine(); if (installErr || !result) { @@ -161,7 +322,7 @@ const updateCommand = defineCommand({ success: false, error: errorMsg, currentVersion, - latestVersion: checkResult.latestVersion, + latestVersion, updateAvailable: true, installed: false, }, ''); @@ -179,7 +340,9 @@ const updateCommand = defineCommand({ if (result.success) { - process.stdout.write(`Updated to ${result.newVersion}. Restart noorm to use the new version.\n`); + status.say(`Updated to ${result.newVersion}. Restart noorm to use the new version.`); + + await offerSkill(args.json); } else if (!args.json) { @@ -196,7 +359,7 @@ const updateCommand = defineCommand({ outputResult(args, { success: result.success, currentVersion, - latestVersion: checkResult.latestVersion, + latestVersion, updateAvailable: true, installed: result.success, error: result.error ?? undefined, diff --git a/src/core/change/executor.ts b/src/core/change/executor.ts index 86d439b7..21c5f9fd 100644 --- a/src/core/change/executor.ts +++ b/src/core/change/executor.ts @@ -23,7 +23,6 @@ import path from 'node:path'; import { readFile, writeFile as fsWriteFile, mkdir } from 'node:fs/promises'; import type { Kysely } from 'kysely'; -import { sql } from 'kysely'; import { attempt, attemptSync } from '@logosdx/utils'; @@ -35,6 +34,7 @@ import type { Permission } from '../policy/index.js'; import type { Dialect } from '../connection/types.js'; import { computeChecksum, computeCombinedChecksum } from '../runner/checksum.js'; import { StatementWatcher } from '../runner/statement-watcher.js'; +import { executeSqlBody } from '../runner/mssql-batches.js'; import { getSqlErrorMessage } from '../shared/index.js'; import type { NoormDatabase } from '../shared/index.js'; import { getLockManager } from '../lock/index.js'; @@ -724,17 +724,18 @@ async function runFileBatch( } // Execute SQL - const [, execErr] = await attempt(() => - watcher.run(file.path, executor, (conn) => sql.raw(sqlContent).execute(conn)), + const [execMessage, execErr] = await attempt(() => + watcher.run(file.path, executor, (conn) => executeSqlBody({ db: conn, dialect: context.dialect }, sqlContent)), ); const durationMs = performance.now() - fileStart; - if (execErr) { + // executeSqlBody reports a failed statement as a message, not a throw. + // getSqlErrorMessage preserves TDS diagnostics (line numbers, error codes, procedure names). + const errorMessage = execErr ? getSqlErrorMessage(execErr) : execMessage; + + if (errorMessage) { - // Capture error info FIRST — use getSqlErrorMessage to preserve - // TDS diagnostics (line numbers, error codes, procedure names) - const errorMessage = getSqlErrorMessage(execErr); failed = true; failedFile = file.path; failureError = errorMessage; diff --git a/src/core/change/manager.ts b/src/core/change/manager.ts index 2acc9739..0cc87339 100644 --- a/src/core/change/manager.ts +++ b/src/core/change/manager.ts @@ -28,7 +28,8 @@ import { stat } from 'node:fs/promises'; import { attempt } from '@logosdx/utils'; -import { parseChange, discoverChanges } from './parser.js'; +import { failureListItem, parseChange, scanChanges } from './parser.js'; +import type { ChangeParseFailure } from './parser.js'; import { deleteChange as deleteFromDisk } from './scaffold.js'; import { ChangeHistory } from './history.js'; import { executeChange, revertChange } from './executor.js'; @@ -99,14 +100,35 @@ export class ChangeManager { * List all changes with status. * * Merges changes from disk with status from database. - * Includes orphaned changes (in DB but not on disk). + * Includes orphaned changes (in DB but not on disk), and folders on disk + * that cannot be parsed: `failed` with the parse error when they never + * ran, otherwise their recorded status. * * @returns Array of changes with merged status */ async list(): Promise { + const { items } = await this.#scan(); + + return items; + + } + + /** + * Merge disk and database state, and report the folders that cannot be read. + * + * An unreadable folder that a batch would run (never run, or `reverted` or + * `stale`) is pending work: the batch fails at its position rather than + * skipping it and running later changes out of order. `unreadable` holds the + * applied ones; they keep their status and only warrant a warning. + */ + async #scan(): Promise<{ + items: ChangeListItem[]; + unreadable: ChangeParseFailure[]; + }> { + // Discover changes from disk - const diskChanges = await discoverChanges( + const { changes: diskChanges, failures } = await scanChanges( this.#context.changesDir, this.#context.sqlDir, ); @@ -115,7 +137,7 @@ export class ChangeManager { const statuses = await this.#history.getAllStatuses(); // Create set of disk names for orphan detection - const diskNames = new Set(diskChanges.map((cs) => cs.name)); + const diskNames = new Set([...diskChanges, ...failures].map((cs) => cs.name)); // Get orphaned changes const orphanedNames = await this.#history.getOrphaned(diskNames); @@ -143,6 +165,25 @@ export class ChangeManager { } + // Add folders that are on disk but cannot be parsed + const unreadable: ChangeParseFailure[] = []; + + for (const failure of failures) { + + const status = statuses.get(failure.name); + + const item = failureListItem(failure, status); + + if (!isPendingChange(item)) { + + unreadable.push(failure); + + } + + result.push(item); + + } + // Add orphaned changes for (const name of orphanedNames) { @@ -170,7 +211,10 @@ export class ChangeManager { } // Sort by name - return result.sort((a, b) => a.name.localeCompare(b.name)); + return { + items: result.sort((a, b) => a.name.localeCompare(b.name)), + unreadable, + }; } @@ -255,11 +299,17 @@ export class ChangeManager { const opts = { ...DEFAULT_BATCH, ...options }; const start = performance.now(); - const warnings = await this.#collectWarnings(); + const { items, unreadable } = await this.#scan(); - // Get pending changes - const list = await this.list(); - const pending = list + const messages = [ + ...(await this.#collectWarnings() ?? []), + ...unreadable.map((f) => `Change '${f.name}' is applied but its files can no longer be read: ${f.message}`), + ]; + const warnings = messages.length > 0 ? messages : undefined; + + // An unreadable pending change keeps its place in the sequence: + // loading it fails, which stops the batch there. + const pending = items .filter(isPendingChange) .slice(0, count); @@ -361,11 +411,7 @@ export class ChangeManager { */ async ff(options: BatchChangeOptions = {}): Promise { - // Get count of pending - const list = await this.list(); - const pendingCount = list.filter(isPendingChange).length; - - return this.next(pendingCount, options); + return this.next(Number.POSITIVE_INFINITY, options); } @@ -633,6 +679,12 @@ export class ChangeManager { if (err) { + // A folder that is there but will not parse needs its own error: + // "not found" sends the user looking for a file that exists. + const [onDisk] = await attempt(() => stat(changePath)); + + if (onDisk) throw err; + // Check if it exists in DB (orphaned) const status = await this.#history.getStatus(name); diff --git a/src/core/change/parser.ts b/src/core/change/parser.ts index 8a45b014..a3e5b70f 100644 --- a/src/core/change/parser.ts +++ b/src/core/change/parser.ts @@ -25,7 +25,7 @@ import { constants as fsConstants } from 'node:fs'; import { attempt } from '@logosdx/utils'; import { observer } from '../observer.js'; -import type { Change, ChangeFile, ChangeFileType } from './types.js'; +import type { Change, ChangeFile, ChangeFileType, ChangeListItem, ChangeStatus } from './types.js'; import { ChangeValidationError, ChangeNotFoundError, @@ -137,9 +137,49 @@ export async function parseChange(folderPath: string, sqlDir?: string): Promise< // Discover All Changes // ───────────────────────────────────────────────────────────── +/** A change folder that exists on disk but could not be parsed. */ +export interface ChangeParseFailure { + name: string; + path: string; + message: string; +} + +/** + * The list entry for a folder that could not be parsed: its recorded status + * when it has history, otherwise `failed` with the parse error, which is what + * marks it as outstanding work. + * + * @example + * const item = failureListItem(failure, statuses.get(failure.name)) + */ +export function failureListItem(failure: ChangeParseFailure, status?: ChangeStatus): ChangeListItem { + + const { date, description } = parseName(failure.name); + + return { + name: failure.name, + path: failure.path, + date, + description, + status: status?.status ?? 'failed', + appliedAt: status?.appliedAt ?? null, + appliedBy: status?.appliedBy ?? null, + revertedAt: status?.revertedAt ?? null, + errorMessage: status ? status.errorMessage : failure.message, + appliedHistoryId: status?.appliedHistoryId ?? null, + isNew: !status, + orphaned: false, + }; + +} + /** * Discover all changes in a directory. * + * Folders that fail to parse are dropped here. Callers that must not lose + * track of them (a change that never ran would silently never run) use + * `scanChanges`. + * * @param changesDir - Directory containing change folders * @param sqlDir - Schema directory for resolving .txt references * @returns Array of parsed changes, sorted by name @@ -158,12 +198,30 @@ export async function discoverChanges( sqlDir?: string, ): Promise { + const { changes } = await scanChanges(changesDir, sqlDir); + + return changes; + +} + +/** + * Discover all changes in a directory, and report the folders that could not + * be parsed instead of dropping them. + * + * @example + * const { changes, failures } = await scanChanges('/project/db/changes', '/project/db/sql') + */ +export async function scanChanges( + changesDir: string, + sqlDir?: string, +): Promise<{ changes: Change[]; failures: ChangeParseFailure[] }> { + // Check directory exists const [exists] = await attempt(() => stat(changesDir)); if (!exists) { - return []; + return { changes: [], failures: [] }; } @@ -178,7 +236,7 @@ export async function discoverChanges( context: { changesDir, operation: 'discover' }, }); - return []; + return { changes: [], failures: [] }; } @@ -187,11 +245,14 @@ export async function discoverChanges( // Parse each change const changes: Change[] = []; + const failures: ChangeParseFailure[] = []; for (const folder of folders) { const folderPath = path.join(changesDir, folder.name); + if (!(await looksLikeChange(folderPath))) continue; + const [change, parseErr] = await attempt(() => parseChange(folderPath, sqlDir)); if (parseErr) { @@ -203,6 +264,8 @@ export async function discoverChanges( context: { folder: folder.name, operation: 'parse' }, }); + failures.push({ name: folder.name, path: folderPath, message: parseErr.message }); + continue; } @@ -212,7 +275,10 @@ export async function discoverChanges( } // Sort by name (date prefix ensures chronological order) - return changes.sort((a, b) => a.name.localeCompare(b.name)); + return { + changes: changes.sort((a, b) => a.name.localeCompare(b.name)), + failures, + }; } @@ -234,15 +300,15 @@ export async function discoverChanges( * * @example * ```typescript - * // Manifest contains: - * // sql/tables/users.sql - * // sql/views/active_users.sql + * // Lines are relative to the schema directory, not the project root: + * // tables/users.sql + * // views/active_users.sql * * const paths = await resolveManifest( - * '/project/changes/.../change/001_refs.txt', - * '/project/sql' + * '/project/db/changes/.../change/001_refs.txt', + * '/project/db/sql' * ) - * // => ['/project/sql/tables/users.sql', '/project/sql/views/active_users.sql'] + * // => ['/project/db/sql/tables/users.sql', '/project/db/sql/views/active_users.sql'] * ``` */ export async function resolveManifest(manifestPath: string, sqlDir: string): Promise { @@ -381,6 +447,24 @@ function parseName(name: string): { date: Date | null; description: string } { } +/** + * A folder with neither `change/` nor `revert/` is not a change attempt: + * shared `$.include()` fragments and archives live beside real changes. + */ +async function looksLikeChange(folderPath: string): Promise { + + for (const subfolder of ['change', 'revert']) { + + const [entry] = await attempt(() => stat(path.join(folderPath, subfolder))); + + if (entry?.isDirectory()) return true; + + } + + return false; + +} + /** * Scan a folder for SQL and manifest files. */ diff --git a/src/core/change/types.ts b/src/core/change/types.ts index a00a26a9..dc71caac 100644 --- a/src/core/change/types.ts +++ b/src/core/change/types.ts @@ -253,6 +253,9 @@ export interface ChangeListItem { * `stale` is set by teardown on changes that did apply but whose objects * no longer exist, so it is pending work in the only sense that matters. * + * A `failed` item that is also new never ran and could not be parsed. A batch + * has to stop at it, not step over it, so it counts as outstanding work. + * * @example * const outstanding = list.filter(isPendingChange); */ @@ -262,6 +265,7 @@ export function isPendingChange(item: ChangeListItem): boolean { item.status === 'pending' || item.status === 'reverted' || item.status === 'stale' + || (item.status === 'failed' && item.isNew) ); } diff --git a/src/core/project-init.ts b/src/core/project-init.ts index 1a2ce8a1..d64245bf 100644 --- a/src/core/project-init.ts +++ b/src/core/project-init.ts @@ -2,7 +2,8 @@ * Project initialization — shared between TUI `InitScreen` and CLI `noorm init`. * * Creates the on-disk structure for a new noorm project: - * - `sql/` and `changes/` directories with .gitkeep + * - `db/sql/` and `db/changes/` directories with .gitkeep + * - `db/AGENTS.md` and `db/CLAUDE.md` agent steering, when absent * - `.noorm/`, `.noorm/state/`, `.noorm/.gitignore` * - `settings.yml` via SettingsManager * - `state.enc` via StateManager @@ -19,7 +20,7 @@ import { readFileSync, appendFileSync, } from 'node:fs'; -import { join } from 'node:path'; +import { join, posix } from 'node:path'; import { attempt } from '@logosdx/utils'; @@ -30,6 +31,13 @@ import { loadIdentityMetadata, } from './identity/index.js'; import { SettingsManager } from './settings/manager.js'; +import { + DEFAULT_CHANGES_PATH, + DEFAULT_DB_DIR, + DEFAULT_SQL_PATH, + isDefaultLayout, +} from './settings/defaults.js'; +import type { PathConfig } from './settings/types.js'; import { StateManager, getStateManager } from './state/index.js'; import { observer } from './observer.js'; @@ -41,6 +49,19 @@ const NOORM_IGNORE_ENTRY = '.noorm/state/'; const NOORM_GITIGNORE_BLOCK = `\n# noorm\n${NOORM_IGNORE_ENTRY}\n`; +const AGENTS_TEMPLATE = [ + 'Use the `/noorm` skill for managing files here.', + '', + 'For data modeling, use ignatius (https://noorm.dev/modeling/) and its `/ignatius-modeling` skill ' + + '(https://noorm.dev/modeling/modeling-skill).', + '', + '', + '', +].join('\n'); + +// Claude Code skips AGENTS.md when a CLAUDE.md is present, so CLAUDE.md imports it. +const CLAUDE_TEMPLATE = '@AGENTS.md\n'; + export interface ProjectInitIdentityInfo { name: string; email: string; @@ -59,6 +80,49 @@ export interface ProjectInitResult { createdFiles: string[]; } +/** + * Creates `db/sql/`, `db/changes/`, and the agent steering files beside them, + * for projects on the default layout. The steering files hold the user's edits, + * so they are written only when absent. + * + * @example + * scaffoldDbLayout('/project', DEFAULT_PATH_CONFIG); // ['db/sql/.gitkeep', ..., 'db/CLAUDE.md'] + */ +export function scaffoldDbLayout(projectRoot: string, paths: PathConfig): string[] { + + if (!isDefaultLayout(paths)) return []; + + const created: string[] = []; + const agentFiles = { + 'AGENTS.md': AGENTS_TEMPLATE, + 'CLAUDE.md': CLAUDE_TEMPLATE, + }; + + for (const dir of [DEFAULT_SQL_PATH, DEFAULT_CHANGES_PATH]) { + + const relativeDir = posix.normalize(dir); + + mkdirSync(join(projectRoot, relativeDir), { recursive: true }); + writeFileSync(join(projectRoot, relativeDir, '.gitkeep'), '', { flag: 'a' }); + created.push(`${relativeDir}/.gitkeep`); + + } + + for (const [name, body] of Object.entries(agentFiles)) { + + const filePath = join(projectRoot, DEFAULT_DB_DIR, name); + + if (existsSync(filePath)) continue; + + writeFileSync(filePath, body); + created.push(`${DEFAULT_DB_DIR}/${name}`); + + } + + return created; + +} + /** * Bootstraps a noorm project: directories, settings, state, and (optionally) identity. * When `identityInfo` is null, a global identity must already exist. @@ -69,22 +133,12 @@ export async function performProjectInit( const { projectRoot, force, identityInfo } = opts; - const sqlPath = join(projectRoot, 'sql'); - const changesPath = join(projectRoot, 'changes'); const noormPath = join(projectRoot, '.noorm'); const statePath = join(noormPath, 'state'); const gitignorePath = join(projectRoot, '.gitignore'); const created: string[] = []; - mkdirSync(sqlPath, { recursive: true }); - writeFileSync(join(sqlPath, '.gitkeep'), '', { flag: 'a' }); - created.push('sql/.gitkeep'); - - mkdirSync(changesPath, { recursive: true }); - writeFileSync(join(changesPath, '.gitkeep'), '', { flag: 'a' }); - created.push('changes/.gitkeep'); - mkdirSync(noormPath, { recursive: true }); mkdirSync(statePath, { recursive: true }); @@ -134,6 +188,9 @@ export async function performProjectInit( const settingsManager = new SettingsManager(projectRoot); await settingsManager.init(force); + + const scaffolded = scaffoldDbLayout(projectRoot, settingsManager.getPaths()); + created.push('.noorm/settings.yml'); const stateManager = new StateManager(projectRoot, { privateKey }); @@ -168,7 +225,7 @@ export async function performProjectInit( return { success: true, createdIdentity, - createdFiles: created, + createdFiles: [...scaffolded, ...created], }; } diff --git a/src/core/runner/mssql-batches.ts b/src/core/runner/mssql-batches.ts index 19ccc8f0..3449dd06 100644 --- a/src/core/runner/mssql-batches.ts +++ b/src/core/runner/mssql-batches.ts @@ -19,6 +19,9 @@ import { getSqlErrorMessage } from '../shared/index.js'; import { splitSqliteStatements } from './sqlite-statements.js'; import type { RunContext } from './types.js'; +/** The two fields body execution reads; a change context supplies them too. */ +type SqlBodyContext = Pick; + /** * Split MSSQL SQL content on the `GO` batch separator. @@ -118,7 +121,7 @@ function isCommentOnly(batch: string): boolean { * } */ export async function executeSqlBody( - context: RunContext, + context: SqlBodyContext, sqlContent: string, ): Promise { @@ -154,7 +157,7 @@ export async function executeSqlBody( * usually the whole file. */ async function executeFragments( - context: RunContext, + context: SqlBodyContext, fragments: string[], label: 'batch' | 'statement', ): Promise { diff --git a/src/core/runner/runner.ts b/src/core/runner/runner.ts index 3ab691cd..45034466 100644 --- a/src/core/runner/runner.ts +++ b/src/core/runner/runner.ts @@ -1215,7 +1215,7 @@ async function executeDryRun(context: RunContext, files: string[]): Promise this.load()) : [undefined]; + const keptPaths = previous ? this.#document?.paths : undefined; + this.#document = createDefaultSettings(); + + if (keptPaths) this.#document.paths = clone(keptPaths); + this.#refreshResolved(); this.#loaded = true; diff --git a/src/core/update/skill.ts b/src/core/update/skill.ts new file mode 100644 index 00000000..4fa749b1 --- /dev/null +++ b/src/core/update/skill.ts @@ -0,0 +1,107 @@ +/** + * Install or update the noorm agent skill. + * + * The skill ships from this repo's `skills/` directory. Either the `skills` + * CLI (through npx) or the GitHub CLI can install it, and both ask which + * coding agents to install it into. + */ +import { spawn } from 'node:child_process'; + +/** A program that can install the skill, and how to reach it when missing. */ +export interface SkillInstaller { + + /** Executable looked up on PATH. */ + bin: 'npx' | 'gh'; + + args: string[]; + + /** What the user installs to get `bin`, and where. */ + provider: string; + +} + +/** In order of preference: npx is what the docs teach, gh is the fallback. */ +export const SKILL_INSTALLERS: readonly [SkillInstaller, SkillInstaller] = [ + { + bin: 'npx', + args: ['skills', 'add', 'noormdev/noorm/skills'], + provider: 'Node.js, https://nodejs.org', + }, + { + bin: 'gh', + args: ['skill', 'install', 'noormdev/noorm', 'skills/noorm'], + provider: 'GitHub CLI 2.90 or newer, https://cli.github.com', + }, +]; + +/** + * The installer as a command line to show the user. + * + * @example + * skillCommand(SKILL_INSTALLERS[0]); // 'npx skills add noormdev/noorm/skills' + */ +export function skillCommand(installer: SkillInstaller): string { + + return [installer.bin, ...installer.args].join(' '); + +} + +/** + * The first installer whose program is on PATH, or null when there is none. + * Checked before prompting, so the prompt names the command that will run. + * + * @example + * const installer = findSkillInstaller(); + * if (!installer) printBothOptions(); + */ +export function findSkillInstaller(): SkillInstaller | null { + + // Bun.which otherwise reads the PATH the process started with + const PATH = process.env['PATH'] ?? ''; + + return SKILL_INSTALLERS.find((installer) => Bun.which(installer.bin, { PATH }) !== null) ?? null; + +} + +/** + * Whether `noorm update` may ask about the skill: a person is at the terminal. + * + * @example + * if (canPromptForSkill({ stdinTty: process.stdin.isTTY, stdoutTty: process.stdout.isTTY })) { ... } + */ +export function canPromptForSkill(opts: { + stdinTty: boolean | undefined; + stdoutTty: boolean | undefined; +}): boolean { + + return Boolean(opts.stdinTty) && Boolean(opts.stdoutTty); + +} + +/** + * Run the installer attached to the terminal, so it can ask which agents to + * install into. Running it again replaces an installed copy. Rejects only + * when the program cannot be started. + * + * @example + * const [code, err] = await attempt(() => installSkill(installer)); + */ +export function installSkill(installer: SkillInstaller): Promise { + + const isWindows = process.platform === 'win32'; + + return new Promise((resolve, reject) => { + + const child = spawn(installer.bin, installer.args, { + stdio: 'inherit', + // npx is npx.cmd on Windows, which Node refuses to spawn without a shell + shell: isWindows, + }); + + child.on('error', reject); + // A signal (Ctrl-C at the installer's prompt) leaves no exit code; that is a failure + child.on('exit', (code) => resolve(code ?? 1)); + + }); + +} diff --git a/src/rpc/commands/run.ts b/src/rpc/commands/run.ts index 27ba5ffb..fa5f8de7 100644 --- a/src/rpc/commands/run.ts +++ b/src/rpc/commands/run.ts @@ -36,7 +36,7 @@ const runFileCommand: RpcCommand = { name: 'run_file', description: 'Execute a single SQL file against the database.', examples: [ - { description: 'run a file', input: { path: 'sql/procedures/usp_get_user.sql' } }, + { description: 'run a file', input: { path: 'db/sql/procedures/usp_get_user.sql' } }, ], inputSchema: runFileSchema, permission: 'run:file', diff --git a/src/sdk/namespaces/changes.ts b/src/sdk/namespaces/changes.ts index c07a38a2..5cf906e9 100644 --- a/src/sdk/namespaces/changes.ts +++ b/src/sdk/namespaces/changes.ts @@ -41,6 +41,7 @@ import { checkProtectedConfig } from '../guards.js'; import type { ContextState } from '../state.js'; import { requireConnection } from '../state.js'; +import { DEFAULT_CHANGES_PATH, DEFAULT_SQL_PATH } from '../../core/settings/defaults.js'; // ───────────────────────────────────────────────────────────── // ChangesNamespace @@ -426,7 +427,7 @@ export class ChangesNamespace { return path.join( this.#state.projectRoot, - this.#state.settings.paths?.changes ?? 'changes', + this.#state.settings.paths?.changes ?? DEFAULT_CHANGES_PATH, ); } @@ -435,7 +436,7 @@ export class ChangesNamespace { return path.join( this.#state.projectRoot, - this.#state.settings.paths?.sql ?? 'sql', + this.#state.settings.paths?.sql ?? DEFAULT_SQL_PATH, ); } diff --git a/src/sdk/namespaces/run.ts b/src/sdk/namespaces/run.ts index 17d43575..53ac11e3 100644 --- a/src/sdk/namespaces/run.ts +++ b/src/sdk/namespaces/run.ts @@ -38,6 +38,7 @@ import { checkProtectedConfig } from '../guards.js'; import type { ContextState } from '../state.js'; import { requireConnection } from '../state.js'; import type { BuildOptions } from '../types.js'; +import { DEFAULT_SQL_PATH } from '../../core/settings/defaults.js'; // ───────────────────────────────────────────────────────────── // RunNamespace @@ -62,13 +63,13 @@ export class RunNamespace { * * @example * ```typescript - * const files = await ctx.noorm.run.discover('sql/') + * const files = await ctx.noorm.run.discover('db/sql/') * ``` */ async discover(dirpath?: string): Promise { const absolutePath = this.#resolvePath( - dirpath ?? this.#state.settings.paths?.sql ?? 'sql', + dirpath ?? this.#state.settings.paths?.sql ?? DEFAULT_SQL_PATH, ); return coreDiscoverFiles(absolutePath); @@ -84,7 +85,7 @@ export class RunNamespace { * * @example * ```typescript - * const results = await ctx.noorm.run.preview(['sql/001.sql', 'sql/002.sql']) + * const results = await ctx.noorm.run.preview(['db/sql/001.sql', 'db/sql/002.sql']) * ``` */ async preview( @@ -186,7 +187,7 @@ export class RunNamespace { const context = await this.#createRunContext(); const sqlPath = path.join( this.#state.projectRoot, - this.#state.settings.paths?.sql ?? 'sql', + this.#state.settings.paths?.sql ?? DEFAULT_SQL_PATH, ); const configForMatch = { @@ -227,7 +228,7 @@ export class RunNamespace { // Reported on the result rather than thrown: an unmatched entry is a // misconfiguration, not an execution failure, and a build that legitimately // matches nothing must keep succeeding. Surfacing it is what stops the - // `sql/`-prefix mistake from reading as a clean build. + // `db/sql/`-prefix mistake from reading as a clean build. // // Both directions, because the exclude side is the one that fails // dangerously: it fences off nothing and the held-back files run. diff --git a/src/tui/screens/change/ChangeEditScreen.tsx b/src/tui/screens/change/ChangeEditScreen.tsx index 1ecfaf9c..0a6b4439 100644 --- a/src/tui/screens/change/ChangeEditScreen.tsx +++ b/src/tui/screens/change/ChangeEditScreen.tsx @@ -23,7 +23,7 @@ import { useAppContext } from '../../app-context.js'; import { Panel, Spinner, StatusMessage, MissingParamPanel } from '../../components/index.js'; import { useAsyncEffect } from '../../hooks/index.js'; import { getErrorMessage, resolveChangesDir, resolveSqlDir } from '../../utils/index.js'; -import { discoverChanges } from '../../../core/change/parser.js'; +import { scanChanges } from '../../../core/change/parser.js'; /** * Edit steps. @@ -63,12 +63,12 @@ export function ChangeEditScreen({ params }: ScreenProps): ReactElement { const [_, err] = await attempt(async () => { // Find the change - const changes = await discoverChanges( + const { changes, failures } = await scanChanges( resolveChangesDir(projectRoot, settings), resolveSqlDir(projectRoot, settings), ); - const change = changes.find((cs) => cs.name === changeName); + const change = [...changes, ...failures].find((cs) => cs.name === changeName); if (!change) { diff --git a/src/tui/screens/change/ChangeFFScreen.tsx b/src/tui/screens/change/ChangeFFScreen.tsx index 78f625a0..4fc29ddd 100644 --- a/src/tui/screens/change/ChangeFFScreen.tsx +++ b/src/tui/screens/change/ChangeFFScreen.tsx @@ -78,13 +78,13 @@ export function ChangeFFScreen({ params: _params }: ScreenProps): ReactElement { const [_, err] = await attempt(async () => { - const { changes, statuses } = await loadChangesWithStatus( + const { changes, statuses, failures } = await loadChangesWithStatus( activeConfig, activeConfigName ?? '', settings, projectRoot, ); if (isCancelled()) return; - const pending = buildPendingChangeList(changes, statuses); + const pending = buildPendingChangeList(changes, statuses, failures); // Validate all pending changes have actual content const emptyChanges: string[] = []; diff --git a/src/tui/screens/change/ChangeListScreen.tsx b/src/tui/screens/change/ChangeListScreen.tsx index 3f08a3a4..42f7bd76 100644 --- a/src/tui/screens/change/ChangeListScreen.tsx +++ b/src/tui/screens/change/ChangeListScreen.tsx @@ -86,13 +86,13 @@ export function ChangeListScreen({ params: _params }: ScreenProps): ReactElement const [_, err] = await attempt(async () => { - const { changes: diskChanges, statuses } = await loadChangesWithStatus( + const { changes: diskChanges, statuses, failures } = await loadChangesWithStatus( activeConfig, activeConfigName ?? '', settings, projectRoot, ); if (isCancelled()) return; - const merged = buildMergedChangeList(diskChanges, statuses); + const merged = buildMergedChangeList(diskChanges, statuses, failures); setChanges(merged); diff --git a/src/tui/screens/change/ChangeNextScreen.tsx b/src/tui/screens/change/ChangeNextScreen.tsx index ec556b6c..e215be4f 100644 --- a/src/tui/screens/change/ChangeNextScreen.tsx +++ b/src/tui/screens/change/ChangeNextScreen.tsx @@ -85,13 +85,13 @@ export function ChangeNextScreen({ params }: ScreenProps): ReactElement { const [_, err] = await attempt(async () => { - const { changes, statuses } = await loadChangesWithStatus( + const { changes, statuses, failures } = await loadChangesWithStatus( activeConfig, activeConfigName ?? '', settings, projectRoot, ); if (isCancelled()) return; - const pending = buildPendingChangeList(changes, statuses); + const pending = buildPendingChangeList(changes, statuses, failures); setPendingChanges(pending); diff --git a/src/tui/screens/home.tsx b/src/tui/screens/home.tsx index df8d42c3..155e14f4 100644 --- a/src/tui/screens/home.tsx +++ b/src/tui/screens/home.tsx @@ -32,7 +32,8 @@ import { CURRENT_VERSIONS } from '../../core/version/types.js'; import { getFullVersionRecord, type FullVersionRecord } from '../../core/version/schema/index.js'; import { fetchOverview, type ExploreOverview } from '../../core/explore/index.js'; import { ChangeHistory } from '../../core/change/history.js'; -import { discoverChanges } from '../../core/change/parser.js'; +import { failureListItem, scanChanges } from '../../core/change/parser.js'; +import { isPendingChange } from '../../core/change/types.js'; import { getLockManager } from '../../core/lock/manager.js'; import type { UnifiedHistoryRecord } from '../../core/change/types.js'; import type { LockStatus } from '../../core/lock/types.js'; @@ -179,7 +180,7 @@ export function HomeScreen({ params: _params }: ScreenProps): ReactElement { const changeHistory = new ChangeHistory(db, activeConfigName ?? '', connDialect); // Discover changes from disk - const diskChanges = await discoverChanges( + const { changes: diskChanges, failures } = await scanChanges( resolveChangesDir(projectRoot, settings), resolveSqlDir(projectRoot, settings), ); @@ -200,6 +201,13 @@ export function HomeScreen({ params: _params }: ScreenProps): ReactElement { } + // An unreadable change that a batch would run is outstanding work too + for (const failure of failures) { + + if (isPendingChange(failureListItem(failure, statuses.get(failure.name)))) pendingCount++; + + } + // Get recent activity (all operation types) const history = await changeHistory.getUnifiedHistory(undefined, 5); diff --git a/src/tui/screens/init/InitScreen.tsx b/src/tui/screens/init/InitScreen.tsx index 2454fc1a..1e372bc0 100644 --- a/src/tui/screens/init/InitScreen.tsx +++ b/src/tui/screens/init/InitScreen.tsx @@ -8,7 +8,7 @@ * 4. Optionally add first config * * Creates: - * - sql/ and changes/ directories + * - db/sql/, db/changes/, db/AGENTS.md, db/CLAUDE.md (default layout only) * - .noorm/settings.yml * - .noorm/state/state.enc * - Keypair in ~/.noorm/ @@ -39,6 +39,9 @@ import { loadIdentityMetadata, } from '../../../core/identity/index.js'; import { SettingsManager } from '../../../core/settings/manager.js'; +import { DEFAULT_PATH_CONFIG } from '../../../core/settings/defaults.js'; +import type { PathConfig } from '../../../core/settings/types.js'; +import { scaffoldDbLayout } from '../../../core/project-init.js'; import { StateManager, getStateManager } from '../../../core/state/index.js'; import { observer } from '../../../core/observer.js'; @@ -102,6 +105,8 @@ export function InitScreen({ params }: ScreenProps): ReactElement { // Project root const projectRoot = process.cwd(); + const [initPaths, setInitPaths] = useState(DEFAULT_PATH_CONFIG); + // Check existing state on mount useEffect(() => { @@ -111,6 +116,10 @@ export function InitScreen({ params }: ScreenProps): ReactElement { const noormDir = join(projectRoot, '.noorm'); const alreadyInitialized = existsSync(noormDir); + const [existing] = await attempt(() => new SettingsManager(projectRoot).load()); + + setInitPaths(existing?.paths ?? DEFAULT_PATH_CONFIG); + // Check if identity key files exist const keysExist = await hasKeyFiles(); setHasExistingKeys(keysExist); @@ -222,16 +231,10 @@ export function InitScreen({ params }: ScreenProps): ReactElement { // Step 1: Create directories updateItem(0, { status: 'running' }); - const sqlPath = join(projectRoot, 'sql'); - const changesPath = join(projectRoot, 'changes'); const noormPath = join(projectRoot, '.noorm'); const statePath = join(noormPath, 'state'); - mkdirSync(sqlPath, { recursive: true }); - writeFileSync(join(sqlPath, '.gitkeep'), '', { flag: 'a' }); - - mkdirSync(changesPath, { recursive: true }); - writeFileSync(join(changesPath, '.gitkeep'), '', { flag: 'a' }); + scaffoldDbLayout(projectRoot, initPaths); mkdirSync(noormPath, { recursive: true }); mkdirSync(statePath, { recursive: true }); @@ -371,7 +374,7 @@ export function InitScreen({ params }: ScreenProps): ReactElement { setStep('complete'); - }, [projectRoot, identityValues, hasExistingKeys, hasExistingMetadata, force, refresh]); + }, [projectRoot, identityValues, hasExistingKeys, hasExistingMetadata, force, refresh, initPaths]); // Handle project setup - add config const handleAddConfig = useCallback(async () => { @@ -467,6 +470,8 @@ export function InitScreen({ params }: ScreenProps): ReactElement { return ( Will create: - - • - {sqlPath}/.gitkeep - - - • - {changesPath}/.gitkeep - + {scaffoldsDb && ( + <> + + • + {sqlPath}/.gitkeep + + + • + {changesPath}/.gitkeep + + + • + ./{DEFAULT_DB_DIR}/AGENTS.md + + + • + ./{DEFAULT_DB_DIR}/CLAUDE.md + + + )} • .noorm/settings.yml diff --git a/src/tui/screens/run/RunBuildScreen.tsx b/src/tui/screens/run/RunBuildScreen.tsx index f921e37f..bc7ae651 100644 --- a/src/tui/screens/run/RunBuildScreen.tsx +++ b/src/tui/screens/run/RunBuildScreen.tsx @@ -40,6 +40,7 @@ import { runCancelMessage, } from '../../utils/index.js'; import { attempt } from '@logosdx/utils'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; type Phase = 'loading' | 'confirm' | 'running' | 'complete' | 'error'; @@ -76,7 +77,7 @@ export function RunBuildScreen({ params: _params }: ScreenProps): ReactElement { // Get project root const projectRoot = process.cwd(); - const schema = settings.paths?.sql ?? 'sql'; + const schema = settings.paths?.sql ?? DEFAULT_SQL_PATH; setSqlPath(schema); // Get effective build paths diff --git a/src/tui/screens/run/RunDirScreen.tsx b/src/tui/screens/run/RunDirScreen.tsx index 7364b245..1b120d2a 100644 --- a/src/tui/screens/run/RunDirScreen.tsx +++ b/src/tui/screens/run/RunDirScreen.tsx @@ -28,6 +28,7 @@ import { useConnection } from '../../hooks/index.js'; import { attempt } from '@logosdx/utils'; import type { SelectListItem } from '../../components/index.js'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; type Phase = 'loading' | 'picker' | 'confirm' | 'checking' | 'rerun-confirm' | 'running' | 'complete' | 'error'; @@ -101,7 +102,7 @@ export function RunDirScreen({ params }: ScreenProps): ReactElement { const cancelledRef = useRef(false); const projectRoot = process.cwd(); - const sqlPath = settings?.paths?.sql ?? 'sql'; + const sqlPath = settings?.paths?.sql ?? DEFAULT_SQL_PATH; // Load files and extract directories on mount useAsyncEffect(async (isCancelled) => { diff --git a/src/tui/screens/run/RunExecScreen.tsx b/src/tui/screens/run/RunExecScreen.tsx index c74bd1cf..2955c6c5 100644 --- a/src/tui/screens/run/RunExecScreen.tsx +++ b/src/tui/screens/run/RunExecScreen.tsx @@ -46,6 +46,7 @@ import { runCancelMessage, } from '../../utils/index.js'; import { attempt } from '@logosdx/utils'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; type Phase = 'loading' | 'picker' | 'confirm' | 'running' | 'complete' | 'error'; @@ -78,7 +79,7 @@ export function RunExecScreen({ params: _params }: ScreenProps): ReactElement { setPhase('loading'); - const sqlPath = settings.paths?.sql ?? 'sql'; + const sqlPath = settings.paths?.sql ?? DEFAULT_SQL_PATH; const sqlFullPath = join(projectRoot, sqlPath); const [files, err] = await attempt(() => discoverFiles(sqlFullPath)); diff --git a/src/tui/screens/run/RunFileScreen.tsx b/src/tui/screens/run/RunFileScreen.tsx index dd665c6a..7a7d41ff 100644 --- a/src/tui/screens/run/RunFileScreen.tsx +++ b/src/tui/screens/run/RunFileScreen.tsx @@ -27,6 +27,7 @@ import { useConnection } from '../../hooks/index.js'; import { attempt } from '@logosdx/utils'; import type { SelectListItem } from '../../components/index.js'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; type Phase = 'loading' | 'picker' | 'confirm' | 'checking' | 'rerun-confirm' | 'running' | 'complete' | 'error'; @@ -65,7 +66,7 @@ export function RunFileScreen({ params }: ScreenProps): ReactElement { setPhase('loading'); - const sqlPath = settings.paths?.sql ?? 'sql'; + const sqlPath = settings.paths?.sql ?? DEFAULT_SQL_PATH; const sqlFullPath = join(projectRoot, sqlPath); const [files, err] = await attempt(() => discoverFiles(sqlFullPath)); @@ -381,7 +382,7 @@ export function RunFileScreen({ params }: ScreenProps): ReactElement { if (phase === 'picker') { const hasFiles = allFiles.length > 0; - const sqlPath = settings?.paths?.sql ?? 'sql'; + const sqlPath = settings?.paths?.sql ?? DEFAULT_SQL_PATH; return ( diff --git a/src/tui/screens/run/RunInspectScreen.tsx b/src/tui/screens/run/RunInspectScreen.tsx index 8481b617..d00dc074 100644 --- a/src/tui/screens/run/RunInspectScreen.tsx +++ b/src/tui/screens/run/RunInspectScreen.tsx @@ -39,6 +39,7 @@ import type { TemplateContext } from '../../../core/template/types.js'; import type { Dialect } from '../../../core/connection/types.js'; import type { NoormDatabase } from '../../../core/shared/index.js'; import type { StateManager } from '../../../core/state/index.js'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; type Phase = 'loading' | 'picker' | 'inspecting' | 'expanded' | 'preview' | 'error'; @@ -568,7 +569,7 @@ export function RunInspectScreen({ params }: ScreenProps): ReactElement { setPhase('loading'); - const sqlPath = settings.paths?.sql ?? 'sql'; + const sqlPath = settings.paths?.sql ?? DEFAULT_SQL_PATH; const sqlFullPath = join(projectRoot, sqlPath); const [files, err] = await attempt(() => discoverFiles(sqlFullPath)); @@ -855,7 +856,7 @@ export function RunInspectScreen({ params }: ScreenProps): ReactElement { if (phase === 'picker') { const hasFiles = allFiles.length > 0; - const sqlPath = settings?.paths?.sql ?? 'sql'; + const sqlPath = settings?.paths?.sql ?? DEFAULT_SQL_PATH; return ( diff --git a/src/tui/screens/run/RunListScreen.tsx b/src/tui/screens/run/RunListScreen.tsx index 4cf64567..f4404907 100644 --- a/src/tui/screens/run/RunListScreen.tsx +++ b/src/tui/screens/run/RunListScreen.tsx @@ -30,6 +30,7 @@ import { useFocusScope } from '../../focus.js'; import { useActiveConfig, useSettings, useGlobalModes } from '../../app-context.js'; import { Panel, Spinner } from '../../components/index.js'; import { getEffectiveBuildPaths } from '../../../core/settings/rules.js'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; /** * RunListScreen component. @@ -171,7 +172,7 @@ export function RunListScreen({ params: _params }: ScreenProps): ReactElement { } - const sqlPath = settings?.paths?.sql ?? 'sql'; + const sqlPath = settings?.paths?.sql ?? DEFAULT_SQL_PATH; return ( diff --git a/src/tui/screens/settings/SettingsBuildScreen.tsx b/src/tui/screens/settings/SettingsBuildScreen.tsx index 4b04d034..579956ba 100644 --- a/src/tui/screens/settings/SettingsBuildScreen.tsx +++ b/src/tui/screens/settings/SettingsBuildScreen.tsx @@ -20,7 +20,7 @@ import { useRouter } from '../../router.js'; import { useAppContext } from '../../app-context.js'; import { Panel, Form } from '../../components/index.js'; import { useSettingsOperation } from '../../hooks/index.js'; -import { DEFAULT_PATH_CONFIG } from '../../../core/settings/defaults.js'; +import { DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; /** * Parse comma-separated string to array. @@ -72,11 +72,11 @@ export function SettingsBuildScreen({ params: _params }: ScreenProps): ReactElem // Get current SQL path const sqlPath = useMemo(() => { - if (!settingsManager) return DEFAULT_PATH_CONFIG.sql ?? './sql'; + if (!settingsManager) return DEFAULT_SQL_PATH; const paths = settingsManager.getPaths(); - return paths.sql ?? DEFAULT_PATH_CONFIG.sql ?? './sql'; + return paths.sql ?? DEFAULT_SQL_PATH; }, [settingsManager]); diff --git a/src/tui/screens/settings/SettingsListScreen.tsx b/src/tui/screens/settings/SettingsListScreen.tsx index 7089cd93..48dcace7 100644 --- a/src/tui/screens/settings/SettingsListScreen.tsx +++ b/src/tui/screens/settings/SettingsListScreen.tsx @@ -23,6 +23,7 @@ import { useRouter } from '../../router.js'; import { useFocusScope } from '../../focus.js'; import { useAppContext } from '../../app-context.js'; import { Panel, SelectList, type SelectListItem } from '../../components/index.js'; +import { DEFAULT_CHANGES_PATH, DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; /** * Settings section item value. @@ -56,7 +57,7 @@ function pathsSectionSummary(settings: Settings | null): string { const paths = settings.paths ?? {}; - return `schema: ${paths.sql ?? './sql'}, changes: ${paths.changes ?? './changes'}`; + return `schema: ${paths.sql ?? DEFAULT_SQL_PATH}, changes: ${paths.changes ?? DEFAULT_CHANGES_PATH}`; } diff --git a/src/tui/screens/settings/SettingsPathsScreen.tsx b/src/tui/screens/settings/SettingsPathsScreen.tsx index abeaefd9..3fa2aecd 100644 --- a/src/tui/screens/settings/SettingsPathsScreen.tsx +++ b/src/tui/screens/settings/SettingsPathsScreen.tsx @@ -19,6 +19,7 @@ import { useRouter } from '../../router.js'; import { useAppContext } from '../../app-context.js'; import { Panel, Form } from '../../components/index.js'; import { useSettingsOperation } from '../../hooks/index.js'; +import { DEFAULT_CHANGES_PATH, DEFAULT_SQL_PATH } from '../../../core/settings/defaults.js'; /** * SettingsPathsScreen component. @@ -36,7 +37,7 @@ export function SettingsPathsScreen({ params: _params }: ScreenProps): ReactElem // Get current paths config const paths = useMemo(() => { - if (!settingsManager) return { sql: './sql', changes: './changes' }; + if (!settingsManager) return { sql: DEFAULT_SQL_PATH, changes: DEFAULT_CHANGES_PATH }; return settingsManager.getPaths(); @@ -49,15 +50,15 @@ export function SettingsPathsScreen({ params: _params }: ScreenProps): ReactElem key: 'sql', label: 'SQL Path', type: 'text', - defaultValue: paths.sql ?? './sql', - placeholder: './sql', + defaultValue: paths.sql ?? DEFAULT_SQL_PATH, + placeholder: DEFAULT_SQL_PATH, }, { key: 'changes', label: 'Changes Path', type: 'text', - defaultValue: paths.changes ?? './changes', - placeholder: './changes', + defaultValue: paths.changes ?? DEFAULT_CHANGES_PATH, + placeholder: DEFAULT_CHANGES_PATH, }, ], [paths], @@ -68,8 +69,8 @@ export function SettingsPathsScreen({ params: _params }: ScreenProps): ReactElem async (values: FormValues) => { await execute({ - sql: String(values['sql'] || './sql'), - changes: String(values['changes'] || './changes'), + sql: String(values['sql'] || DEFAULT_SQL_PATH), + changes: String(values['changes'] || DEFAULT_CHANGES_PATH), }); }, diff --git a/src/tui/utils/change-loader.ts b/src/tui/utils/change-loader.ts index dbf1b296..7bd86d06 100644 --- a/src/tui/utils/change-loader.ts +++ b/src/tui/utils/change-loader.ts @@ -18,7 +18,9 @@ import type { Config, Settings } from '../../core/index.js'; import type { Change, ChangeStatus, ChangeListItem } from '../../core/change/types.js'; import type { NoormDatabase } from '../../core/shared/index.js'; -import { discoverChanges } from '../../core/change/parser.js'; +import { failureListItem, scanChanges } from '../../core/change/parser.js'; +import type { ChangeParseFailure } from '../../core/change/parser.js'; +import { isPendingChange } from '../../core/change/types.js'; import { ChangeHistory } from '../../core/change/history.js'; import { createConnection } from '../../core/connection/factory.js'; import { resolveChangesDir, resolveSqlDir } from './paths.js'; @@ -29,6 +31,9 @@ import { resolveChangesDir, resolveSqlDir } from './paths.js'; export interface ChangesWithStatus { changes: Change[]; statuses: Map; + + /** Folders on disk that could not be parsed; they must not vanish from the views */ + failures: ChangeParseFailure[]; } /** @@ -52,7 +57,7 @@ export async function loadChangesWithStatus( projectRoot: string, ): Promise { - const changes = await discoverChanges( + const { changes, failures } = await scanChanges( resolveChangesDir(projectRoot, settings), resolveSqlDir(projectRoot, settings), ); @@ -68,7 +73,7 @@ export async function loadChangesWithStatus( await conn.destroy(); - return { changes, statuses }; + return { changes, statuses, failures }; } @@ -80,8 +85,13 @@ export async function loadChangesWithStatus( export function buildPendingChangeList( changes: Change[], statuses: Map, + failures: ChangeParseFailure[] = [], ): ChangeListItem[] { + const unreadable = failures + .map((failure) => failureListItem(failure, statuses.get(failure.name))) + .filter(isPendingChange); + return changes .filter((cs) => { @@ -90,7 +100,7 @@ export function buildPendingChangeList( return !status || status.status === 'pending' || status.status === 'reverted'; }) - .map((cs) => ({ + .map((cs): ChangeListItem => ({ name: cs.name, path: cs.path, date: cs.date, @@ -105,6 +115,7 @@ export function buildPendingChangeList( changeFiles: cs.changeFiles, revertFiles: cs.revertFiles, })) + .concat(unreadable) .sort((a, b) => { const dateA = a.date?.getTime() ?? 0; @@ -177,9 +188,10 @@ export function buildAppliedChangeList( export function buildMergedChangeList( changes: Change[], statuses: Map, + failures: ChangeParseFailure[] = [], ): ChangeListItem[] { - const merged: ChangeListItem[] = []; + const merged: ChangeListItem[] = failures.map((failure) => failureListItem(failure, statuses.get(failure.name))); for (const cs of changes) { @@ -213,6 +225,12 @@ export function buildMergedChangeList( } + for (const failure of failures) { + + diskNames[failure.name] = true; + + } + for (const [name, status] of statuses) { if (!diskNames[name]) { diff --git a/src/tui/utils/paths.ts b/src/tui/utils/paths.ts index d460e5c3..f8150f2c 100644 --- a/src/tui/utils/paths.ts +++ b/src/tui/utils/paths.ts @@ -13,21 +13,22 @@ import { join } from 'path'; import type { Settings } from '../../core/settings/types.js'; +import { DEFAULT_CHANGES_PATH, DEFAULT_SQL_PATH } from '../../core/settings/defaults.js'; /** * Resolve the absolute changes directory from settings. * - * Uses settings.paths.changes with fallback to 'changes'. + * Uses settings.paths.changes with fallback to DEFAULT_CHANGES_PATH. * * @example * ```typescript * const dir = resolveChangesDir('/project', settings); - * // '/project/changes' or '/project/custom-changes' + * // '/project/db/changes' or '/project/custom-changes' * ``` */ export function resolveChangesDir(projectRoot: string, settings: Settings | null): string { - const changesPath = settings?.paths?.changes ?? 'changes'; + const changesPath = settings?.paths?.changes ?? DEFAULT_CHANGES_PATH; return join(projectRoot, changesPath); @@ -36,17 +37,17 @@ export function resolveChangesDir(projectRoot: string, settings: Settings | null /** * Resolve the absolute SQL directory from settings. * - * Uses settings.paths.sql with fallback to 'sql'. + * Uses settings.paths.sql with fallback to DEFAULT_SQL_PATH. * * @example * ```typescript * const dir = resolveSqlDir('/project', settings); - * // '/project/sql' or '/project/custom-sql' + * // '/project/db/sql' or '/project/custom-sql' * ``` */ export function resolveSqlDir(projectRoot: string, settings: Settings | null): string { - const sqlPath = settings?.paths?.sql ?? 'sql'; + const sqlPath = settings?.paths?.sql ?? DEFAULT_SQL_PATH; return join(projectRoot, sqlPath); diff --git a/tests/cli/run/_setup.ts b/tests/cli/run/_setup.ts index 499be847..6f91fc63 100644 --- a/tests/cli/run/_setup.ts +++ b/tests/cli/run/_setup.ts @@ -59,7 +59,7 @@ export async function setupProject(): Promise { NOORM_CONNECTION_DIALECT: 'sqlite', NOORM_CONNECTION_DATABASE: dbPath, NOORM_PATHS_SQL: './sql', - NOORM_PATHS_CHANGESETS: './changes', + NOORM_PATHS_CHANGES: './changes', NOORM_NAME: '__test__', NOORM_ISTEST: 'true', NOORM_IDENTITY: 'cli-run-tester', diff --git a/tests/cli/screens/change/change-dry-run.test.tsx b/tests/cli/screens/change/change-dry-run.test.tsx index e03f0fe6..db3e46be 100644 --- a/tests/cli/screens/change/change-dry-run.test.tsx +++ b/tests/cli/screens/change/change-dry-run.test.tsx @@ -228,10 +228,10 @@ describe('cli: ChangeFFScreen global modes', () => { // A real pending change on disk: the screen refuses to fast-forward // template-only or empty changes before it ever builds a manager. - const changeDir = join(tempDir, 'changes', 'add-widgets-table', 'change'); + const changeDir = join(tempDir, 'db', 'changes', 'add-widgets-table', 'change'); await mkdir(changeDir, { recursive: true }); - await mkdir(join(tempDir, 'sql'), { recursive: true }); + await mkdir(join(tempDir, 'db', 'sql'), { recursive: true }); await writeFile( join(changeDir, '001.sql'), 'CREATE TABLE widgets (id INTEGER PRIMARY KEY)', diff --git a/tests/cli/screens/init/init-flow.test.tsx b/tests/cli/screens/init/init-flow.test.tsx index 5313e1d2..3f82e839 100644 --- a/tests/cli/screens/init/init-flow.test.tsx +++ b/tests/cli/screens/init/init-flow.test.tsx @@ -145,6 +145,7 @@ mock.module('../../../../src/core/settings/manager.js', () => { SettingsManager: class MockSettingsManager { init = vi.fn(() => Promise.resolve()); + setPaths = vi.fn(() => Promise.resolve()); load = vi.fn(() => Promise.resolve()); exists = vi.fn(() => Promise.resolve(false)); isLoaded = true; diff --git a/tests/cli/screens/init/init-screen.test.tsx b/tests/cli/screens/init/init-screen.test.tsx index 310859e8..3df5803e 100644 --- a/tests/cli/screens/init/init-screen.test.tsx +++ b/tests/cli/screens/init/init-screen.test.tsx @@ -101,6 +101,7 @@ mock.module('../../../../src/core/state/manager.js', () => ({ mock.module('../../../../src/core/settings/manager.js', () => ({ SettingsManager: vi.fn().mockImplementation(() => ({ init: vi.fn(() => Promise.resolve()), + setPaths: vi.fn(() => Promise.resolve()), load: vi.fn(() => Promise.resolve()), exists: vi.fn(() => Promise.resolve(false)), isLoaded: true, diff --git a/tests/cli/screens/init/project-setup.test.tsx b/tests/cli/screens/init/project-setup.test.tsx index 8cda610f..3afe58bd 100644 --- a/tests/cli/screens/init/project-setup.test.tsx +++ b/tests/cli/screens/init/project-setup.test.tsx @@ -53,7 +53,7 @@ describe('cli: screens/init/ProjectSetup', () => { , ); - expect(lastFrame()).toContain('./sql'); + expect(lastFrame()).toContain('./db/sql'); }); @@ -65,7 +65,20 @@ describe('cli: screens/init/ProjectSetup', () => { , ); - expect(lastFrame()).toContain('./changes'); + expect(lastFrame()).toContain('./db/changes'); + + }); + + it('should list the agent steering files it will create', () => { + + const { lastFrame } = render( + + {}} onSkipConfig={() => {}} onCancel={() => {}} /> + , + ); + + expect(lastFrame()).toContain('./db/AGENTS.md'); + expect(lastFrame()).toContain('./db/CLAUDE.md'); }); diff --git a/tests/cli/update.test.ts b/tests/cli/update.test.ts new file mode 100644 index 00000000..45482ed0 --- /dev/null +++ b/tests/cli/update.test.ts @@ -0,0 +1,191 @@ +/** + * The subprocess runs the dist build, whose version is 0.0.0-dev, so the + * update check answers "up to date" without touching the network. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'bun:test'; +import { mkdir } from 'node:fs/promises'; + +import { downloadProgressRenderer, updateStatus } from '../../src/cli/update.js'; +import { TMP_BASE, cleanupProject, runCli, setupProject, type TestProject } from './run/_setup.js'; + +const MB = 1024 * 1024; + +describe('cli: update status lines', () => { + + function status(opts: { isTty: boolean; json: boolean }) { + + const writes: string[] = []; + + return { writes, status: updateStatus((text) => writes.push(text), opts) }; + + } + + it('should replace a pending line in place on a terminal', () => { + + const { writes, status: lines } = status({ isTty: true, json: false }); + + lines.pending('Checking for updates...'); + lines.say('Update available: 1.5.0 → 1.5.1'); + + expect(writes[0]).toBe('Checking for updates...'); + expect(writes[1]?.startsWith('\rUpdate available: 1.5.0 → 1.5.1')).toBe(true); + expect(writes[1]?.endsWith('\n')).toBe(true); + + }); + + it('should print every line in full off a terminal', () => { + + const { writes, status: lines } = status({ isTty: false, json: false }); + + lines.pending('Checking for updates...'); + lines.say('Update available: 1.5.0 → 1.5.1'); + + expect(writes).toEqual(['Checking for updates...\n', 'Update available: 1.5.0 → 1.5.1\n']); + + }); + + it('should print nothing under --json', () => { + + const { writes, status: lines } = status({ isTty: false, json: true }); + + lines.pending('Checking for updates...'); + lines.say('Already on the latest version (1.5.1)'); + lines.endLine(); + + expect(writes).toEqual([]); + + }); + + // A resumed download keeps the progress it had, so the notice must not overwrite it. + it('should put a retry notice on its own line below the download line', () => { + + const { writes, status: lines } = status({ isTty: true, json: false }); + const render = downloadProgressRenderer(lines.raw, true); + + render?.({ received: 5 * MB, total: 10 * MB }); + lines.endLine(); + lines.say('download stalled — resuming (attempt 2/5)...'); + + expect(writes[1]).toBe('\n'); + expect(writes[2]).toBe('download stalled — resuming (attempt 2/5)...\n'); + + }); + + it('should end an open download line before an error, and add nothing once it is closed', () => { + + const { writes, status: lines } = status({ isTty: true, json: false }); + const render = downloadProgressRenderer(lines.raw, true); + + render?.({ received: 5 * MB, total: 10 * MB }); + lines.endLine(); + lines.endLine(); + + expect(writes.slice(1)).toEqual(['\n']); + + }); + +}); + +describe('cli: update download progress', () => { + + function render(isTty: boolean) { + + const writes: string[] = []; + const renderer = downloadProgressRenderer((text) => writes.push(text), isTty); + + return { writes, renderer }; + + } + + it('should render nothing off a terminal, where every tick would be its own line', () => { + + expect(render(false).renderer).toBeNull(); + + }); + + it('should rewrite one line in place while the download runs', () => { + + const { writes, renderer } = render(true); + + renderer?.({ received: 5 * MB, total: 10 * MB }); + renderer?.({ received: 6 * MB, total: 10 * MB }); + + expect(writes[0]?.startsWith('\rDownloading 5.0 / 10.0 MB (50%)')).toBe(true); + expect(writes[1]?.startsWith('\rDownloading 6.0 / 10.0 MB (60%)')).toBe(true); + expect(writes.some((text) => text.includes('\n'))).toBe(false); + + }); + + // A rewrite shorter than the line it replaces must not leave its tail behind. + it('should pad every rewrite to the same width', () => { + + const { writes, renderer } = render(true); + + renderer?.({ received: 999 * MB, total: 1000 * MB }); + renderer?.({ received: 1 * MB, total: 0 }); + + expect(writes[1]?.length).toBe(writes[0]?.length); + + }); + + it('should end on a Downloaded line and ignore ticks after it', () => { + + const { writes, renderer } = render(true); + + renderer?.({ received: 10 * MB, total: 10 * MB }); + renderer?.({ received: 10 * MB, total: 10 * MB }); + + expect(writes).toHaveLength(1); + expect(writes[0]?.startsWith('\rDownloaded 10.0 MB (100%)')).toBe(true); + expect(writes[0]?.endsWith('\n')).toBe(true); + + }); + + it('should show megabytes alone when the size is unknown', () => { + + const { writes, renderer } = render(true); + + renderer?.({ received: 3 * MB, total: 0 }); + + expect(writes[0]?.startsWith('\rDownloading 3.0 MB')).toBe(true); + expect(writes[0]).not.toContain('%'); + + }); + +}); + +describe('cli: update --json', () => { + + let project: TestProject; + + beforeAll(async () => { + + await mkdir(TMP_BASE, { recursive: true }); + + }); + + beforeEach(async () => { + + project = await setupProject(); + + }); + + afterEach(async () => { + + await cleanupProject(project); + + }); + + // Status lines printed ahead of the document made `noorm update --json | jq` fail. + it('should write exactly one JSON document to stdout', () => { + + const { stdout, status } = runCli(project, ['update', '--json']); + + const payload = JSON.parse(stdout); + + expect(status).toBe(0); + expect(payload).toMatchObject({ success: true, updateAvailable: false, installed: false }); + + }); + +}); diff --git a/tests/cli/utils/change-context.test.ts b/tests/cli/utils/change-context.test.ts index c1f8926e..eb9d5bc5 100644 --- a/tests/cli/utils/change-context.test.ts +++ b/tests/cli/utils/change-context.test.ts @@ -115,8 +115,8 @@ describe('tui: createChangeManager', () => { capturedContexts.length = 0; tempDir = await mkdtemp(join(tmpdir(), 'noorm-change-context-test-')); - changesDir = join(tempDir, 'changes'); - sqlDir = join(tempDir, 'sql'); + changesDir = join(tempDir, 'db', 'changes'); + sqlDir = join(tempDir, 'db', 'sql'); await mkdir(changesDir, { recursive: true }); await mkdir(sqlDir, { recursive: true }); diff --git a/tests/cli/utils/change-loader.test.ts b/tests/cli/utils/change-loader.test.ts new file mode 100644 index 00000000..34168897 --- /dev/null +++ b/tests/cli/utils/change-loader.test.ts @@ -0,0 +1,83 @@ +import { describe, it, expect } from 'bun:test'; + +import { buildMergedChangeList, buildPendingChangeList } from '../../../src/tui/utils/change-loader.js'; +import type { ChangeStatus } from '../../../src/core/change/types.js'; +import type { ChangeParseFailure } from '../../../src/core/change/parser.js'; + +function status(name: string, state: ChangeStatus['status']): ChangeStatus { + + return { + name, + status: state, + appliedAt: null, + appliedBy: null, + revertedAt: null, + errorMessage: null, + }; + +} + +const broken: ChangeParseFailure = { + name: '2026-02-01-broken', + path: '/project/db/changes/2026-02-01-broken', + message: "Manifest 'x.txt' references missing file: views/missing.sql", +}; + +describe('tui: change loader with unreadable changes', () => { + + describe('buildPendingChangeList', () => { + + it('should list an unreadable change that never ran, so the screens do not say there is nothing pending', () => { + + const pending = buildPendingChangeList([], new Map(), [broken]); + + expect(pending.map((c) => c.name)).toEqual(['2026-02-01-broken']); + expect(pending[0]?.status).toBe('failed'); + expect(pending[0]?.errorMessage).toContain('missing.sql'); + + }); + + it('should list an unreadable change that was reverted', () => { + + const statuses = new Map([[broken.name, status(broken.name, 'reverted')]]); + + expect(buildPendingChangeList([], statuses, [broken]).map((c) => c.name)).toEqual([broken.name]); + + }); + + it('should not list an unreadable change that is applied', () => { + + const statuses = new Map([[broken.name, status(broken.name, 'success')]]); + + expect(buildPendingChangeList([], statuses, [broken])).toEqual([]); + + }); + + }); + + describe('buildMergedChangeList', () => { + + it('should show an unreadable change as failed, not orphaned', () => { + + const merged = buildMergedChangeList([], new Map(), [broken]); + + expect(merged).toHaveLength(1); + expect(merged[0]?.status).toBe('failed'); + expect(merged[0]?.orphaned).toBe(false); + + }); + + it('should keep the recorded status of an applied change whose files can no longer be read', () => { + + const statuses = new Map([[broken.name, status(broken.name, 'success')]]); + const merged = buildMergedChangeList([], statuses, [broken]); + + expect(merged).toHaveLength(1); + expect(merged[0]?.status).toBe('success'); + expect(merged[0]?.orphaned).toBe(false); + + }); + + }); + +}); diff --git a/tests/core/change/manager.test.ts b/tests/core/change/manager.test.ts index 5fe234af..857400f1 100644 --- a/tests/core/change/manager.test.ts +++ b/tests/core/change/manager.test.ts @@ -701,4 +701,179 @@ describe('change: manager', () => { }); + // sqlite's prepare() runs only the first statement and reports success + describe('multi-statement change files', () => { + + it('should run every statement of a change file on sqlite', async () => { + + await createTestChange('multi-ok', [{ + name: '001.sql', + content: 'CREATE TABLE multi_first (id INTEGER PRIMARY KEY);\nCREATE TABLE multi_second (id INTEGER PRIMARY KEY);', + }]); + + const manager = new ChangeManager(buildContext()); + const result = await manager.run('multi-ok'); + + expect(result.status).toBe('success'); + expect(await tableExists('multi_first')).toBe(true); + expect(await tableExists('multi_second')).toBe(true); + + }); + + it('should fail the change and name the statement when a later statement fails', async () => { + + await createTestChange('multi-bad', [{ + name: '001.sql', + content: 'CREATE TABLE multi_bad_first (id INTEGER PRIMARY KEY);\nSELECT * FROM no_such_table_xyz;', + }]); + + const manager = new ChangeManager(buildContext()); + const result = await manager.run('multi-bad'); + + expect(result.status).toBe('failed'); + expect(result.error).toContain('[statement 2 of 2]'); + + }); + + }); + + describe('changes that cannot be parsed', () => { + + async function createBrokenManifestChange(name: string): Promise { + + await createTestChange(name, [{ name: '001_views.txt', content: 'views/missing.sql\n' }]); + + } + + it('should fail ff at the broken change instead of reporting success', async () => { + + await createTestChange('2025-01-01-good', [ + { name: '001.sql', content: 'CREATE TABLE parse_good (id INTEGER PRIMARY KEY)' }, + ]); + await createBrokenManifestChange('2025-01-02-broken'); + await createTestChange('2025-01-03-after', [ + { name: '001.sql', content: 'CREATE TABLE parse_after (id INTEGER PRIMARY KEY)' }, + ]); + + const manager = new ChangeManager(buildContext()); + const result = await manager.ff(); + + expect(result.status).toBe('partial'); + expect(result.executed).toBe(1); + expect(result.failed).toBe(1); + + const broken = result.changes.find((c) => c.name === '2025-01-02-broken'); + + expect(broken?.status).toBe('failed'); + expect(broken?.error).toContain('missing.sql'); + expect(await tableExists('parse_good')).toBe(true); + expect(await tableExists('parse_after')).toBe(false); + + }); + + it('should ignore a folder that has neither change/ nor revert/', async () => { + + await mkdir(join(changesDir, 'lib'), { recursive: true }); + await writeFile(join(changesDir, 'lib', 'shared.sql'), 'SELECT 1'); + await createTestChange('2025-07-01-real', [ + { name: '001.sql', content: 'CREATE TABLE parse_real (id INTEGER PRIMARY KEY)' }, + ]); + + const manager = new ChangeManager(buildContext()); + const result = await manager.ff(); + + expect((await manager.list()).map((c) => c.name)).toEqual(['2025-07-01-real']); + expect(result.status).toBe('success'); + expect(result.failed).toBe(0); + expect(await tableExists('parse_real')).toBe(true); + + }); + + it('should fail ff --dry-run for the same change', async () => { + + await createBrokenManifestChange('2025-02-01-broken'); + + const manager = new ChangeManager(buildContext()); + const result = await manager.ff({ dryRun: true }); + + expect(result.status).toBe('failed'); + expect(result.failed).toBe(1); + + }); + + it('should name the missing file, not "not found", when run is asked for the broken change', async () => { + + await createBrokenManifestChange('2025-05-01-broken'); + + const manager = new ChangeManager(buildContext()); + const [, err] = await attempt(() => manager.run('2025-05-01-broken')); + + expect(err?.message).toContain('missing.sql'); + expect(err?.message).not.toContain('not found'); + + }); + + it('should list the unapplied broken change as failed with the parse error', async () => { + + await createBrokenManifestChange('2025-03-01-broken'); + + const manager = new ChangeManager(buildContext()); + const list = await manager.list(); + const item = list.find((c) => c.name === '2025-03-01-broken'); + + expect(item?.status).toBe('failed'); + expect(item?.errorMessage).toContain('missing.sql'); + expect(item?.orphaned).toBe(false); + + }); + + it('should fail ff at a reverted change whose manifest later breaks', async () => { + + await mkdir(join(sqlDir, 'views'), { recursive: true }); + await writeFile(join(sqlDir, 'views', 'v.sql'), 'CREATE VIEW parse_reverted_view AS SELECT 1 AS a'); + await createTestChange( + '2025-06-01-reverted', + [{ name: '001_views.txt', content: 'views/v.sql\n' }], + [{ name: '001.sql', content: 'DROP VIEW IF EXISTS parse_reverted_view' }], + ); + + const manager = new ChangeManager(buildContext()); + + await manager.run('2025-06-01-reverted'); + await manager.revert('2025-06-01-reverted'); + await rm(join(sqlDir, 'views', 'v.sql')); + + const result = await manager.ff(); + + expect(result.status).toBe('failed'); + expect(result.changes[0]?.error).toContain('v.sql'); + expect(result.warnings?.join('\n') ?? '').not.toContain('is applied'); + + }); + + it('should keep an applied change visible and warn when its manifest later breaks', async () => { + + await mkdir(join(sqlDir, 'views'), { recursive: true }); + await writeFile(join(sqlDir, 'views', 'v.sql'), 'CREATE VIEW parse_view AS SELECT 1 AS a'); + await createTestChange('2025-04-01-applied', [{ name: '001_views.txt', content: 'views/v.sql\n' }]); + + const manager = new ChangeManager(buildContext()); + + expect((await manager.run('2025-04-01-applied')).status).toBe('success'); + + await rm(join(sqlDir, 'views', 'v.sql')); + + const item = (await manager.list()).find((c) => c.name === '2025-04-01-applied'); + const result = await manager.ff(); + + expect(item?.status).toBe('success'); + expect(item?.orphaned).toBe(false); + expect(result.status).toBe('success'); + expect(result.failed).toBe(0); + expect(result.warnings?.join('\n')).toContain('2025-04-01-applied'); + + }); + + }); + }); diff --git a/tests/core/project-init.test.ts b/tests/core/project-init.test.ts index 4d4462a1..bb537972 100644 --- a/tests/core/project-init.test.ts +++ b/tests/core/project-init.test.ts @@ -1,9 +1,14 @@ import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; -import { mkdtempSync, rmSync, existsSync, readFileSync } from 'node:fs'; +import { mkdtempSync, mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'node:fs'; import { tmpdir, homedir } from 'node:os'; import { join } from 'node:path'; -import { performProjectInit } from '../../src/core/project-init.js'; +import { performProjectInit, scaffoldDbLayout } from '../../src/core/project-init.js'; +import { DEFAULT_PATH_CONFIG } from '../../src/core/settings/defaults.js'; +import { SettingsManager } from '../../src/core/settings/manager.js'; +import { resolveChangesDir, resolveSqlDir } from '../../src/tui/utils/paths.js'; + +const hasGlobalIdentity = existsSync(join(homedir(), '.noorm', 'identity.key')); describe('core: performProjectInit', () => { @@ -33,8 +38,10 @@ describe('core: performProjectInit', () => { }); expect(result.success).toBe(true); - expect(existsSync(join(tmpDir, 'sql', '.gitkeep'))).toBe(true); - expect(existsSync(join(tmpDir, 'changes', '.gitkeep'))).toBe(true); + expect(existsSync(join(tmpDir, 'db', 'sql', '.gitkeep'))).toBe(true); + expect(existsSync(join(tmpDir, 'db', 'changes', '.gitkeep'))).toBe(true); + expect(existsSync(join(tmpDir, 'sql'))).toBe(false); + expect(existsSync(join(tmpDir, 'changes'))).toBe(false); expect(existsSync(join(tmpDir, '.noorm', 'settings.yml'))).toBe(true); expect(existsSync(join(tmpDir, '.noorm', '.gitignore'))).toBe(true); expect(existsSync(join(tmpDir, '.noorm', 'state'))).toBe(true); @@ -120,4 +127,144 @@ describe('core: performProjectInit', () => { }); + // Init makes the directories and writes `paths` separately. If they disagree, + // `build` reads a folder init never made and succeeds on zero files. + it.skipIf(!hasGlobalIdentity)('should write settings paths that resolve to the directories it created', async () => { + + await performProjectInit({ + projectRoot: tmpDir, + force: false, + identityInfo: null, + }); + + const manager = new SettingsManager(tmpDir); + const settings = await manager.load(); + + expect(settings.paths?.sql).toBe('./db/sql'); + expect(settings.paths?.changes).toBe('./db/changes'); + expect(resolveSqlDir(tmpDir, settings)).toBe(join(tmpDir, 'db', 'sql')); + expect(resolveChangesDir(tmpDir, settings)).toBe(join(tmpDir, 'db', 'changes')); + expect(existsSync(resolveSqlDir(tmpDir, settings))).toBe(true); + expect(existsSync(resolveChangesDir(tmpDir, settings))).toBe(true); + + }); + + it.skipIf(!hasGlobalIdentity)('should not overwrite agent steering files on re-init', async () => { + + await performProjectInit({ + projectRoot: tmpDir, + force: false, + identityInfo: null, + }); + + const agentsPath = join(tmpDir, 'db', 'AGENTS.md'); + const claudePath = join(tmpDir, 'db', 'CLAUDE.md'); + + writeFileSync(agentsPath, 'Always use PascalCase.\n'); + writeFileSync(claudePath, '@AGENTS.md\n\nUse plan mode.\n'); + + const second = await performProjectInit({ + projectRoot: tmpDir, + force: true, + identityInfo: null, + }); + + expect(readFileSync(agentsPath, 'utf-8')).toBe('Always use PascalCase.\n'); + expect(readFileSync(claudePath, 'utf-8')).toBe('@AGENTS.md\n\nUse plan mode.\n'); + expect(second.createdFiles).not.toContain('db/AGENTS.md'); + expect(second.createdFiles).not.toContain('db/CLAUDE.md'); + + }); + + it.skipIf(!hasGlobalIdentity)('should keep a legacy project layout on --force', async () => { + + mkdirSync(join(tmpDir, '.noorm'), { recursive: true }); + writeFileSync(join(tmpDir, '.noorm', 'settings.yml'), 'paths:\n sql: ./sql\n changes: ./changes\n'); + mkdirSync(join(tmpDir, 'sql'), { recursive: true }); + mkdirSync(join(tmpDir, 'changes'), { recursive: true }); + + const result = await performProjectInit({ + projectRoot: tmpDir, + force: true, + identityInfo: null, + }); + + const settings = await new SettingsManager(tmpDir).load(); + + expect(settings.paths).toEqual({ sql: './sql', changes: './changes' }); + expect(existsSync(join(tmpDir, 'db'))).toBe(false); + expect(result.createdFiles.some((f) => f.startsWith('db/'))).toBe(false); + + }); + +}); + +describe('core: scaffoldDbLayout', () => { + + let tmpDir: string; + + beforeEach(() => { + + tmpDir = mkdtempSync(join(tmpdir(), 'noorm-scaffold-')); + + }); + + afterEach(() => { + + rmSync(tmpDir, { recursive: true, force: true }); + + }); + + it('should nest sql/ and changes/ under db/', () => { + + const created = scaffoldDbLayout(tmpDir, DEFAULT_PATH_CONFIG); + + expect(created).toEqual([ + 'db/sql/.gitkeep', + 'db/changes/.gitkeep', + 'db/AGENTS.md', + 'db/CLAUDE.md', + ]); + expect(existsSync(join(tmpDir, 'db', 'sql'))).toBe(true); + expect(existsSync(join(tmpDir, 'db', 'changes'))).toBe(true); + + }); + + it('should template AGENTS.md for the user and import it from CLAUDE.md', () => { + + scaffoldDbLayout(tmpDir, DEFAULT_PATH_CONFIG); + + const agents = readFileSync(join(tmpDir, 'db', 'AGENTS.md'), 'utf-8'); + const claude = readFileSync(join(tmpDir, 'db', 'CLAUDE.md'), 'utf-8'); + + expect(agents).toContain('Use the `/noorm` skill for managing files here.'); + expect(agents).toContain('`/ignatius-modeling` skill'); + expect(agents).toContain('https://noorm.dev/modeling/'); + expect(agents).toContain(''); + expect(claude.split('\n')).toContain('@AGENTS.md'); + + }); + + it('should create nothing for a project on its own layout', () => { + + const created = scaffoldDbLayout(tmpDir, { sql: './sql', changes: './changes' }); + + expect(created).toEqual([]); + expect(existsSync(join(tmpDir, 'db'))).toBe(false); + + }); + + it('should leave an existing CLAUDE.md untouched and report only what it created', () => { + + mkdirSync(join(tmpDir, 'db'), { recursive: true }); + writeFileSync(join(tmpDir, 'db', 'CLAUDE.md'), 'mine\n'); + + const created = scaffoldDbLayout(tmpDir, DEFAULT_PATH_CONFIG); + + expect(readFileSync(join(tmpDir, 'db', 'CLAUDE.md'), 'utf-8')).toBe('mine\n'); + expect(created).toContain('db/AGENTS.md'); + expect(created).not.toContain('db/CLAUDE.md'); + + }); + }); diff --git a/tests/core/settings/manager.test.ts b/tests/core/settings/manager.test.ts index 54fdbe55..888a83f1 100644 --- a/tests/core/settings/manager.test.ts +++ b/tests/core/settings/manager.test.ts @@ -6,6 +6,7 @@ import { SettingsManager, resetSettingsManager, DEFAULT_SETTINGS, + DEFAULT_PATH_CONFIG, } from '../../../src/core/settings/index.js'; import type { Stage, Rule } from '../../../src/core/settings/types.js'; @@ -347,6 +348,78 @@ ui: }); + it('should keep the paths already in the file when overwriting', async () => { + + const { manager, cleanup } = createTestContext(); + + try { + + await manager.init(); + await manager.setPaths({ sql: './sql', changes: './changes' }); + + await manager.init(true); + + expect(manager.getPaths()).toEqual({ sql: './sql', changes: './changes' }); + expect(readFileSync(manager.settingsFilePath, 'utf-8')).toContain('./sql'); + + } + finally { + + cleanup(); + + } + + }); + + it('should give an overwritten file with no paths the default layout', async () => { + + const { manager, cleanup } = createTestContext(); + + try { + + await manager.init(); + writeFileSync(manager.settingsFilePath, 'build:\n include: []\n'); + + await manager.init(true); + + expect(manager.getPaths()).toEqual(DEFAULT_PATH_CONFIG); + + } + finally { + + cleanup(); + + } + + }); + + // The resolved view merges NOORM_* variables; saving it would turn one + // shell's override into the project's settings. + it('should not write an environment override into the file', async () => { + + const { manager, cleanup } = createTestContext(); + + process.env['NOORM_PATHS_SQL'] = './env-sql'; + + try { + + await manager.init(); + + const written = readFileSync(manager.settingsFilePath, 'utf-8'); + + expect(written).not.toContain('env-sql'); + expect(written).toContain('./db/sql'); + + } + finally { + + delete process.env['NOORM_PATHS_SQL']; + cleanup(); + + } + + }); + }); describe('exists', () => { @@ -441,8 +514,8 @@ ui: const paths = manager.getPaths(); - expect(paths.sql).toBe('./sql'); - expect(paths.changes).toBe('./changes'); + expect(paths.sql).toBe('./db/sql'); + expect(paths.changes).toBe('./db/changes'); } finally { diff --git a/tests/core/update/skill.test.ts b/tests/core/update/skill.test.ts new file mode 100644 index 00000000..d354a36b --- /dev/null +++ b/tests/core/update/skill.test.ts @@ -0,0 +1,184 @@ +import { describe, it, expect, beforeEach, afterEach, spyOn } from 'bun:test'; +import { chmod, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { attempt } from '@logosdx/utils'; + +import { offerSkill } from '../../../src/cli/update.js'; +import { + SKILL_INSTALLERS, + canPromptForSkill, + findSkillInstaller, + installSkill, + skillCommand, +} from '../../../src/core/update/skill.js'; + +const [NPX, GH] = SKILL_INSTALLERS; + +describe('update: skill', () => { + + let binDir: string; + let originalPath: string | undefined; + let originalArgsFile: string | undefined; + + /** Put a fake program on the test PATH that records its arguments and exits with `code`. */ + async function fakeBin(name: string, code = 0): Promise { + + await writeFile(join(binDir, name), `#!/bin/sh\nprintf '%s\\n' "$@" > "$FAKE_ARGS_FILE"\nexit ${code}\n`); + await chmod(join(binDir, name), 0o755); + + } + + beforeEach(async () => { + + binDir = await mkdtemp(join(tmpdir(), 'noorm-skill-bin-')); + originalPath = process.env['PATH']; + originalArgsFile = process.env['FAKE_ARGS_FILE']; + + // Only the fakes are reachable, so the machine's own npx and gh never count. + process.env['PATH'] = binDir; + process.env['FAKE_ARGS_FILE'] = join(binDir, 'args.txt'); + + }); + + afterEach(async () => { + + process.env['PATH'] = originalPath; + + if (originalArgsFile === undefined) delete process.env['FAKE_ARGS_FILE']; + else process.env['FAKE_ARGS_FILE'] = originalArgsFile; + + await rm(binDir, { recursive: true, force: true }); + + }); + + describe('findSkillInstaller', () => { + + it.skipIf(process.platform === 'win32')('should prefer npx when both npx and gh are installed', async () => { + + await fakeBin('npx'); + await fakeBin('gh'); + + expect(findSkillInstaller()).toBe(NPX); + + }); + + // Node.js is not required to run the noorm binary, so a machine + // without npx may still have the GitHub CLI. + it.skipIf(process.platform === 'win32')('should fall back to gh when npx is missing', async () => { + + await fakeBin('gh'); + + expect(findSkillInstaller()).toBe(GH); + + }); + + it('should find nothing when neither npx nor gh is installed', () => { + + expect(findSkillInstaller()).toBeNull(); + + }); + + }); + + describe('installSkill', () => { + + it.skipIf(process.platform === 'win32')('should run `npx skills add` for the noorm skill and return its exit code', async () => { + + await fakeBin('npx', 3); + + const code = await installSkill(NPX); + const args = (await readFile(join(binDir, 'args.txt'), 'utf-8')).trim().split('\n'); + + expect(code).toBe(3); + expect(args).toEqual(['skills', 'add', 'noormdev/noorm/skills']); + + }); + + // An exact skill path skips gh's search of the whole repository. + it.skipIf(process.platform === 'win32')('should run `gh skill install` with the skill path', async () => { + + await fakeBin('gh'); + + const code = await installSkill(GH); + const args = (await readFile(join(binDir, 'args.txt'), 'utf-8')).trim().split('\n'); + + expect(code).toBe(0); + expect(args).toEqual(['skill', 'install', 'noormdev/noorm', 'skills/noorm']); + + }); + + it.skipIf(process.platform === 'win32')('should reject when the program cannot be started', async () => { + + const [, err] = await attempt(() => installSkill(NPX)); + + expect(err).toBeInstanceOf(Error); + expect(err).toHaveProperty('code', 'ENOENT'); + + }); + + }); + + describe('canPromptForSkill', () => { + + it('should prompt only when both stdin and stdout are terminals', () => { + + expect(canPromptForSkill({ stdinTty: true, stdoutTty: true })).toBe(true); + expect(canPromptForSkill({ stdinTty: false, stdoutTty: true })).toBe(false); + expect(canPromptForSkill({ stdinTty: true, stdoutTty: false })).toBe(false); + expect(canPromptForSkill({ stdinTty: undefined, stdoutTty: undefined })).toBe(false); + + }); + + }); + + // The prompt needs a terminal; these are the paths that stay quiet or print commands. + describe('offerSkill', () => { + + async function printedBy(run: () => Promise): Promise { + + const write = spyOn(process.stdout, 'write').mockImplementation(() => true); + + await run(); + + const printed = write.mock.calls.map((call) => String(call[0])).join(''); + + write.mockRestore(); + + return printed; + + } + + it('should print nothing for --json, even on a terminal', async () => { + + expect(await printedBy(() => offerSkill(true, true))).toBe(''); + + }); + + it.skipIf(process.platform === 'win32')('should print the command it would run when there is no terminal', async () => { + + await fakeBin('gh'); + + const printed = await printedBy(() => offerSkill(false, false)); + + expect(printed).toContain(skillCommand(GH)); + expect(printed).not.toContain(skillCommand(NPX)); + + }); + + // With nothing to run, the user needs both ways in and where to get them. + it('should print both commands and where to get each program when neither is installed', async () => { + + const printed = await printedBy(() => offerSkill(false, true)); + + expect(printed).toContain(skillCommand(NPX)); + expect(printed).toContain(skillCommand(GH)); + expect(printed).toContain('https://nodejs.org'); + expect(printed).toContain('https://cli.github.com'); + + }); + + }); + +}); diff --git a/tests/integration/change/mssql-batches.test.ts b/tests/integration/change/mssql-batches.test.ts new file mode 100644 index 00000000..a8b430fb --- /dev/null +++ b/tests/integration/change/mssql-batches.test.ts @@ -0,0 +1,174 @@ +/** + * Integration test: change files with `GO` batch separators on live SQL Server. + * + * `GO` is a client-side directive, not T-SQL, so the change executor has to + * split it as the runner does for `sql/` files. Drives `ChangeManager`'s + * public API and asks the server itself which tables exist. + * + * Requires the docker-compose.test.yml SQL Server container (11433); skips + * itself when unreachable. + */ +import { describe, it, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtemp, rm, mkdir, writeFile } from 'fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { Kysely, sql } from 'kysely'; + +import { attempt } from '@logosdx/utils'; + +import { ChangeManager } from '../../../src/core/change/manager.js'; +import { migrateSchema } from '../../../src/core/version/schema/index.js'; +import { getNoormTables, noormDb } from '../../../src/core/shared/index.js'; +import { resetLockManager } from '../../../src/core/lock/index.js'; +import type { NoormDatabase } from '../../../src/core/shared/index.js'; +import type { ConnectionResult } from '../../../src/core/connection/types.js'; +import type { ChangeContext } from '../../../src/core/change/types.js'; + +import { createTestConnection, isContainerRunning } from '../../utils/db.js'; + +const CONFIG_NAME = '__change_mssql_batches__'; +const FIRST_TABLE = 'noorm_it_batch_first'; +const SECOND_TABLE = 'noorm_it_batch_second'; +const OK_CHANGE = '2026-01-01-mssql-batches'; +const BAD_CHANGE = '2026-01-02-mssql-bad-batch'; + +describe('change: GO batches on live mssql', () => { + + let conn: ConnectionResult | null = null; + let reachable = false; + let tempDir: string; + let changesDir: string; + let sqlDir: string; + + beforeAll(async () => { + + reachable = await isContainerRunning('mssql'); + + if (!reachable) return; + + resetLockManager(); + + conn = await createTestConnection('mssql'); + + await migrateSchema(conn.db as unknown as Kysely, 'mssql'); + + tempDir = await mkdtemp(join(tmpdir(), 'noorm-mssql-change-')); + changesDir = join(tempDir, 'changes'); + sqlDir = join(tempDir, 'sql'); + + await mkdir(join(changesDir, OK_CHANGE, 'change'), { recursive: true }); + await mkdir(join(changesDir, BAD_CHANGE, 'change'), { recursive: true }); + await mkdir(sqlDir, { recursive: true }); + + await writeFile( + join(changesDir, OK_CHANGE, 'change', '001_two_tables.sql'), + `CREATE TABLE ${FIRST_TABLE} (id INT PRIMARY KEY)\nGO\nCREATE TABLE ${SECOND_TABLE} (id INT PRIMARY KEY)\nGO\n`, + ); + await writeFile( + join(changesDir, BAD_CHANGE, 'change', '001_bad_second_batch.sql'), + 'SELECT 1\nGO\nSELECT * FROM noorm_it_no_such_table\nGO\n', + ); + + }); + + afterAll(async () => { + + if (!conn) return; + + const db = conn.db as unknown as Kysely; + const tables = getNoormTables('mssql'); + + await attempt(() => sql.raw(`DROP TABLE IF EXISTS ${FIRST_TABLE}`).execute(db)); + await attempt(() => sql.raw(`DROP TABLE IF EXISTS ${SECOND_TABLE}`).execute(db)); + + // Child rows first — executions carries an FK to change. + await attempt(() => + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (noormDb(db, 'mssql') as any) + .deleteFrom(tables.executions) + .where('change_id', 'in', + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (noormDb(db, 'mssql') as any) + .selectFrom(tables.change) + .select('id') + .where('config_name', '=', CONFIG_NAME), + ) + .execute(), + ); + + await attempt(() => + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (noormDb(db, 'mssql') as any) + .deleteFrom(tables.change) + .where('config_name', '=', CONFIG_NAME) + .execute(), + ); + + await conn.destroy(); + await rm(tempDir, { recursive: true, force: true }); + + resetLockManager(); + + }); + + function buildContext(): ChangeContext { + + return { + db: conn!.db as unknown as Kysely, + configName: CONFIG_NAME, + identity: { name: 'Test User', email: 'test@example.com', source: 'config' }, + projectRoot: tempDir, + changesDir, + sqlDir, + access: { user: 'admin', agent: 'admin' }, + channel: 'user', + dialect: 'mssql', + }; + + } + + async function tableExists(name: string): Promise { + + const db = conn!.db as unknown as Kysely; + + const rows = await sql<{ c: number }>` + SELECT COUNT(*) AS c FROM sys.tables WHERE name = ${name} + `.execute(db); + + return Number(rows.rows[0]?.c ?? 0) > 0; + + } + + it('should run every GO-separated batch of a change file', async () => { + + if (!reachable) { + + console.warn('Skipping mssql: container not reachable'); + + return; + + } + + const manager = new ChangeManager(buildContext()); + const result = await manager.run(OK_CHANGE); + + expect(result.error ?? null).toBeNull(); + expect(result.status).toBe('success'); + expect(await tableExists(FIRST_TABLE)).toBe(true); + expect(await tableExists(SECOND_TABLE)).toBe(true); + + }); + + it('should name the batch that failed', async () => { + + if (!reachable) return; + + const manager = new ChangeManager(buildContext()); + const result = await manager.run(BAD_CHANGE); + + expect(result.status).toBe('failed'); + expect(result.error).toContain('[batch 2 of 2]'); + + }); + +}); diff --git a/tests/integration/cli/setup.ts b/tests/integration/cli/setup.ts index 597826b7..91458121 100644 --- a/tests/integration/cli/setup.ts +++ b/tests/integration/cli/setup.ts @@ -390,7 +390,7 @@ export async function setupTestProject(): Promise { NOORM_CONNECTION_DIALECT: 'sqlite', NOORM_CONNECTION_DATABASE: dbPath, NOORM_PATHS_SQL: './sql', - NOORM_PATHS_CHANGESETS: './changes', + NOORM_PATHS_CHANGES: './changes', NOORM_NAME: '__test__', NOORM_ISTEST: 'true', // Bypass identity check in headless mode (CI has no ~/.noorm/identity.key) diff --git a/tests/sdk/run-build-filtering.test.ts b/tests/sdk/run-build-filtering.test.ts index 7fca2b2d..64292daa 100644 --- a/tests/sdk/run-build-filtering.test.ts +++ b/tests/sdk/run-build-filtering.test.ts @@ -171,10 +171,10 @@ async function makeSqlProject(): Promise { const projectRoot = await mkdtemp(join(tmpdir(), 'noorm-build-filter-')); - await mkdir(join(projectRoot, 'sql', 'a'), { recursive: true }); - await mkdir(join(projectRoot, 'sql', 'b'), { recursive: true }); - await writeFile(join(projectRoot, 'sql', 'a', '001_a.sql'), 'select 1;'); - await writeFile(join(projectRoot, 'sql', 'b', '001_b.sql'), 'select 1;'); + await mkdir(join(projectRoot, 'db', 'sql', 'a'), { recursive: true }); + await mkdir(join(projectRoot, 'db', 'sql', 'b'), { recursive: true }); + await writeFile(join(projectRoot, 'db', 'sql', 'a', '001_a.sql'), 'select 1;'); + await writeFile(join(projectRoot, 'db', 'sql', 'b', '001_b.sql'), 'select 1;'); return projectRoot; @@ -221,7 +221,7 @@ describe('sdk: RunNamespace.build() filtering', () => { const result = await run.build(); expect(result.files.map((f) => f.filepath)).toEqual([ - join(projectRoot, 'sql', 'a', '001_a.sql'), + join(projectRoot, 'db', 'sql', 'a', '001_a.sql'), ]); }); @@ -259,8 +259,8 @@ describe('sdk: RunNamespace.build() filtering', () => { const result = await run.build(); expect(result.files.map((f) => f.filepath).sort()).toEqual([ - join(projectRoot, 'sql', 'a', '001_a.sql'), - join(projectRoot, 'sql', 'b', '001_b.sql'), + join(projectRoot, 'db', 'sql', 'a', '001_a.sql'), + join(projectRoot, 'db', 'sql', 'b', '001_b.sql'), ].sort()); }); @@ -281,7 +281,7 @@ describe('sdk: RunNamespace.build() filtering', () => { const result = await run.build(); expect(result.files.map((f) => f.filepath)).toEqual([ - join(projectRoot, 'sql', 'a', '001_a.sql'), + join(projectRoot, 'db', 'sql', 'a', '001_a.sql'), ]); });