Import the lightcone-cli docs as a starting point

.github/workflows/build.yml

@@ -0,0 +1,26 @@
+name: Build Docs
+
+# Builds the site in strict mode, so a broken link or a nav entry
+# pointing at a missing page fails the PR rather than the deploy.
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+
+      - name: Install docs dependencies
+        run: uv sync --locked
+
+      - name: Build site
+        run: uv run zensical build --strict

.gitignore

@@ -0,0 +1,7 @@
+# Built site
+/site
+
+# Environment
+.venv/
+__pycache__/
+.cache/

README.md

@@ -1,2 +1,25 @@
 # docs
-Documentation for the Lightcone Research stack
+
+Documentation for the Lightcone Research stack, built with
+[Zensical](https://zensical.org/).
+
+## Preview locally
+
+The site only needs [uv](https://docs.astral.sh/uv/getting-started/installation):
+
+```bash
+uv sync
+uv run zensical serve        # live preview at http://127.0.0.1:8000
+uv run zensical build --strict   # what CI runs
+```
+
+## Layout
+
+- `zensical.toml` — site config and navigation
+- `docs/` — page sources (Markdown), plus `assets/` and `stylesheets/`
+
+## Origin
+
+The initial content is the `docs/` tree of
+[lightcone-cli](https://github.com/LightconeResearch/lightcone-cli) as of
+`main` @ `3aa823b`, imported unchanged.

docs/api/assets.md

@@ -0,0 +1,53 @@
+# lightcone.engine.assets
+
+One output: its directory, its manifest, and whether it is still
+current. The classification rule lives here, next to the manifest it
+reads and the hashes it compares — and it is the one place in the
+engine where a bug is quiet rather than loud, which is why it may not
+have two implementations.
+
+Source: `src/lightcone/engine/assets.py`.
+
+## Key symbols
+
+| Symbol | Role |
+|---|---|
+| `classify(...)` | The one rule: `current` / `behind` / `stale`, with the why. Two callers — the worker and the read-only walk. |
+| `Verdict.calls_for_a_remake(refresh=)` | The one place a state becomes an action: `stale` always, `behind` only when asked. |
+| `data_version(path)` | Content hash of a directory or file — computed in the worker, before anything is annexed. |
+| `Versions` | Per-run memo so a shared declared input hashes once, not once per dependent. |
+| `read(sidecar)` / `write(...)` | The manifest, `.<output_id>.manifest.json`. Both take the sidecar's own path, so a caller holding an output path has to say `manifest_path` out loud. |
+| `output_path(root, u, id, fmt)` | The output's file, guarded: any part that is not a single path component is refused, and so is a format that could not be an extension. |
+| `manifest_path(output)` | The sidecar beside it, named from the id alone — so it keeps its path, and its history, across a re-declared format. |
+| `ContentNotFetchedError` | An annexed file whose content is not in this clone, in either shape it takes. |
+
+## What must stay true
+
+- **One `classify`, two callers, one differing value.** The worker
+  hands live input digests; check mode hands `None` for anything
+  upstream that will run ("this is going to change"). That value is
+  the entire difference — never a second body of logic. History (the
+  foreign-write fact) enters the same way: computed by whoever has
+  git, handed in as a value.
+- **The comparison is fourfold**: `definition_version`, the declared
+  input *set* (separate on purpose — a dropped dependency moves
+  neither hash), each recorded input digest, then `env_version`.
+  `stale` wins over `behind`; `behind` does not propagate and a behind
+  upstream still feeds its dependents.
+- **A skip returns the *recorded* digest, never a recomputed one** —
+  on a bytes-free clone, rehashing dangling symlinks would quietly
+  report a different output.
+- **Unfetched content refuses loudly, in both shapes.** A pointer file
+  hashes to a well-formed digest of the wrong thing; a dangling
+  symlink drops out of an `is_file()` walk without a word. Both raise
+  `ContentNotFetchedError` naming `git annex get`; only dangling
+  symlinks are added back to the directory walk.
+- **`calls_for_a_remake` has three callers** (worker, check, the
+  cascade walk) and no inline re-spellings — the third copy is where
+  they start to disagree.
+
+## Tests
+
+`tests/test_assets.py` — pure; nothing on disk beyond `tmp_path`.
+The pointer-file and dangling-symlink traps are pinned against real
+annex shapes in `tests/test_dataset.py`.

docs/api/container.md

@@ -0,0 +1,76 @@
+# lightcone.engine.image & container
+
+The container hatch, split down the pure/impure line. `image.py` is
+what a containerized project *declares* and how that becomes an
+identity — pure, no subprocess anywhere. `container.py` is building,
+storing and entering images — impure, every command through
+`project._run`. The exec side (the mount table) lives with the other
+backends in `sandbox/oci.py`.
+
+Sources: `src/lightcone/engine/image.py`,
+`src/lightcone/engine/container.py`, `src/lightcone/engine/sandbox/oci.py`.
+
+## Key symbols
+
+| Symbol | Role |
+|---|---|
+| `image.declaration(root)` | The `[tool.lightcone.image]` table, validated — a closed key set (`base`, `apt-install`, `run-commands`, `env`), because every key is hashed. |
+| `image.tag(root)` | `lc-env-<16 hex>` over the rendered Containerfile *and* the identity document. |
+| `image.archive_path(root, tag)` | `.datalad/environments/<tag>/image` — the `datalad containers-add` layout. |
+| `container.build(root)` | Build + save + commit, idempotent; returns `(Runtime, "built" \| "present")`. |
+| `container.runtime_for_run(root, *, build)` | One function, two strictnesses: `lc build`/materialize-preflight may build and commit; the probe and worker only ever find, fetch, and load. |
+| `container.backend(...)` | The single construction point for the exec backend — the only mode branch. |
+| `container.sync(...)` | The in-container environment converge: network on, project `:rw`, host uv cache mounted, into `.lightcone/venv`. |
+| `Runtime` | Facts only — root/mode/name/tag/id/arch — never mechanism. |
+
+## What must stay true
+
+- **The user never sees a Containerfile.** The render exists only in a
+  transient build context; the image's `LABEL` carries the identity
+  document so the archive stays self-describing. There is deliberately
+  no `pip-install` key — the Python environment is the lock's
+  business, never the image's.
+- **The engine never enters the image.** The container is the
+  *recipe's* world: driver, git, annex, and classification stay the
+  host's `lc`; exactly two things run in-image — the sync and each
+  exec. Network is uncontrolled on every mechanism, symmetrically, and
+  the attestation says so — no consumer may read a promise into
+  "containerized".
+- **No project file enters the build context** — that is what makes
+  "code edits never rebuild" structural rather than incidental.
+- **The dataset is the store; runtime stores are caches.** Execution
+  pins the archive's config-blob **id** (readable with no runtime),
+  never a tag; a dropped archive never substitutes — a rebuild is a
+  new archive under a new id.
+- **Builds and archive commits happen only on a clean tree, and only
+  after the graph resolves** — a refusal over a typo must not cost a
+  minutes-long build, and `dataset.save` commits the whole index.
+- **The mount table is the mechanism** (`sandbox/oci.py`): project
+  `:ro`, the write scope `:rw` — the directory a recipe's output lands
+  in, or `results/` for a probe — declared inputs `:ro`, private HOME,
+  `--tmpfs /tmp`, over a `--read-only` rootfs — without that flag a
+  stray write *succeeds* into the ephemeral layer and vanishes while
+  the attestation claims `fs: declared`. Mounts are resolved source,
+  **declared** destination — the one policy shape that keeps its paths
+  unresolved, because they are addresses the recipe uses.
+- **Runtime differences are spellings, never shapes.** One
+  `OCIBackend`, data-parameterized; the podman family is stated once
+  (`_PODMAN_FAMILY`) and asked positively, so a new runtime falls
+  outside it by default. podman-hpc adds exactly one step (`migrate`,
+  outside the load branch) and joins `_SHARED_STORE_RUNTIMES`.
+  Detection order podman-hpc → podman → docker; docker's daemon is
+  probed at detection.
+- **The architecture gate refuses before the load** — a wrong-arch
+  `load` succeeds and then dies as `exec format error` deep inside a
+  recipe. Ignorance passes; a recorded mismatch refuses, naming the
+  fix.
+
+## Tests
+
+`tests/test_image.py` (pure: structure and ordering, tag sensitivity
+both ways, the `env_version` frame), `tests/test_container.py`
+(lifecycle against the stubbed `_run` — every refusal on recorded
+argv), `tests/test_sandbox_oci.py` (the mount table, pure), and
+`tests/test_container_smoke.py` — the runtime's answer, gated by
+`LC_CONTAINER_TESTS_REQUIRED=1` in CI, building a real image and
+proving the record on a bytes-free clone with a real `datalad rerun`.

docs/api/crate.md

@@ -0,0 +1,68 @@
+# lightcone.engine.crate
+
+The publication view: the repository described as a Workflow Run
+RO-Crate. The project *is* the crate — `ro-crate-metadata.json` sits at
+the root, describes what the repository already holds, and a deposit is
+`git archive`, not an export step. lc's manifests stay the canonical
+record; the crate is the same facts in schema.org vocabulary for
+archives and viewers that will never run `lc`.
+
+Source: `src/lightcone/engine/crate.py` (converged by
+`materialize._converge_crate`).
+
+## Key symbols
+
+| Symbol | Role |
+|---|---|
+| `render(root, graph, *, license, dsid, writer)` | The document, as bytes. A pure function of repository state — git comes in as the `writer` callable, the dataset id as a value. |
+| `license_of(root)` | `[project].license` from `pyproject.toml`; empty means no crate is maintained. Presence is publication intent. |
+| `CRATE_FILENAME` | `ro-crate-metadata.json`. |
+
+## What must stay true
+
+- **The clock never enters the render.** `datePublished` is the newest
+  manifest `finished_at` (the spec file's last-commit date for a
+  never-materialized project) and must override rocrate's
+  construction-time default. Entities build in sorted order,
+  serialization is `sort_keys` — render-twice-identical is the one
+  byte-level claim, and it is what makes convergence sound. The
+  serialization also compacts every one-element array to its value,
+  as RO-Crate 1.1 recommends: which properties hold one value depends
+  on the project, so the rule lives in one place, not in each builder.
+- **Maintenance is derived, never configured.** RO-Crate requires a
+  license; materialize must not refuse to run science over a missing
+  key, and inventing one asserts terms over someone's data. Absent ⇒
+  one report line; removed later ⇒ the file is left, and the line says
+  it is no longer maintained.
+- **Run identity comes free from `git_sha`** — the driver reads HEAD
+  once per run, so grouping manifests by it *is* grouping by run: one
+  `OrganizeAction` per materialize, a `ControlAction` per execution, a
+  `HowToStep` per output id (deduped across universes — a step is spec
+  structure, an action is one execution).
+- **The `Person` is the author of the output's *saving* commit** (via
+  `writer`), never the manifest's `git_sha` — that is the commit the
+  run *started* at and can be someone else's.
+- **An output is a `File`, not a `Dataset` of parts.** It is one file,
+  so there is one annex key to look up and one `sha256` to publish —
+  the same number its manifest records as `data_version`, and the one
+  `sha256sum` prints.
+- **The manifest is not transliterated.** `env_version`,
+  `definition_version` and `hermeticity` get no invented schema.org
+  spelling — the manifest itself is in the crate as a `File`,
+  `subjectOf` its output. Real vocabulary comes from the workflow-run
+  `@context`, without which `containerImage` and `sha256` are
+  undefined terms JSON-LD silently drops — the pre-rebuild exporter's
+  failure mode.
+- **The rerun entry point does not regenerate the crate** — it is one
+  task's executor, so the crate lags until the next materialize.
+  Recorded residue, not a bug.
+
+## Tests
+
+`tests/test_crate.py` — pure: fixture manifests, a hand-built graph, a
+stub writer, no git anywhere; structure and ordering assertions plus
+the single render-twice byte check. `tests/test_crate_smoke.py` — the
+official `rocrate-validator` against Provenance Run Crate 0.5:
+REQUIRED clean, RECOMMENDED pinned to the recorded `_FLOOR` set (a new
+failure is a regression, a disappearing one is the floor to shrink),
+required in CI via `LC_CRATE_TESTS_REQUIRED=1`.

docs/api/dataset.md

@@ -0,0 +1,69 @@
+# lightcone.engine.dataset
+
+The git + git-annex seam: how a project stores what it produced.
+Storage follows the DataLad model — git carries the pointers and the
+history, git-annex carries the bytes — reached through ordinary `git`
+commands. Every command goes through `project._run`, so there is one
+monkeypatch point and every invocation is inspectable.
+
+Source: `src/lightcone/engine/dataset.py` (+
+`templates/files/gitattributes.tmpl` for the routing policy).
+
+## Key symbols
+
+| Symbol | Role |
+|---|---|
+| `save(root, paths, message)` | Stage scoped, commit — with `-c annex.thin=true` and `-c annex.dotfiles=true`, per-add and never written to config. |
+| `restore(root, paths)` | `git clean` always; `git checkout HEAD --` only when HEAD has the path. Never `-- .`. |
+| `status(root)` | The dirty question, scoped to the project (`-- .`, prefix-stripped) so a project inside a larger repository works. |
+| `head(root)` | The commit a run started at — read once per run, by the driver. |
+| `last_writer(root, *paths)` | Who last touched an output or its manifest — the foreign-write question. Answers "cannot say" as empty, never an error. |
+| `require_committer(root)` | Refuses a repository with no git identity, before any recipe spends time. Asked as `git var`, the question a commit itself asks. |
+| `dataset_id(root)` | The DataLad dataset UUID, read via `git config -f`. |
+| `set_annex_filter_required(root)` | Set `filter.annex.required=true`, so a `git add` that cannot reach git-annex fails loudly instead of staging raw bytes. |
+| `annex_filter_required(root)` | Whether that flag is already set — `lc init --check`'s question. |
+
+## What must stay true
+
+- **Nobody is ever asked to run a git-annex command.** `filter=annex`
+  plus the `.gitattributes` policy make an ordinary `git add` do the
+  right thing; `annex.largefiles=nothing` comes first and outputs and
+  data opt out — last match wins, and
+  `test_analysis_code_stays_in_git_and_stays_writable` pins it against
+  a real annex.
+- **Manifests stay in git**, exempted back out of the annex, so a
+  bytes-free clone can classify a whole project.
+- **An unfetched file exists, in two shapes** — an unlocked pointer
+  file (readable, hashes to the wrong thing) and a locked dangling
+  symlink (drops out of naive walks silently). `assets.data_version`
+  refuses both with `ContentNotFetchedError`; detection handles both
+  regardless of which shape lc writes, because `annex.thin` and
+  `git annex lock` are the researcher's to set.
+- **Thin is per-add and only where lc writes.** Thin's hazard is an
+  in-place write rewriting the annex object under its own key; lc
+  always resets output directories rather than writing in place, but
+  `data/` is the researcher's, and their tools (`h5py`, astropy
+  `mode='update'`) do open files for update — so the flag never
+  reaches repository config.
+- **`restore` is asymmetric on purpose:** a first materialization has
+  no HEAD version to go back to, and a failed task must not discard
+  edits made elsewhere while the graph ran.
+- **Committing an archive or dot-named file needs `annex.dotfiles`** —
+  git-annex routes dotfiles to git whatever `largefiles` says, and
+  without the flag an image archive lands as a git blob, silently.
+- **`filter.annex.required=true` is the storage policy's safety net.**
+  Without it, a `git add` whose shell cannot resolve git-annex prints
+  an error, exits 0, and stages the raw bytes into git history —
+  measured, and pinned by
+  `test_stock_plumbing_without_required_stages_raw_bytes_silently`.
+  It is the *only* thing convergence adds to what `git annex init`
+  wrote: no filter driver is rewritten, and no hook is touched, so how
+  git dispatches git-annex stays git-annex's own business and stays
+  resolved from `PATH`.
+
+## Tests
+
+`tests/test_dataset.py`, deliberately against **real tools**
+(`real_tools` fixture): whether bytes land in the annex or as a blob
+in git is not a question a stub can answer, and every bug this seam
+has had was invisible to one.

docs/api/identity.md

@@ -0,0 +1,53 @@
+# lightcone.engine.identity
+
+What a materialized output is identified by: two hashes that answer
+different questions, and the lock scan that decides whether an
+environment can be audited at all.
+
+Source: `src/lightcone/engine/identity.py`.
+
+## Key symbols
+
+| Symbol | Role |
+|---|---|
+| `definition_version(recipe, decisions)` | What the spec says an output *is* — the rebuild trigger. |
+| `env_version(root)` | What it ran under: lock bytes ‖ interpreter pin ‖ install settings ‖ image document. The `behind` trigger. |
+| `scan_lock(root)` | Refusals, reports and advisories about what the lock pins. |
+
+## What must stay true
+
+- **`env_version` is not part of `definition_version`.** That is the
+  whole shape of the invalidation model: an environment edit stales
+  nothing, it makes outputs *behind*. (The original design nested
+  them; staling every output in every project on an engine upgrade was
+  the bug, not the cost.)
+- **Both hashes are length-framed** — label, length, bytes per field —
+  so a boundary shift between adjacent fields cannot yield the same
+  digest from different inputs. Mutation-checked in the suite.
+- **The lock is hashed as raw bytes, never parsed.** A comment reflow
+  moves `env_version`, deliberately: over-invalidation costs a report
+  line, while a parse of our own can silently disagree with uv.
+- **The install-settings list is closed** (`_INSTALL_SETTINGS`), every
+  key hashed whether or not the project sets it — a setting outside
+  the list must not move the hash, one merely *matching* today's
+  default must. Settings are read where uv reads them (`uv.toml`
+  **replaces** `[tool.uv]`, measured); only values are hashed, never
+  which file supplied them. User-level uv config is deliberately out
+  of reach — machine state, not project state — and the residue is
+  tracked as issue #176.
+- **The git commit is recorded, never hashed, and never a signal** —
+  one sha covers the whole tree, so hashing it stales everything on a
+  README edit. The honest consequence: editing `src/fit.py` remakes
+  nothing unless the file is declared as an ASTRA input. Do not add a
+  heuristic that scans recipes for repo paths.
+- **The lock scan refuses only what cannot be audited** — path,
+  directory, and editable dependencies (two syncs of one lock can
+  install different code). A registry package with no wheel is a
+  report; a non-default group is advisory; the project's own package
+  is exempt. Names compare in PEP 503 form, or a project named
+  `my_project` fails to recognise itself.
+
+## Tests
+
+`tests/test_identity.py` — pure, and written as sensitivity tests in
+both directions: what must move each hash, and what must not.