[codex] add checkpoint replay guide

README.md

@@ -92,6 +92,7 @@ IgnitionRL
 ## Headless Demo
 
 To author an environment from a blank TypeScript project, follow the first guide in [`docs/BUILD_YOUR_FIRST_ENVIRONMENT.md`](docs/BUILD_YOUR_FIRST_ENVIRONMENT.md).
+To turn a stored learner checkpoint into an inference run and replay, follow [`docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md`](docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md).
 
 After cloning and installing dependencies, generate a local project with traces, metrics and JSON exports:
 

docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md

@@ -0,0 +1,229 @@
+# Export and Replay a Trained Policy
+
+This guide shows how a learner checkpoint becomes a new inference run and a
+replayable episode trace. It uses the current CLI and demo project format, so
+each command can be run from the repository root.
+
+The shortest loop is:
+
+1. create or train a project;
+2. list the checkpoint;
+3. inspect the checkpoint detail;
+4. run inference from that checkpoint;
+5. replay the inference run.
+
+## 1. Create a Demo Project With a Checkpoint
+
+Use `target-2d` for a compact discrete example:
+
+```sh
+bun run --cwd packages/cli start demo target-2d ./target-2d-demo.ignitionrl \
+  --seed 42 \
+  --random-episodes 3 \
+  --learner-episodes 25 \
+  --max-steps 120 \
+  --json
+```
+
+This creates two runs:
+
+- `target-2d-random`;
+- `target-2d-tabular-q`.
+
+The learner run writes its checkpoint to:
+
+```txt
+runs/target-2d-tabular-q/checkpoints/final.json
+```
+
+You can use the same flow with other current demos:
+
+| Demo | Source run | Checkpoint | Inference support |
+| --- | --- | --- | --- |
+| `grid-world` | `grid-world-tabular-q` | `final` | `tabular-q-learning` |
+| `target-2d` | `target-2d-tabular-q` | `final` | `tabular-q-learning` |
+| `drone-target` | `drone-target-linear-policy-search` | `final` | `linear-policy-search` |
+
+`drone-target` also creates `drone-target-linear-policy-search-inference`
+during the demo, but you can still rerun `infer` manually with a new run id.
+
+## 2. List Checkpoints
+
+List checkpoints for the trained run:
+
+```sh
+bun run --cwd packages/cli start checkpoints ./target-2d-demo.ignitionrl \
+  --run-id target-2d-tabular-q \
+  --json
+```
+
+The important fields are:
+
+- `id`: checkpoint id, usually `final`;
+- `runId`: source training run;
+- `envId`: environment the checkpoint was trained against;
+- `algorithm`: learner algorithm that wrote the payload;
+- `path`: project-relative JSON file location;
+- `metadata.kind`: learner family metadata for UI and compatibility checks.
+
+## 3. Inspect Checkpoint Detail
+
+Build a Studio-ready checkpoint detail view:
+
+```sh
+bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \
+  target-2d-tabular-q \
+  final \
+  --export \
+  --json
+```
+
+This returns:
+
+- the selected checkpoint;
+- the source run summary;
+- source metric names;
+- linked inference runs, if any already exist;
+- an optional `studio-checkpoint-view` export.
+
+Use `--include-payload` only when you need to inspect the raw learner payload:
+
+```sh
+bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \
+  target-2d-tabular-q \
+  final \
+  --include-payload \
+  --json
+```
+
+Payloads can be large. A tabular learner checkpoint includes the Q-table; neural
+adapter checkpoints include model metadata and serialized weights.
+
+## 4. Run the Checkpoint in Inference Mode
+
+Create a new run from the stored checkpoint:
+
+```sh
+bun run --cwd packages/cli start infer ./target-2d-demo.ignitionrl \
+  target-2d-tabular-q \
+  --checkpoint-id final \
+  --episodes 3 \
+  --max-steps 120 \
+  --run-id target-2d-tabular-q-inference \
+  --seed 42:inference \
+  --json
+```
+
+`infer` loads the checkpoint, creates an exploitation-only policy for supported
+learners and writes a normal persisted run. The new run has:
+
+- a run manifest under `runs/<runId>/run.json`;
+- episode traces under `runs/<runId>/traces/`;
+- metric records under `runs/<runId>/metrics.jsonl`;
+- exported run, replay and reward-debugger artifacts under `exports/`.
+
+The inference run is intentionally separate from the training run. That keeps
+training metrics, inference metrics and replay traces comparable without
+overwriting the source checkpoint.
+
+## 5. Replay the Inference Run
+
+Read the persisted trace:
+
+```sh
+bun run --cwd packages/cli start replay ./target-2d-demo.ignitionrl \
+  target-2d-tabular-q-inference \
+  --frame 0 \
+  --json
+```
+
+The replay payload includes:
+
+- `totalFrames`;
+- episode summary fields such as `totalReward`, `success`, `terminated` and `truncated`;
+- named reward-term totals;
+- observation dimension ranges;
+- action distribution;
+- the selected frame with observation, action, reward terms and cumulative reward.
+
+If the policy is bad, replay should make that obvious. For example, an
+inference run that repeats one action will show a one-entry action distribution
+and reward terms dominated by negative progress or step penalties.
+
+## 6. Refresh Studio Artifacts
+
+After inference, refresh the workspace export:
+
+```sh
+bun run --cwd packages/cli start studio ./target-2d-demo.ignitionrl \
+  --run-id target-2d-tabular-q-inference \
+  --score-by summary.totalReward \
+  --export \
+  --json
+```
+
+Refresh the selected run export:
+
+```sh
+bun run --cwd packages/cli start run ./target-2d-demo.ignitionrl \
+  target-2d-tabular-q-inference \
+  --export \
+  --json
+```
+
+These write JSON artifacts that the current Studio shell can load offline:
+
+- `exports/studio-workspace-view.json`;
+- `exports/runs/<runId>/studio-run-view.json`;
+- `exports/runs/<runId>/replays/<episodeId>.json`;
+- `exports/runs/<runId>/reward-debuggers/<episodeId>.json`.
+
+## Compatibility Rules
+
+Checkpoint inference is strict. The CLI chooses the loader from the source run,
+not from a user-supplied algorithm flag.
+
+The checkpoint must match:
+
+- source run `envId`;
+- source run `algorithm`;
+- checkpoint `envId`;
+- checkpoint `algorithm`;
+- observation shape expected by the environment;
+- action space expected by the learner.
+
+Current supported inference sources are:
+
+| Environment | Source algorithm | Loader |
+| --- | --- | --- |
+| `GridWorld-v0` | `tabular-q-learning` | `createTabularQLearner()` |
+| `GridWorld-v0` | `dqn` | `createDqnLearner()` |
+| `Target2D-v0` | `tabular-q-learning` | `createTabularQLearner()` |
+| `Target2D-v0` | `dqn` | `createDqnLearner()` |
+| `DroneTarget-v0` | `linear-policy-search` | `createLinearPolicySearchLearner()` |
+
+Unsupported combinations fail before creating a new run.
+
+## Common Failure Modes
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| `Unsupported checkpoint inference source: <env> with <algorithm>.` | The CLI does not have a loader for that environment/algorithm pair yet. | Use one of the supported pairs or add an inference loader for the new learner. |
+| Checkpoint command cannot find `final`. | The source run did not write that checkpoint id, or the wrong run id was used. | Run `checkpoints <projectDir> --run-id <sourceRunId> --json` and use the returned `id`. |
+| Inference creates a run but the replay performs poorly. | The checkpoint loaded correctly, but the policy did not generalize or the seed exposes a bad scenario. | Compare training and inference metrics, then inspect replay reward terms and action distribution. |
+| `replay` says the run has no replay trace. | The run has no persisted traces or the wrong run id was passed. | Use `episodes <projectDir> --run-id <runId> --json` to find traceable runs. |
+| Checkpoint detail output is huge. | `--include-payload` prints raw checkpoint data such as Q-tables or weights. | Omit `--include-payload` unless debugging serialization. |
+
+## Future Studio Flow
+
+The future desktop UI should map to the same artifacts:
+
+1. The Checkpoints panel lists `checkpoints` output for the selected run.
+2. Selecting a checkpoint opens the `studio-checkpoint-view` payload.
+3. Clicking "Run inference" calls the same SDK path as `infer`.
+4. The new inference run appears in history as a normal run linked to its source checkpoint.
+5. The Replay panel reads the exported replay payload.
+6. The Reward panel reads the exported reward-debugger payload.
+
+The CLI flow is therefore not a throwaway shortcut. It is the headless version
+of the Studio checkpoint and replay workflow.