diff --git a/README.md b/README.md index e7c028f..10b7a80 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,7 @@ IgnitionRL ## Headless Demo To author an environment from a blank TypeScript project, follow the first guide in [`docs/BUILD_YOUR_FIRST_ENVIRONMENT.md`](docs/BUILD_YOUR_FIRST_ENVIRONMENT.md). +To turn a stored learner checkpoint into an inference run and replay, follow [`docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md`](docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md). After cloning and installing dependencies, generate a local project with traces, metrics and JSON exports: diff --git a/docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md b/docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md new file mode 100644 index 0000000..e958953 --- /dev/null +++ b/docs/EXPORT_AND_REPLAY_TRAINED_POLICY.md @@ -0,0 +1,229 @@ +# Export and Replay a Trained Policy + +This guide shows how a learner checkpoint becomes a new inference run and a +replayable episode trace. It uses the current CLI and demo project format, so +each command can be run from the repository root. + +The shortest loop is: + +1. create or train a project; +2. list the checkpoint; +3. inspect the checkpoint detail; +4. run inference from that checkpoint; +5. replay the inference run. + +## 1. Create a Demo Project With a Checkpoint + +Use `target-2d` for a compact discrete example: + +```sh +bun run --cwd packages/cli start demo target-2d ./target-2d-demo.ignitionrl \ + --seed 42 \ + --random-episodes 3 \ + --learner-episodes 25 \ + --max-steps 120 \ + --json +``` + +This creates two runs: + +- `target-2d-random`; +- `target-2d-tabular-q`. + +The learner run writes its checkpoint to: + +```txt +runs/target-2d-tabular-q/checkpoints/final.json +``` + +You can use the same flow with other current demos: + +| Demo | Source run | Checkpoint | Inference support | +| --- | --- | --- | --- | +| `grid-world` | `grid-world-tabular-q` | `final` | `tabular-q-learning` | +| `target-2d` | `target-2d-tabular-q` | `final` | `tabular-q-learning` | +| `drone-target` | `drone-target-linear-policy-search` | `final` | `linear-policy-search` | + +`drone-target` also creates `drone-target-linear-policy-search-inference` +during the demo, but you can still rerun `infer` manually with a new run id. + +## 2. List Checkpoints + +List checkpoints for the trained run: + +```sh +bun run --cwd packages/cli start checkpoints ./target-2d-demo.ignitionrl \ + --run-id target-2d-tabular-q \ + --json +``` + +The important fields are: + +- `id`: checkpoint id, usually `final`; +- `runId`: source training run; +- `envId`: environment the checkpoint was trained against; +- `algorithm`: learner algorithm that wrote the payload; +- `path`: project-relative JSON file location; +- `metadata.kind`: learner family metadata for UI and compatibility checks. + +## 3. Inspect Checkpoint Detail + +Build a Studio-ready checkpoint detail view: + +```sh +bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \ + target-2d-tabular-q \ + final \ + --export \ + --json +``` + +This returns: + +- the selected checkpoint; +- the source run summary; +- source metric names; +- linked inference runs, if any already exist; +- an optional `studio-checkpoint-view` export. + +Use `--include-payload` only when you need to inspect the raw learner payload: + +```sh +bun run --cwd packages/cli start checkpoint ./target-2d-demo.ignitionrl \ + target-2d-tabular-q \ + final \ + --include-payload \ + --json +``` + +Payloads can be large. A tabular learner checkpoint includes the Q-table; neural +adapter checkpoints include model metadata and serialized weights. + +## 4. Run the Checkpoint in Inference Mode + +Create a new run from the stored checkpoint: + +```sh +bun run --cwd packages/cli start infer ./target-2d-demo.ignitionrl \ + target-2d-tabular-q \ + --checkpoint-id final \ + --episodes 3 \ + --max-steps 120 \ + --run-id target-2d-tabular-q-inference \ + --seed 42:inference \ + --json +``` + +`infer` loads the checkpoint, creates an exploitation-only policy for supported +learners and writes a normal persisted run. The new run has: + +- a run manifest under `runs//run.json`; +- episode traces under `runs//traces/`; +- metric records under `runs//metrics.jsonl`; +- exported run, replay and reward-debugger artifacts under `exports/`. + +The inference run is intentionally separate from the training run. That keeps +training metrics, inference metrics and replay traces comparable without +overwriting the source checkpoint. + +## 5. Replay the Inference Run + +Read the persisted trace: + +```sh +bun run --cwd packages/cli start replay ./target-2d-demo.ignitionrl \ + target-2d-tabular-q-inference \ + --frame 0 \ + --json +``` + +The replay payload includes: + +- `totalFrames`; +- episode summary fields such as `totalReward`, `success`, `terminated` and `truncated`; +- named reward-term totals; +- observation dimension ranges; +- action distribution; +- the selected frame with observation, action, reward terms and cumulative reward. + +If the policy is bad, replay should make that obvious. For example, an +inference run that repeats one action will show a one-entry action distribution +and reward terms dominated by negative progress or step penalties. + +## 6. Refresh Studio Artifacts + +After inference, refresh the workspace export: + +```sh +bun run --cwd packages/cli start studio ./target-2d-demo.ignitionrl \ + --run-id target-2d-tabular-q-inference \ + --score-by summary.totalReward \ + --export \ + --json +``` + +Refresh the selected run export: + +```sh +bun run --cwd packages/cli start run ./target-2d-demo.ignitionrl \ + target-2d-tabular-q-inference \ + --export \ + --json +``` + +These write JSON artifacts that the current Studio shell can load offline: + +- `exports/studio-workspace-view.json`; +- `exports/runs//studio-run-view.json`; +- `exports/runs//replays/.json`; +- `exports/runs//reward-debuggers/.json`. + +## Compatibility Rules + +Checkpoint inference is strict. The CLI chooses the loader from the source run, +not from a user-supplied algorithm flag. + +The checkpoint must match: + +- source run `envId`; +- source run `algorithm`; +- checkpoint `envId`; +- checkpoint `algorithm`; +- observation shape expected by the environment; +- action space expected by the learner. + +Current supported inference sources are: + +| Environment | Source algorithm | Loader | +| --- | --- | --- | +| `GridWorld-v0` | `tabular-q-learning` | `createTabularQLearner()` | +| `GridWorld-v0` | `dqn` | `createDqnLearner()` | +| `Target2D-v0` | `tabular-q-learning` | `createTabularQLearner()` | +| `Target2D-v0` | `dqn` | `createDqnLearner()` | +| `DroneTarget-v0` | `linear-policy-search` | `createLinearPolicySearchLearner()` | + +Unsupported combinations fail before creating a new run. + +## Common Failure Modes + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `Unsupported checkpoint inference source: with .` | The CLI does not have a loader for that environment/algorithm pair yet. | Use one of the supported pairs or add an inference loader for the new learner. | +| Checkpoint command cannot find `final`. | The source run did not write that checkpoint id, or the wrong run id was used. | Run `checkpoints --run-id --json` and use the returned `id`. | +| Inference creates a run but the replay performs poorly. | The checkpoint loaded correctly, but the policy did not generalize or the seed exposes a bad scenario. | Compare training and inference metrics, then inspect replay reward terms and action distribution. | +| `replay` says the run has no replay trace. | The run has no persisted traces or the wrong run id was passed. | Use `episodes --run-id --json` to find traceable runs. | +| Checkpoint detail output is huge. | `--include-payload` prints raw checkpoint data such as Q-tables or weights. | Omit `--include-payload` unless debugging serialization. | + +## Future Studio Flow + +The future desktop UI should map to the same artifacts: + +1. The Checkpoints panel lists `checkpoints` output for the selected run. +2. Selecting a checkpoint opens the `studio-checkpoint-view` payload. +3. Clicking "Run inference" calls the same SDK path as `infer`. +4. The new inference run appears in history as a normal run linked to its source checkpoint. +5. The Replay panel reads the exported replay payload. +6. The Reward panel reads the exported reward-debugger payload. + +The CLI flow is therefore not a throwaway shortcut. It is the headless version +of the Studio checkpoint and replay workflow.