docs: define reproducible baseline-versus-playbook experiment

aicp/experiment.py

@@ -0,0 +1,32 @@
+"""Validate experimental benchmark manifests without running agents."""
+import json
+from pathlib import Path
+
+CONDITIONS = ("baseline", "playbook")
+REQUIRED = {"schema_version", "experiment_id", "agent", "fixture_revision", "tasks", "repetitions", "timeout_seconds"}
+
+
+def validate_manifest(manifest):
+    if not isinstance(manifest, dict) or set(manifest) != REQUIRED or manifest["schema_version"] != 1:
+        raise ValueError("Invalid experimental manifest fields")
+    for key in ("experiment_id", "agent", "fixture_revision"):
+        if not isinstance(manifest[key], str) or not manifest[key].strip():
+            raise ValueError("Missing experiment metadata: " + key)
+    if type(manifest["repetitions"]) is not int or not 1 <= manifest["repetitions"] <= 100:
+        raise ValueError("Repetitions must be between 1 and 100")
+    if type(manifest["timeout_seconds"]) is not int or not 1 <= manifest["timeout_seconds"] <= 3600:
+        raise ValueError("Timeout must be between 1 and 3600 seconds")
+    tasks = manifest["tasks"]
+    if not isinstance(tasks, list) or not tasks or len(tasks) > 100 or any(
+        not isinstance(t, str) or not t.strip() for t in tasks
+    ) or len(set(tasks)) != len(tasks):
+        raise ValueError("Tasks must be a nonempty unique list")
+    return manifest
+
+
+def load_manifest(path):
+    if not isinstance(path, Path):
+        path = Path(path)
+    if path.is_symlink() or not path.is_file():
+        raise ValueError("Manifest must be a regular file")
+    return validate_manifest(json.loads(path.read_text(encoding="utf-8")))

docs/EXPERIMENTAL_PROTOCOL.md

@@ -0,0 +1,40 @@
+# Baseline versus playbook experimental protocol
+
+**Status:** preregistered proposal for a personal experimental side project. No productivity claims or measured outcomes are asserted.
+
+## Research question
+
+For the same agent and realistic, synthetic software-engineering tasks, does the AI Coding Playbook change correctness, review effort, completion time, or scope compliance relative to ordinary agent use?
+
+## Conditions
+
+- **Baseline:** the selected agent receives a task objective, repository, and identical acceptance criteria, without playbook-specific contracts or workflows.
+- **Playbook:** the same agent receives the same task and acceptance criteria using AICP's task contract, instructions, and verification workflow.
+- **Evaluation:** an independent harness uses held-out checks that the agent cannot read or modify during execution. The same checks score both conditions.
+- **No cherry-picking:** report all runs, including launch failures, timeouts, and incomplete work.
+
+## Controls and reproducibility
+
+1. Freeze task fixture revision, agent version, model identifier, configuration, prompt budget, tool permissions, timeout, and compute resources.
+2. Start every run from a fresh, clean checkout of the same fixture commit. Use separate disposable environments per run; never reuse agent-modified state.
+3. Randomize or counterbalance baseline/playbook order. Run at least three independent repetitions per condition for the pilot; label small-sample results exploratory.
+4. Capture exact commands, hashes of task/fixture and checks, environment metadata, wall time, agent exit, human interventions, and costs **only when observed**.
+5. Keep evaluator tests inaccessible to the agent until execution ends. Audit modified files and score from a fresh verification environment.
+6. Define task completion and review criteria before examining results. Do not infer productivity improvements from lines of code or agent-reported success.
+7. Store only synthetic content. Never publish secrets, private repositories, credentials, or proprietary engineering material.
+
+## Primary and secondary outcomes
+
+**Primary:** fraction of attempts passing all independent acceptance tests, with timeouts/failures counted as unsuccessful.
+
+**Secondary:** elapsed wall-clock time; number of human interventions; human review minutes; regressions; out-of-scope file changes; cost or token consumption when reliably available. Missing telemetry is `null`, not zero.
+
+Report task-level results and paired differences. Include distributions, confidence intervals when sample sizes permit, raw run counts, and limitations. Do not aggregate different agent models into a single vendor ranking.
+
+## Pilot and expansion
+
+Pilot with five small, deterministic synthetic tasks: bug fix, test addition, refactoring, documentation, and feature implementation. Expand to 20–30 tasks only after fixture and scoring validity are demonstrated. Publish negative and inconclusive results as well as positive results.
+
+## Threats to validity
+
+Agent nondeterminism, prompt leakage, model updates, order effects, task selection, imperfect cost reporting, unblinded reviewers, and unequal context budgets can bias outcomes. AICP's current `aicp run` host checks and Git audit are **not** a substitute for an independent held-out evaluator. Live agent integrations remain opt-in and are not run by CI.

docs/README.md

@@ -16,6 +16,10 @@ Start with [Getting started](GETTING_STARTED.md) for quick setup, the [practical
 - [Local verification](VERIFICATION.md)
 - [Platform compatibility and agent integration](COMPATIBILITY.md)
 
+## Research and evaluation
+
+- [Baseline vs. playbook experimental protocol](EXPERIMENTAL_PROTOCOL.md)
+
 ## Engineering guides
 
 - [Prompt catalog](../prompts/README.md)

evaluations/experiments/pilot.json

@@ -0,0 +1,15 @@
+{
+  "schema_version": 1,
+  "experiment_id": "synthetic-pilot-v1",
+  "agent": "record-exact-agent-version-and-model-before-running",
+  "fixture_revision": "replace-with-frozen-fixture-commit-sha",
+  "tasks": [
+    "bug-fix",
+    "test-addition",
+    "refactoring",
+    "documentation",
+    "feature"
+  ],
+  "repetitions": 3,
+  "timeout_seconds": 900
+}

tests/test_experiment.py

@@ -0,0 +1,38 @@
+"""Experimental manifest validation tests."""
+import unittest
+from aicp.experiment import validate_manifest
+
+
+class ExperimentManifestTests(unittest.TestCase):
+    def setUp(self):
+        self.manifest = {
+            "schema_version": 1,
+            "experiment_id": "pilot",
+            "agent": "codex-version",
+            "fixture_revision": "abc123",
+            "tasks": ["bug-fix", "test-addition"],
+            "repetitions": 3,
+            "timeout_seconds": 900,
+        }
+
+    def test_valid(self):
+        self.assertEqual(validate_manifest(self.manifest), self.manifest)
+
+    def test_duplicate_tasks(self):
+        self.manifest["tasks"].append("bug-fix")
+        with self.assertRaises(ValueError):
+            validate_manifest(self.manifest)
+
+    def test_invalid_repetitions(self):
+        self.manifest["repetitions"] = True
+        with self.assertRaises(ValueError):
+            validate_manifest(self.manifest)
+
+    def test_unexpected_fields(self):
+        self.manifest["result"] = "pass"
+        with self.assertRaises(ValueError):
+            validate_manifest(self.manifest)
+
+
+if __name__ == "__main__":
+    unittest.main()