Feature: eval harness by sanjayankur31 · Pull Request #220 · NeuroML/klea

sanjayankur31 · 2026-10-28T14:43:34Z

includes the ADRs that document the design of the evaluation harness. (ADRs 0056-0060)

…g, invariants)

ADR-0056 (Test Case Format): update Date from 2025-01-01 to 2026-10-27 ADR-0057 (Run Manager & Isolation): update Date from 2025-01-01 to 2026-10-27 AGENTS.md: add guidance that ADRs must use today date in YYYY-MM-DD format This ensures ADRs use current dates and prevents future use of placeholder dates.

…S.md - ADR files use YYYY-MM-DD format for the Date field in MADR header - Session log files use YYYY-MM-DD-HHMM format where HHMM is 24-hour time - Separate files per session avoid merge conflicts when multiple sessions run the same day - Updated documentation to explicitly state these conventions for clarity

Convert the metrics collection ADR to the devdocs/adr-template.md structure: - add YAML frontmatter with status, date and decision-makers - rename Problem to Context and Problem Statement - add Decision Drivers and Considered Options sections - rename Decision to Decision Outcome with explicit chosen option - demote Consequences under the Decision Outcome and rephrase as Good/Bad bullets - convert the comparison tables into Pros and Cons of the Options with one heading per option - fold Rationale, Implementation Notes, Examples, Related Decisions and References under a final More Information section

Convert the run manager ADR to the devdocs/adr-template.md structure: - add YAML frontmatter with status, date and decision-makers - rename Problem to Context and Problem Statement - add Decision Drivers and Considered Options sections - rename Decision to Decision Outcome with explicit chosen option - demote Consequences under the Decision Outcome and rephrase as Good/Bad bullets - convert the comparison tables into Pros and Cons of the Options with one heading per option - fold Rationale, Implementation Notes, Examples, Related Decisions and References under a final More Information section

Convert the test case format ADR to the devdocs/adr-template.md structure: - add YAML frontmatter with status, date and decision-makers - rename Problem to Context and Problem Statement - add Decision Drivers and Considered Options sections - rename Decision to Decision Outcome with explicit Chosen option - demote Consequences under the Decision Outcome and rephrase as Good/Bad bullets - convert the alternatives comparison table into Pros and Cons of the Options with one heading per option - fold Rationale, Implementation Notes, Related Decisions and References under a final More Information section

Update ADR-0056 and ADR-0057 references to the current ADR titles (ADR-0057 Run Manager & Isolation, ADR-0058 Metrics Collection & Aggregation) and drop the duplicated metrics schema in ADR-0057 in favour of ADR-0058.

Specify a filesystem directory-per-run result layout (run.json, metrics.jsonl, transcript.jsonl, artifacts/, COMPLETE marker) with an append-only runs/index.jsonl index, atomic rename-on-completion writes, schema-version stamping per ADR-0058, and configurable artifact retention.

Move ADR-0056 (test case format), ADR-0057 (run manager & isolation), ADR-0058 (metrics collection & aggregation), ADR-0059 (result storage) and ADR-0060 (harness execution) from proposed to accepted. Also revise the ADR-0058 primary vs. secondary metrics hierarchy: task_success, pass@k and turn_count are primary evaluative metrics, while token_usage, latency and cost_estimate are secondary/optional; latency is explicitly not an evaluation signal.

sanjayankur31 · 2026-10-28T14:44:58Z

@khanak0509 : this is what I've come up with for the evaluation harness---what do you think? im slowly beginning to work on the code now---starting with the package scaffold + jsonl bits.

sanjayankur31 · 2026-10-28T14:45:20Z

(will merge this in a couple of days---just up here for us to review once)

khanak0509 · 2026-10-28T16:20:42Z

Hii, looking solid. Will take a deeper look, just need a bit of time.

Add a streaming JSONL test-case loader (klea_eval.cases) that validates every case against the ADR-0056 schema and invariants, raising CaseValidationError (klea_eval.errors) with source path, line number and case_id. Cover the loader and every rejection rule with tests in eval_pkg/tests/test_cases.py, and align the ADR field names (output_format, rubric_type).

sanjayankur31 · 2026-11-10T21:48:25Z

@khanak0509 : i've resolved the issues you'd noted and implemented the harness now. Needs testing---could you take a peek to see if it looks OK in general please?

Copilot

🟡 Changes recommended

Unresolved critical defects allow tool execution outside isolated workspaces, deletion or orphaning of retained runs, and publication of incomplete or misleading evaluation results.

11 open issues

What changed in this PR

Adds a dedicated klea-eval package for running reproducible evaluations against Klea's HTTP streaming API, with run storage, scoring, inspection commands and supporting design documentation.

Changes:

Introduces deterministic run identities, workspace publication, HTTP/SSE execution, aggregation and sequential scheduling.
Adds offline tests, sample suites, packaging and CI integration.
Documents execution and storage contracts, model selection and operational workflows.

File	Description
`utils_pkg/tests/conftest.py`	Isolates NiceGUI test storage per process.
`ty.toml`	Adds evaluation package import resolution.
`scripts/render_mermaid.py`	Modernizes annotations and marks Typer declarations.
`requirements-dev.txt`	Includes evaluation development dependencies.
`rag_pkg/tests/conftest.py`	Isolates NiceGUI test storage per process.
`eval_pkg/tests/test_workspace.py`	Tests workspace lifecycle and publication.
`eval_pkg/tests/test_smoke.py`	Checks package import, version and CLI startup.
`eval_pkg/tests/test_scoring.py`	Tests stored-run scoring and aggregation.
`eval_pkg/tests/test_scheduler.py`	Tests scheduling, isolation and exit codes.
`eval_pkg/tests/test_run_config.py`	Tests model overrides and credential requests.
`eval_pkg/tests/test_identity.py`	Pins deterministic run identity behavior.
`eval_pkg/tests/test_http_runner.py`	Tests streaming execution and error handling.
`eval_pkg/tests/test_cli_validate.py`	Tests suite-validation diagnostics.
`eval_pkg/tests/test_cli_run.py`	Tests CLI model-override parsing.
`eval_pkg/tests/fixtures/sse/truncated_turn.sse`	Supplies an interrupted-stream fixture.
`eval_pkg/tests/fixtures/sse/error_turn.sse`	Supplies a failed-turn fixture.
`eval_pkg/tests/fixtures/sse/complete_turn.sse`	Supplies a completed-turn fixture.
`eval_pkg/setup.cfg`	Defines distribution dependencies and CLI entry point.
`eval_pkg/samples/smoke.jsonl`	Adds a single-turn sample suite.
`eval_pkg/samples/multi_turn.jsonl`	Adds a conversational refinement sample.
`eval_pkg/samples/multi_case.jsonl`	Adds a multi-case sample suite.
`eval_pkg/Readme.md`	Introduces the harness and links its guide.
`eval_pkg/pyproject.toml`	Configures packaging and pytest markers.
`eval_pkg/klea_eval/workspace.py`	Manages run directories and atomic publication.
`eval_pkg/klea_eval/ui/cli.py`	Exposes execution, validation and inspection commands.
`eval_pkg/klea_eval/ui/__init__.py`	Introduces the UI subpackage.
`eval_pkg/klea_eval/scoring.py`	Computes run scores and suite aggregates.
`eval_pkg/klea_eval/scheduler.py`	Coordinates sequential unattended suite execution.
`eval_pkg/klea_eval/runners/sse.py`	Parses streamed event frames.
`eval_pkg/klea_eval/runners/http.py`	Executes turns and collects streamed results.
`eval_pkg/klea_eval/runners/__init__.py`	Exports runner interfaces and errors.
`eval_pkg/klea_eval/identity.py`	Derives deterministic UUID-based identities.
`eval_pkg/klea_eval/errors.py`	Defines location-aware validation errors.
`eval_pkg/klea_eval/api.py`	Provides stored-run inspection and retention.
`eval_pkg/klea_eval/__init__.py`	Defines package metadata and version.
`eval_pkg/AGENTS.md`	Adds package contributor guidance.
`devdocs/system/eval-harness-run-guide.md`	Documents setup, execution and troubleshooting.
`devdocs/README.md`	Indexes the guide and related decisions.
`devdocs/backlog.md`	Expands the evaluation work tracker.
`devdocs/adr/0060-harness-execution.md`	Defines orchestration and failure semantics.
`devdocs/adr/0059-result-storage.md`	Defines result layout and persistence contracts.
`CHANGELOG.md`	Records model selection and validation changes.
`AGENTS.md`	Clarifies documentation and session-log dates.
`.github/workflows/ci.yml`	Installs the harness and runs eval-marked tests.
`.agents/2026-10-28-1703.md`	Records loader work and follow-up tasks.

_{Review effort: Balanced}

+    per_case: dict[str, list[int]] = {}
+    for position, entry in enumerate(entries):
+        if _is_completed(entry):
+            per_case.setdefault(entry.case_id, []).append(position)
+
+    dropped: set[int] = set()
+    for positions in per_case.values():
+        dropped.update(positions[:-keep])


+    case = case_id.strip()
+    if not case:
+        raise ValueError("case_id must be a non-empty string")
+    return f"{case}:{_seed_text(harness_seed)}"


+        if kind == "complete":
+            terminal = True
+            text = _message_for_user(event)


+    payload: dict[str, Any] = {
+        "run_id": context.run_id,
+        "user_id": context.user_id,
+        "chat_id": context.chat_id,
+        "query": _turn_input(turn),


+    run_dir = workspace.runs_dir(root) / run_id
+    try:
+        summary = metrics.load_run(run_dir)
+    except metrics.MetricsError as exc:


+        passed = case_run.metrics is not None and case_run.metrics.task_success is True
+        typer.echo(f"{case_run.case_id}: {'pass' if passed else 'fail'}")


+    """Render a report from already collected evaluation results."""
+    # Deferred import: the reporting stack pulls in the renderers and is only
+    # needed once a command actually renders something.
+    from klea_eval.reporting import write_report  # ty: ignore[import-not-found]


khanak0509 · 2026-11-11T08:57:02Z

Sure!

…s list Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

khanak0509

The harness shape is really solid: module split, offline tests, careful storage with fsync / COMPLETE / rename, and i really liked the honesty in the docs :) really cool.

but i think a few more tweaks are needed before we lean on this for a real benchmark, because some bugs would quietly make the numbers look better than they are:

re-running with a different -m can skip every case and show the previous model’s results, since run_id only depends on case_id and seed
retries and overwrite keep the same chat_id, so the backend chat history isn’t fresh
run.json and the index get written after COMPLETE, so a crash in between leaves a run that gets skipped forever
overwrite deletes the old run before the new one has even succeeded
failed runs never get an index line, so suite scores can look too good or pick up an older success for that case
ungraded live runs print as fail even when exit is 0, and nothing grades expected_outputs, tools, or rubrics yet… so everything currently fails with style
truncated SSE complete with a bad payload still counts as success :(
tool and file cases aren’t isolated because client_context.cwd isn’t sent
CLI run defaults --workspace to "." so it bypasses KLEA_EVAL_WORKSPACE
report still imports a module that isn’t there

khanak0509 · 2026-11-11T20:55:45Z

and like for pass@k formula and --repeat, narrower retries, ADR wording, fixture path and input_dir, soft retention, LLM judge, schedule CLI, and a klea_utils version floor - we can open separate issues so it won’t get cluttered here and we can handle each separately.

sanjayankur31 · 2026-11-11T21:14:35Z

Thanks for the review. Let me go through each item and fix it 👍

…-eval-harness

Derive the run identity from the model overrides, resolved models and endpoint so the same case run under a different model set or endpoint is no longer skipped as a duplicate run. Thread the resolved models through the pipeline and run manager and persist them in the run metadata.

A case that ran but carried no grade kept the stored status 'completed', so it was counted as completed even though the CLI already labelled it 'ungraded'. Keep the stored status and the run.json stamp at 'completed' instead of rewriting CaseRun.status in __post_init__, and split the completed cases at read time into the 'completed' (graded) and 'ungraded' counters. The two counters are now exhaustive over STATUS_COMPLETED and mutually exclusive, and the exit code is unchanged: only a failed case or task_success is False yields exit 1. Drop the now-unused STATUS_UNGRADED constant.

…evdoc

…ion log

sanjayankur31 added 17 commits October 27, 2026 14:25

docs: update evaluation harness note

0cdfe3b

docs(adr): ADR-0056 test case storage format (JSONL schema, versionin…

087b804

…g, invariants)

chore: use ascii glyphs

d75c138

Merge branch 'development' into feat-eval-harness

e322779

Merge branch 'development' into feat-eval-harness

e61c0e4

docs(adr): correct stale ADR-0057/ADR-0058 cross-references

4173cb4

Update ADR-0056 and ADR-0057 references to the current ADR titles (ADR-0057 Run Manager & Isolation, ADR-0058 Metrics Collection & Aggregation) and drop the duplicated metrics schema in ADR-0057 in favour of ADR-0058.

Merge branch 'development' into feat-eval-harness

d3d66e4

docs(adr): add ADR-0060 harness execution

2eafc6b

Merge branch 'development' into feat-eval-harness

83a47a6

feat(eval-pkg): scaffold package

8aaf42d

sanjayankur31 added documentation Improvements or additions to documentation enhancement New feature or request labels Oct 28, 2026

sanjayankur31 added this to NeuroML backlog Oct 28, 2026

github-project-automation Bot moved this to 🆕 New in NeuroML backlog Oct 28, 2026

sanjayankur31 moved this from 🆕 New to 👀 In review in NeuroML backlog Oct 28, 2026

sanjayankur31 linked an issue Oct 28, 2026 that may be closed by this pull request

Design: Agent: evaluation harness #173

Open

sanjayankur31 added 2 commits October 28, 2026 15:24

docs(devdocs): index ADR-0051 through ADR-0060 in README

78208dc

docs: update backlog

e5ad893

sanjayankur31 added 2 commits October 28, 2026 16:45

chore(agents): update session log + backlog

7347279

sanjayankur31 changed the title ~~Feature: design of eval harness~~ Feature: eval harness Nov 10, 2026

sanjayankur31 requested a review from khanak0509 November 10, 2026 21:47

sanjayankur31 requested a balanced review from Copilot November 10, 2026 21:48

Copilot started reviewing on behalf of sanjayankur31 November 10, 2026 21:49

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

sanjayankur31 and others added 2 commits November 11, 2026 09:00

fix(eval-harness): fix warning message formatting and update survivor…

2a6a261

…s list Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix(eval-harness): add klea_utils to setup.cfg dependencies

cfe5dea

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

khanak0509 requested changes Nov 11, 2026

View reviewed changes

sanjayankur31 added 18 commits November 12, 2026 12:10

Merge branch 'development' into feat-eval-harness

9100c83

Merge branch 'feat-eval-harness' of github.com:NeuroML/klea into feat…

6bea333

…-eval-harness

Merge branch 'development' into feat-eval-harness

1185ee1

Merge branch 'development' into feat-eval-harness

eed8fa4

fix(eval): allocate a fresh chat id per attempt and overwrite

2a09913

fix(eval): write run.json before the COMPLETE marker

3f90e52

fix(eval): re-index completed runs missing an index entry

99f2670

Merge branch 'development' into feat-eval-harness

303d0dd

fix(workspace): retain the previous run until an overwrite is published

6ded28e

fix(runner): fail a turn whose complete payload cannot be parsed

44ad162

fix(eval): send the staged run workspace as the request client context

20c1d0b

fix(eval): honour KLEA_EVAL_WORKSPACE when --workspace is not given

2a9697d

feat(eval): add a minimal report writer for klea-eval report

2f83f8b

docs(eval): record deferred grading/indexing work and sync the eval d…

85bcf59

…evdoc

docs(eval): record eval-harness fixes in the changelog and add a sess…

fb6ce7a

…ion log

Merge branch 'development' into feat-eval-harness

a2d1f4b

Navigation Menu

Repository navigation

Feature: eval harness - #220

Feature: eval harness#220
sanjayankur31 wants to merge 91 commits into
developmentfrom
feat-eval-harness

sanjayankur31 commented Oct 28, 2026

Uh oh!

sanjayankur31 commented Oct 28, 2026

Uh oh!

sanjayankur31 commented Oct 28, 2026

Uh oh!

khanak0509 commented Oct 28, 2026

Uh oh!

sanjayankur31 commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

khanak0509 commented Nov 11, 2026

Uh oh!

khanak0509 left a comment •

edited

Loading

Uh oh!

khanak0509 commented Nov 11, 2026

Uh oh!

sanjayankur31 commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		passed = case_run.metrics is not None and case_run.metrics.task_success is True
		typer.echo(f"{case_run.case_id}: {'pass' if passed else 'fail'}")

Conversation

sanjayankur31 commented Oct 28, 2026

Uh oh!

sanjayankur31 commented Oct 28, 2026

Uh oh!

sanjayankur31 commented Oct 28, 2026

Uh oh!

khanak0509 commented Oct 28, 2026

Uh oh!

sanjayankur31 commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

khanak0509 commented Nov 11, 2026

Uh oh!

khanak0509 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

khanak0509 commented Nov 11, 2026

Uh oh!

sanjayankur31 commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

khanak0509 left a comment •

edited

Loading