Repository navigation
feat(novelty): routing takes the catalog field, skips disabled and mirrored skills, adds all-vs-all - #172
feat(novelty): routing takes the catalog field, skips disabled and mirrored skills, adds all-vs-all#172napetrov wants to merge 19 commits into
Conversation
…anifest --catalog already took the competitors from a skill directory; this adds the name the flag is asked for under, and makes the catalog reader the one catalog scan uses. A skills.yaml at the root now decides which skills are competitors. A declared skill that cannot be loaded is skipped with a warning instead of making the whole field unreadable. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A shell glob over a skills directory passes every file beside the skills, and the first helper script (activation-rule.mjs) aborted the whole routing run. A skill: competitor that is not a SKILL.md, or a directory without one, is now left out of the field with a warning. The artifact under test is still refused, and a text: competitor is unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…the field No harness lists a skill that sets disable-model-invocation, so the model cannot select it, and a routing card for one measured a choice that does not exist. Such skills are now left out of a --catalog and of --competitors, with a warning on stderr. --include-disabled keeps them. A disabled artifact is still routed, with a warning. The flag is read with the truthy set repos.budget already uses. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A plugin copy of a skills/ tree is a byte-identical SKILL.md under another path with the same name. The catalog read both, so the field named the skill twice and the run was refused. Catalog entries are now grouped by the SHA-256 of SKILL.md; the copy nearest the catalog root stands for the card, and the others are named in the warnings. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… table Every skill with queries becomes the gold in turn and its queries are routed against the whole field. The output names the skills picked for other skills' queries (attractors), with how many of those queries each took and from which skills. Lexical arm always; the model arm when --model is given. Takes --queries-by-skill and --out, and no --artifact. The --artifact flag is now optional for a single run. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…-model counts its calls --competitors-from-catalog was an alias of --catalog, which is already on main. Drop the alias. The changelog and docs now describe the --catalog change itself: a skills.yaml at the catalog root decides which skills are competitors. Two catalog skills with one name and different content are refused, naming both paths, instead of falling through to check_field's "pass each competitor once". With --model, all-vs-all prints the number of model calls (one per query) to stderr before it makes them, and the docs note the cost. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 3ab7f1b. The field logic holds up: the manifest read, the name-clash check before the mirror dedupe, and the disable-flag spellings behave as described. The problems are in the new --all-vs-all mode and its tests. Five things are worth fixing before merge, and one nit.
What I ran, on a Linux review host at 3ab7f1b:
make ci-local: 20 passed, 8 failed, 7 skipped. The 8 failures are exit 127 (ruff, zizmor, pre-commit, bandit, mypy, deptry, mkdocs, pip-audit missing from my venv):make devstops at the tools install on this base, which #179 fixed onmain. Static checks, mypy and the strict docs build pass on GitHub's CI at this head. The test job passed: 7,100 passed, 21 skipped, changed-line coverage gate passed.tests/test_novelty_routing.py: 79 passed.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run of about 60 mutants on the new code: 35 killed, the survivors are in the test comment.
Checked, no issues
- Disable flag:
true,True,'true',yes,on,1read as disabled;false,no,ydo not. This matchesrepos/budget.py. - Clash and mirrors: a disabled skill that shares a name with an enabled one is dropped first, so there is no false clash. Ordering is deterministic.
- Call count: the stderr count equals the calls made on a valid queries file.
- Confusion and attractor figures:
own_rate,wrongly_picked,sibling_queriesmatch hand counts, and a query with no shared word is(none). - Changelog fragments: format and location match the existing ones.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #172 +/- ##
==========================================
- Coverage 91.98% 91.96% -0.02%
==========================================
Files 223 223
Lines 27743 27999 +256
Branches 4560 4622 +62
==========================================
+ Hits 25520 25750 +230
- Misses 1574 1588 +14
- Partials 649 661 +12
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 70b7a72, which fix the nit from my review:
70b7a72:dokimi/docs/skill-novelty.md:602: the sentence now says a disabled artifact is still routed, with a warning, and the flag does not change that; the doubled "before quoting either:" at line 647 is dropped.
make ci-local at 70b7a72: 20 passed, 8 failed, 7 skipped.
Not approving yet. Still open: the bug and the four risks from my review.
… file, max-usd doc Rank with the gold last so a lexical tie is no win; count n_failed and n_abstained in the model arm, warn on failures and exit non-zero when every call failed; catch OSError/UnicodeDecodeError on --queries-by-skill; say routing has --max-usd; add tests for the surviving mutants. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…rowth caps Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 924aba9. All five points from my earlier review are fixed and tested, and the field logic still holds. Two things are worth fixing before merge, both in the new --all-vs-all command, and there are two nits inline plus one in the body.
What I ran, on a Linux review host at 924aba9:
make ci-local: 28 passed, 1 failed, 7 skipped. The failure ispip-auditongitpython 3.2.0(CVE-2026-114145, fix 3.2.1), a pinned dependency the PR does not touch. Tests: 7,258 passed, 1 skipped. The changed-line coverage gate passed.tests/test_novelty_routing.pyandtests/test_novelty_routing_all_vs_all.py: 99 passed.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run of about 55 mutants on the new code. The survivors from my earlier review are killed, except
gold=0, which you explained. Several new ones survive (below).
Nit: the all-vs-all tests leave a few outputs unpinned. With the full suite at 99 passed, these mutants survive: seed=0 passed to all_vs_all from _cmd_routing_all_vs_all; the top-five cap in _print_attractors changed to three; the name tie-break on equal wrongly_picked (and in from) dropped or reversed; the gold != name filter removed from sibling_queries, which only changes rate for an attractor that has queries of its own (the exact-row tests use a hub with none). Suggestion: add one case with two attractors tied on count and one attractor that also has queries, and assert seed in the saved payload.
Checked, no issues
- Tie scoring: two identical cards give
confusion {"a": {"b": 1}, "b": {"a": 1}}andown_rate0.0 in both field orders. A query with no shared word is(none). - Failed calls: a transport that raises on every call gives
n_failedequal to the query count, a stderr warning and a non-zero exit with the partial result written. - Queries file: a missing file, a non-UTF-8 file, a top-level list,
null, non-string items, an empty file and an unknown skill name each exit with a message, not a traceback. --max-usd: with a cap of 1.0 and a mock priced at 0.6 per call, the run stops after 2 calls withspend cap reached, and no--outfile is written. The docs now say so.- Field logic:
true,True,'true',yes,on,1read as disabled;falseand'off'do not. Byte-identical mirrors collapse to the copy nearest the root, and a same-name clash with different bytes names both paths. - Docs and changelog: the flags, outputs and defaults in
skill-novelty.md, the walkthrough and the five fragments match the code.
| payload = {"mode": "all_vs_all", "queries_by_skill": args.queries_by_skill, | ||
| "n_candidates": len(field), "result": result} | ||
| out.write_text(json.dumps(payload, indent=2), encoding="utf-8") | ||
| if model_arm and failed and failed >= result["n_queries"]: |
There was a problem hiding this comment.
A partial model failure exits 0 and prints OK Saved.
The non-zero exit only fires when failed >= result["n_queries"]. With a transport that answers once and then raises rate limited, five queries over db and perf print:
WARNING: 4 model call(s) failed (first error: rate limited); ...
OK Saved all-vs-all routing: ...
model: hub picked for 1 of 5 sibling queries (from db)
Exit code 0. The saved model arm has own_rate {db: 0.0, perf: 0.0} and n_failed 4. perf never got an answer, yet it reads as a measured 0.0. A rate limit mid-run in a script or CI step looks like success, and the attractor line looks like a measurement. The warning scrolls past, and the figures are lower bounds only if the reader knows to look at n_failed.
Suggestion: exit non-zero whenever failed is non-zero (the file is still written), or add a --allow-failed-calls escape. Report own_rate as null for a skill whose queries all failed. Add a test with one success and several failures; the existing tests only cover all-fail.
There was a problem hiding this comment.
Fixed in 5e27afa: a run with any failed model call now writes the result and exits non-zero (no 'OK Saved'). Failed calls are left out of every rate: a skill whose queries all failed has own_rate null (n_scored 0, n_unscored n), not 0.0. Test: test_a_partial_model_failure_exits_non_zero_and_leaves_failed_queries_out (one success, two failures).
| of strings. | ||
| """ | ||
| try: | ||
| data = json.loads(Path(path).read_text(encoding="utf-8-sig")) |
There was a problem hiding this comment.
A queries file with a repeated skill key silently keeps only the last list.
{"a": ["profile"], "a": ["slow"]} loads as {'a': ['slow']} with no error. A hand-merged file measures fewer queries than the author wrote, and n_queries is the only trace.
Suggestion: parse with object_pairs_hook and raise RoutingInputError(f"{path}: skill {key!r} appears more than once") on a repeated key. Add a test for it.
There was a problem hiding this comment.
Fixed in 5e27afa: load_queries_by_skill parses with an object_pairs_hook and raises RoutingInputError ": skill 'a' appears more than once". Test: test_a_skill_named_twice_in_the_queries_file_is_refused.
| raise RoutingInputError("all-vs-all: the field names a skill more than once") | ||
| unknown = sorted(set(queries_by_skill) - set(names)) | ||
| if unknown: | ||
| raise RoutingInputError("the queries name skills that are not in the field: " |
There was a problem hiding this comment.
Nit: a queries file written for the whole catalog fails on a disabled skill without naming the cause. The field drops disable-model-invocation skills by default, so a file that names one ends with the queries name skills that are not in the field: `gamma` . The earlier "left out catalog skill" warning has scrolled past by then; the same run with --include-disabled succeeds.
Suggestion: append "; a skill that sets disable-model-invocation is left out unless you pass --include-disabled" to this message.
There was a problem hiding this comment.
Fixed in 5e27afa: the message now ends with '; a skill that sets disable-model-invocation is left out unless you pass --include-disabled'. Test: test_the_not_in_the_field_error_names_the_disable_flag_and_its_override.
|
|
||
| def _check_all_vs_all_args(args: argparse.Namespace) -> None: | ||
| """Refuse the flag combinations ``--all-vs-all`` cannot run with.""" | ||
| if args.artifact: |
There was a problem hiding this comment.
Nit: --queries is accepted with --all-vs-all and ignored. --queries /nonexistent.json --all-vs-all ... prints OK Saved all-vs-all routing. --artifact is refused here for the same reason.
Suggestion: add if args.queries: raise SystemExit("--all-vs-all takes --queries-by-skill, not --queries") next to the --artifact check.
There was a problem hiding this comment.
Fixed in 5e27afa: --all-vs-all now refuses --queries and --queries-from-headings with 'takes --queries-by-skill, not --queries or --queries-from-headings'. Test: test_all_vs_all_refuses_the_single_artifact_query_flags (both flags).
Single-mode routing.json keeps the selector model; the all-vs-all model arm did not, so files from two different models were indistinguishable. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
… its own The existing guard never ran: in its fixture no attractor has queries, so "rate over all queries" and "own queries not excluded" mutants passed. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at adbe542. The two commits since 924aba9 are fine: model is recorded in the model arm and the new attractor-rate test fails when the gold != name filter is removed. The four threads from my last review are unchanged at this head, so the PR is not ready: T1 and T2 are still risks.
What I ran, on a Linux review host at adbe542:
make ci-local: 28 passed, 1 failed, 7 skipped. The failure ispip-audit, which fails on the lock files; this PR does not touch them. Tests: 7,261 passed, 1 skipped.- Reproductions for the open threads, with the LLM transport mocked.
- A mutation check of the two new tests: removing
model=client.modelfails bothmodel-a/model-bcases (2 failed, 31 passed); removing thegold != namefilter insibling_queriesfails the new attractor test (1 failed, 32 passed).
Still open from my earlier review:
dokimi/dokimi/commands/novelty.py:364: a partial model failure still exits 0 and printsOK Saved. Re-run at this head: 5 queries overdbandperf, a transport that answers once then raisesrate limited, gives exit 0,own_rate{db: 0.5, perf: 0.0}andn_failed4.dokimi/dokimi/novelty/routing.py:541:{"a":["x"],"a":["y"]}still loads as{'a': ['y']}with no error.dokimi/dokimi/novelty/routing.py:600: the "not in the field" error still does not namedisable-model-invocationor--include-disabled.dokimi/dokimi/commands/novelty.py:375:--queriesand--queries-from-headingsare still ignored under--all-vs-allwhen--queries-by-skillis also given;--queries /nonexistent.json --queries-by-skill q.json --all-vs-all --out o.jsonprintsOK Saved all-vs-all routing.
@xaleryb's points on the own_rate denominator, the lexical tie among non-gold cards and the missing field rule in the payload wait for the author.
Checked, no issues
- Docs and changelog: the flags, outputs and defaults in
skill-novelty.md, the walkthrough and the five fragments match the code, including the newmodelsentence;mkdocs build --strictpasses. - Repo rules: SPDX header on the new test file, no schema or
with_docschanges, no generated artifacts, changelog fragments inchangelog.d/.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/scripts/growth_baseline.json # dokimi/tests/test_novelty_routing.py
…ree, partial failures exit non-zero Rates exclude unscored queries (no content word, failed model call) and report null when a skill has none scored. A tie among other cards is (tie). Repeated skill keys in the queries file are refused, the not-in-the-field error names disable-model-invocation, --queries is refused with --all-vs-all, and routing results record the field rule. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Review-body items from @xaleryb's comment, in 5e27afa: (1) own_rate denominator: rates are now over scored queries; a query with no content word or a failed model call is unscored, reported in n_scored/n_unscored per skill, and a skill with none scored has own_rate null (test_a_query_with_no_content_word_is_not_scored_as_a_miss). (2) lexical tie among non-gold cards: two or more other cards tied at the top are a (tie) bucket in every field order, never an attractor (test_a_tie_among_other_cards_is_a_tie_in_any_field_order). (3) field rule: routing.json, single and all-vs-all, now carries field {include_disabled, competitors, left_out} (test_the_saved_payload_records_the_field_rule, test_a_single_artifact_routing_run_records_the_field_rule_too). |
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/skill-novelty-walkthrough.md # dokimi/docs/skill-novelty.md # dokimi/dokimi/commands/novelty.py # dokimi/scripts/growth_baseline.json
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi novelty routing` parameters changed:
+ added option `--all-vs-all`
+ added option `--include-disabled`
+ added option `--queries-by-skill`
~ changed option `--artifact`: required: True -> FalseUpdated on every push; the |
Part of #4.
Depends on #164 and #168: rebase after both merge (routing.py and the routing tests overlap with those branches).
Summary
novelty routingbuilds its field the way a harness does: the catalog's own manifest, without skills the model cannot select, and with each mirror counted once. A new--all-vs-allmode routes every skill's queries against the whole field and reports who takes whose queries.Evidence
Eleven new tests fail against
intel/main; the other 68 in the file pass there. The run revertscommands/novelty.pyonly, because the new tests importall_vs_all,load_queries_by_skillanddisables_model_invocationfromnovelty/routing.py, so reverting that file too would fail the whole module on import:The
routing.py-level tests (the attractor table, the(none)rule, the disable-flag spelling intest_the_disable_flag_is_read_the_way_the_repo_budget_reads_it) need the new module and cannot be run againstintel/mainat all.Local checks on the rebased head:
Merge Danger
Door: two-way. Every mode is additive except the default field, which now differs for catalogs with disabled or mirrored skills. Reverting restores the old field.
Blast Radius: medium. Existing routing figures change for any catalog that has disabled or duplicated skills, so older numbers are not comparable with new ones.
--catalogor--competitorsfield no longer includes a skill that setsdisable-model-invocation, and an artifact with that flag is routed with a warning. To keep the old field, pass--include-disabled.skills.yamlpresent,--catalogtakes its entries, and a declared skill that does not load is skipped with a note rather than aborting the run.SKILL.mdunder another path is dropped, keeping the copy nearest the catalog root.skill:reference to a file that is notSKILL.md, or to a directory without one, is skipped with a warning.text:and plain paths are unchanged.--all-vs-allmakes one selector call per query over the whole field when--modelis given. The CLI prints the call count on stderr before it starts.--artifactis now optional only with--all-vs-all; a single run without it still fails.all_vs_allpayload written to its own--outfile.Checklist
pytestpassesruff,mypy, and schema checks passdocs/skill-novelty.mdanddocs/skill-novelty-walkthrough.md🤖 Generated with Claude Code