(experiment) Refresh skill guidance

[sdairs]

Existing skills can load unrelated guidance, require repeated setup or confirmation, and apply workload-dependent advice as universal rules. This refresh scopes activation and reference loading across all 11 skills, preserves domain/API constraints, and ties operational workflows to the user's existing context and requested outcome.

For example, a supplied-DDL review no longer requires live discovery or a full compliance report; a collector retry preserves existing credentials; connecting to managed Postgres does not automatically reset its password. Performance recommendations preserve nulls, precision, duplicates and requested query windows.

Review and cherry-pick boundaries

Based on upstream 2f6ec4b17a81a435dd116f9ac19d7b45d44dbd61. Each skill was completed and committed before starting the next. Required metadata, references, packaging and build changes are included in the relevant skill commit. All 11 skill patches were checked to apply independently to that baseline. The shared catalog/maintainer update is a follow-on commit intended for the combined refresh; the eval proposal is separate.

Commit	Change
395a7f0	refactor(clickhouse-best-practices): scope guidance and repair validation
451af1a	refactor(chdb-datastore): scope activation and qualify pandas migration
b8ea7c5	refactor(chdb-sql): route by API and preserve the chosen stack
37e8550	refactor(clickhouse-architecture-advisor): make reviews workload-driven
5b0feae	refactor(clickhouse-js-node-coding): replace answer mandates with API guidance
8b51c45	refactor(clickhouse-js-node-troubleshooting): diagnose from available evidence
dad2918	refactor(clickhouse-js-node-rowbinary): scope format and optimization guidance
6d9605e	refactor(clickhouse-managed-postgres-rca): tie diagnosis to incident evidence
3802b3b	refactor(clickstack-otel-collector): preserve existing setup and verify scoped ingestion
27c3709	refactor(infra-clickhouse): reuse existing infrastructure and scope provisioning
f3b5538	refactor(infra-postgres): preserve credentials and verify scoped operations
6024917	docs: align repository maintenance guidance and skill catalog
b08d374	docs: propose an external skill evaluation harness and comparison plan

Notable changes

• Shorter skill entry points route to relevant examples and references; full review formats are optional. Narrow descriptions avoid turning generic pandas, SQL or app tasks into stack migrations or infrastructure setup.
• Best-practices rules qualify Nullable, key ordering, LowCardinality, JOIN pushdown and partition advice. The compiler preserves complete rule bodies, including multiple examples and caveats.
• SQL validation now parses every SQL fence using clickhouse format, fails on malformed SQL or a missing executable, and never executes the example queries. CI installs a pinned parser release.
• RCA distinguishes observations from hypotheses and respects the incident window. Collector and infrastructure recipes preserve existing resources and credentials, bound retries, and verify the requested result. Postgres connection examples explicitly verify TLS.
• RowBinary wire-format invariants and codec source remain intact; specialization and format selection are scoped to the task. The new format reference is included in package assets.

Validation

• All 31 best-practices rules validate; all 5 parser/compiler tests pass; TypeScript checks pass.
• All 67 SQL fences parse on ClickHouse 25.8.24.21 (the CI pin) and the local 26.9.1.585 build. Parsing does not validate runtime semantics or performance.
• Generated best-practices documentation is current; its 33 relative file links resolve.
• All 11 skill frontmatters and applicable version metadata agree. No new broken relative Markdown file links; pre-existing historical/package-document links remain unchanged. External URLs and anchors were not exhaustively checked.
• 72 affected Bash examples pass syntax checks with documented placeholders substituted; collector/scenario YAML parses. These commands were not executed against infrastructure.
• RowBinary npm pack --dry-run --ignore-scripts includes the new reference. Codec source is unchanged; the codec test suite was not rerun.
• Full branch diff passes git diff --check.

No cloud resources were provisioned, no live collector/CLI integration workflow was exercised, and no model-quality evals were run. The operational recipes still need version-pinned integration coverage. Reduced instruction size is not evidence of improved model behavior.

External eval proposal

docs/skill-evaluation-proposal.md specifies an external harness to build later: no-skill/baseline/candidate arms, isolated per-skill and full-catalog discovery tests, pinned model/runtime settings, scenario and fixture contracts, independent semantic checks, blinded rubrics, cost/latency reporting, and a result format to bring back to this PR. It includes scenarios for all 11 skills and a proposed runner interface; there is no implemented eval command in this repo.

Keep this PR in draft while reviewing the individual changes and, if desired, building and running that separate evaluation project. Any results should identify the exact evaluated PR SHA and the skill commits recommended for adoption.

References used

• OpenAI — Rethinking skills and prompts for GPT-6 Astra, September 11, 2026.
• Anthropic — The new rules of context engineering for Claude 5 generation models.
• Anthropic — Skill authoring best practices.
• Anthropic — Prompting Claude Fable 5.
• Anthropic — Prompting Claude Fable 5.1.

Reviewed September 12, 2026. These informed narrower activation, progressive disclosure, removal of redundant behavioral mandates and model-specific evaluation. Anthropic recommends evaluation before extensive changes; this PR follows the requested changes-first, external-evals-later sequence and makes no measured quality claim.

[sdairs]

Ran a small remote eval on GPT-6 Astra (Codex) and Claude Opus 5: three tasks covering NULL semantics, sort-key selection, and a narrow SQL expression explanation, with no skill, baseline skill, and PR skill variants; n=2 per cell, 36 runs total. All 36 answers were correct on the core task. This tests explicit use of the best-practices skill, not discovery or the other skills.

The signal is same correctness, greater efficiency: versus baseline, the PR reduced average answer length/tool calls by 23%/40% on Astra and 18%/22% on Opus, with 24% lower reported Opus cost. Astra also stayed more focused; Opus still produced unnecessary detail and unsupported performance estimates. Promising evidence for retaining these changes, with more to improve and validate on held-out tasks.

[sdairs]

Eval summary for b08d374 vs main 2f6ec4b: smaller entrypoints and targeted workflow gains; overall efficiency remains mixed.

Best-practices skill: three advisory tasks, n=2 per task/arm/model, high effort.

Candidate vs baseline	GPT-6 Astra	Opus 5
Mean agent time	−27%	−14%
Mean tool calls	−40%	−22%
Mean reported cost	Unavailable	−24%
Core correctness	6/6 → 6/6	6/6 → 6/6

• Context: PR entrypoints are 66.8% smaller. Description-routing probes saved 663 initial tokens in all 18 Sol pairs; expected selection was 18/18 vs 17/18.
• Local follow-ups, not yet pushed (through c7850c9): focused chDB SQL references reduced guidance loading; targeted fixes corrected observed Node aggregate overflow, unsafe write-replay advice, and pandas compatibility failures.
• DataStore Astra replication: improved guidance passed 3/3 vs 2/3, but used 23% more tokens, 36% more calls, and 7% more agent time.

Small samples support these specific findings, not package-wide superiority. No-skill also solved all core best-practices tasks; runner friction affects timing.