Tighten brainstorming research gate and evidence contract

[Jonksar]

Coordination PR for obra/superpowers#2116. This targets that PR's fork branch, not dev, so Jonksar can merge the feedback patch into the branch behind the existing upstream PR. It remains draft only because the required complete-diff human review checkbox is still open.

Who is submitting this PR? (required)

Field	Value
Your model + version	gpt-5.6-sol at high reasoning for the skill change; gpt-5.6-luna at high reasoning for every task-completion parent and research child
Harness + version	Codex desktop; task-completion evals used Codex CLI 0.146.0 through Pier 0.3.0
All plugins installed	OpenAI-curated Superpowers, GitHub, Gmail, Slack, Figma, Google Drive, Notion, and Superhuman; Ponytail; OpenAI-bundled Browser, Chrome, Computer Use, Sites, and Visualize; OpenAI primary-runtime Documents, PDF, Presentations, Spreadsheets, and Template Creator; Claude frontend-design; local Reiterate skills
Human partner who reviewed this diff	Joonatan Samuel requested this follow-up; complete-diff review is pending, so this PR is draft

What problem are you trying to solve?

The research step in obra#2116 depended on the model deciding that technology felt unfamiliar or that local context was insufficient. That misses the motivating failure: a confident but stale pg-boss recommendation did not feel unfamiliar. The first observable rewrite still under-triggered when work changed an existing third-party integration rather than introducing a dependency.

Maintainer feedback asked for one observable predicate, a mirrored gate, a positive source-evidence recipe, measured read-only compliance at N>=10, adversarial familiar-technology coverage, and honest runtime/cost reporting.

What does this PR change?

The brainstorming skill now applies one observable research gate after path-scaled clarification on every path. The gate covers behavior defined by an external library, service API, or protocol; dependency changes; difficult-to-reverse decisions; and version-sensitive API behavior.

When true, it dispatches exactly one focused read-only child, gives that child a source-specific evidence contract, waits, and consumes the findings in the selected path's probe plan, short design, or approach comparison. External projects require implementation and test links; closed services and protocols may use official versioned specifications and API documentation.

Is this change appropriate for the core library?

Yes. The failure and gate apply across languages, repositories, and domains. This adds no dependency or third-party integration; it changes only the general-purpose brainstorming workflow already proposed in obra#2116.

What alternatives did you consider?

• Keep the unfamiliar/current-knowledge trigger: rejected because confidence is not an observable proxy for staleness.
• Trigger only when adding or replacing a dependency: rejected after a focused Luna run invoked research in only 4/10 tasks that changed existing external APIs, libraries, or protocols.
• Require external source files for every finding: rejected because closed service APIs and standards-based protocols may have authoritative versioned documentation but no public implementation repository.
• Dispatch multiple researchers or loop until convergence: rejected because one focused child is enough to obtain evidence and the measured overhead is material.

Does this PR contain multiple unrelated changes?

No. The checklist, graph, explanatory prose, and evidence contract are representations of the same brainstorming research gate.

Existing PRs

• I have reviewed all open AND closed PRs for duplicates or prior art
• Related PRs: obra/superpowers#2116, #386, #1512, #2063, #2123, #1890

This is not a competing upstream submission: it is the requested follow-up patch for obra#2116. obra#386 is the older feature request; obra#2063 supplied the three-path router this preserves. obra#1512 concerns recommendation comparison dimensions rather than source-grounded research dispatch. The closed PRs address broader brainstorming orientation or routing issues, not this observable gate and evidence contract.

Environment tested

Harness	Harness version	Model	Reasoning
Codex CLI via Pier — control primary	Codex CLI 0.146.0 / Pier 0.3.0	gpt-5.6-luna	high
Codex CLI via Pier — experiment primary	Codex CLI 0.146.0 / Pier 0.3.0	gpt-5.6-luna	high
Codex CLI via Pier — experiment research child	Codex CLI 0.146.0 / Pier 0.3.0	gpt-5.6-luna	high
Codex desktop — skill change	current local build	gpt-5.6-sol	high

Local skill editing and PR preparation used gpt-5.6-sol at high reasoning. Every executed task-completion agent used the same Luna configuration: gpt-5.6-luna, reasoning_effort = "high", Codex CLI 0.146.0, and the same config file. Both paired arms used the same ten task checksums, concurrency 3, and zero retries. The only intentional treatment difference was the exact final conditional research gate and evidence contract.

New harness support (required if this PR adds a new harness)

Not applicable. This PR adds no harness support.

Evaluation

Initial prompt/session: The upstream change was motivated by brainstorming recommendations that were not grounded in established code, including a confident but stale pg-boss recommendation. This follow-up started from the five-point maintainer review on obra#2116.

Task-completion control vs experiment

The original battery mainly exercised internal library features; only its HTTPX multipart case clearly required external-contract research. This replacement battery was hand-picked for existing third-party APIs, libraries, and protocols. The control prompt asked Luna to implement and verify with no research gate. The experiment added the exact final conditional gate and evidence contract; it did not state that the gate was true.

Metric	Control / baseline	Research experiment	Experiment vs control
Tasks	10	10	Same N and checksums
Strict success	60% (6/10)	60% (6/10)	0 percentage points
Mean partial score	99.9567%	99.7766%	-0.1802 percentage points
Mean trial wall time (agent + verifier)	20m 45.8s	32m 02.4s	+11m 16.6s (+54.3%)
Cumulative trial wall time	3h 27m 38s	5h 20m 24s	+1h 52m 46s (+54.3%)
Batch wall time at concurrency 3	1h 16m 39.8s	1h 59m 19.9s	+42m 40.1s (+55.7%)
Mean model cost per trial	$0.29814	$0.16871	-$0.12943 (-43.4%)
Total model cost	$2.98139	$1.68713	-$1.29426 (-43.4%)
Research path invoked	0% (0/10)	100% (10/10)	+100 percentage points
Exactly one research child when invoked	n/a	100% (10/10)	Required only in experiment
No research-child implementation edits	n/a	100% (10/10)	Trace-audited
Luna/high trace audit	100% of recorded contexts	100% of recorded parent + child contexts	Same effective configuration

Strict success was unchanged. Dateutil changed from control failure to experiment success, while kgateway changed from control success to experiment failure; the other strict outcomes were unchanged. Mean partial score declined by 0.1802 points and wall time increased by 54.3%. Recorded model cost happened to decrease by 43.4%, but N=10 is too small to treat that as a stable efficiency claim.

Per-test results

✅ strict means every verifier check passed; ❌ strict means at least one failed. Partial score retains feature-to-pass and pass-to-pass credit. Research marks come from trace-confirmed dedicated child sessions, not agent self-report.

Test case	Why research should trigger	Control research	Control strict	Control partial	Experiment research	Experiment strict	Experiment partial
actionlint — action pinning	GitHub Actions and reusable-workflow uses: semantics	❌	✅	100.0000%	✅	✅	100.0000%
dateutil — RFC 5545 timezone interop	iCalendar VCALENDAR/VTIMEZONE protocol behavior	❌	❌	99.9524%	✅	✅	100.0000%
DynamoDB Toolbox — conditional attributes	DynamoDB conditions plus Zod/JSON Schema contracts	❌	✅	100.0000%	✅	✅	100.0000%
FastAPI — deprecation response headers	RFC 8898/8594/8288/7231 and OpenAPI metadata	❌	❌	99.9083%	✅	❌	99.9694%
Google GenAI — streamed function args	Version-sensitive streamed partialArgs/willContinue API	❌	✅	100.0000%	✅	✅	100.0000%
GoReleaser — retry/publish auditing	HTTP, Artifactory, blob providers, and Retry-After	❌	✅	100.0000%	✅	✅	100.0000%
gql — incremental GraphQL delivery	Versioned multipart HTTP and WebSocket protocol	❌	❌	99.8792%	✅	❌	99.7585%
kgateway — consistent hash policy	TrafficPolicy to Envoy RouteAction.hash_policy	❌	✅	100.0000%	✅	❌	99.0741%
Numba — stencil boundary modes	Explicit llvmlite 0.46.0 dependency interaction	❌	✅	100.0000%	✅	✅	100.0000%
Testem — bail on test failure	Mocha, Jasmine2, and QUnit adapter behavior	❌	❌	99.8273%	✅	❌	98.9637%

All 20 paired trials completed with zero infrastructure errors and zero retries. Trace inspection found exactly one research child in every experiment trial and none in control. All recorded parent and child turn contexts reported gpt-5.6-luna at high reasoning. The child traces contained no file_change or apply_patch events; research used read/search/fetch operations and temporary files outside the task workspace.

An earlier diagnostic experiment with the narrower “introduces or replaces a dependency” wording invoked research in only 4/10 of these expected-trigger cases. That result drove the explicit existing-library/service/protocol clause. It is not mixed into the paired performance table above, which reports only the exact final wording.

The requested use of Luna was honored for every task-completion parent and research child. No current-Claude arm was run.

Repository validation:

• bash tests/writing-skills/test-render-graphs.sh — 8 passed, 0 failed
• bash tests/codex-plugin-sync/test-sync-to-codex-plugin.sh — passed
• TZ=UTC bash tests/codex/test-package-codex-plugin.sh — all passed from a normal clone (the script intentionally rejects linked worktrees)
• git diff --check — passed
• Fresh read-only review of the complete final diff — no concrete issues remain
• Reviewer-facing explainer rendered, but publication was blocked because the local publisher does not support the superpowers repository; no docs checkout was modified

Rigor

• If this is a skills change: I used superpowers:writing-skills and completed adversarial pressure testing (results above)
• This change was tested adversarially, not just on the happy path
• I did not modify carefully-tuned content (Red Flags table, rationalizations, "human partner" language) without extensive evals showing the change is an improvement

No Red Flags table, rationalization list, or human-partner language was modified.

Human review

• A human has reviewed the COMPLETE proposed diff before submission

This remains a draft coordination PR in Jonksar's fork. Joonatan must review the complete one-file diff before it is merged into the branch behind obra#2116.