Add RDNA4 GPU support (RX 9070 XT, RX 9070, RX 9060 XT, RX 9000 XT) f… - #1032
Open
Maxritz wants to merge 317 commits into
Open
Add RDNA4 GPU support (RX 9070 XT, RX 9070, RX 9060 XT, RX 9000 XT) f…#1032Maxritz wants to merge 317 commits into
Maxritz wants to merge 317 commits into
Conversation
Fix GEAK version in compatibility matrix
Removing component table from release notes
…drop-gh fix(install): download release wheel and hotfix via public curl instead of gh CLI
Restore single-node default to route GPU/serving work through the Ray backend when INFERENCE_OPTIMIZER_RAY_EXEC is unset. Multi-node stays off and pytest keeps the local subprocess path.
…izer hardening (AMD-AGI#1018) * prompt: strip version tags, de-duplicate cross-source rules, slim builder sections - Remove v0.8/§3.x/M3/Inv-9.1/DESIGN §14.1/M5+ version tokens from all generated sections; keep surrounding sentences intact. - SESSION CONTEXT: collapse the phase-chain re-statement into a one-line pointer to PHASE CONTRACT; remove deadline auto-flush sentence. - PIPELINE & TIME BUDGET: remove trailing explore-rebench and deadline auto-flush prose (canonical location: Hard rules / §3a). - DECISION FRAMEWORK: trim step 3 (rebench) and step 4 (never-propose) to pointer sentences; remove step 7 (deadline); slim skip_to_close back-reference in step 6. - PHASE CONTRACT (§3a): delete dead is_phase_interleave_enabled import, call, and the entire `if interleave:` block (function returns False); pass interleave=False explicitly to render_phase_proposable_bullets. Add `policy_denied` mention to the denial sentence so tests remain semantically intact. - Grid hint (explore): slim to param shape + dedup rule; remove provenance prose and breadth explanation (kept in IDEA GENERATION and Hard rules). - IDEA GENERATION: shorten fingerprint sentence to one line. Co-authored-by: Cursor <cursoragent@cursor.com> * prompt(orchestration.md): compress Phase awareness, specialist dials, and CRITICAL block Phase awareness (was ~257 lines, now ~100): - Remove duplicate 6-phase chain declaration and per-tick Phase block format description (both in PHASE CONTRACT / §3a). - Remove duplicate skip_to_close caveat (canonical: §3a); replace with a one-line pointer. - Compress per-phase intent map: drop action-set relists (already in §3a bullets), keep unique content per phase — EXPLORE GPU-specialist rules, atomic proposals, advisory proposal scores, plateau behavior; KERNEL drain-pending-KEEPs, no-lever skip_to_sweep, never-fabricate. - Remove "never propose profile/roofline" duplicate (canonical: Hard rules). - Remove duplicate "allowed_actions set" rule (canonical: §3a). One specialist, four dials (was ~140 lines, now ~45): - Replace per-dial default mechanics prose (code-enforced; some prose was wrong: domain-anchored defaults are mode=patch/lane=gpu, not research/cpu) with a concise decision-guidance table. - Merge "When to pick which scope" into the dials section. - Trim 5 emit_intent examples to 2 (domain-anchored + freeform wave). CRITICAL role block (was ~62 lines, now ~11): - Delete "YOU do" / "Specialists do" lists (duplicates phase pipeline and specialist's own prompt respectively). - Delete numbered workflow (duplicates pipeline). - Delete wave-mechanics detail (duplicates specialist-dispatch sections). - Retain the push-hard posture: demand concrete deliverables, re-dispatch on vague results, keep momentum with overlapping waves, ease off at target. Remove version tokens: §3.9 Inv-9.1 → semantic phrase; DESIGN §14.1 → "emit_intent schema". Co-authored-by: Cursor <cursoragent@cursor.com> * prompt(cli): wire macro_cycle through _build_orchestration_prompt _build_orchestration_prompt now accepts macro_cycle: int = 0 and forwards it to build_orchestration_prompt, which passes it to _section_macro_posture. At the CLI call site, the live value is read from coordinator.shared_state.macro_cycle (populated after _seed_shared_state, so it reflects the resumed cycle on --resume). Previously the parameter existed in build_orchestration_prompt but was never passed by the CLI, causing MACRO POSTURE to always render macro_cycle=0. Co-authored-by: Cursor <cursoragent@cursor.com> * test(prompt): update assertions after version-token removal - test_orchestration_prompt_has_no_scoreboard_block: replace assert "Inv-9.1" in prompt with assert "no system-side" in prompt.lower() (the §3.9 Inv-9.1 tag was removed; the semantic no-scoreboard invariant is still present via "There is no system-side priority list"). - test_orchestration_md_has_no_score_view: replace assert "§3.9" in fragment with assert "decision priority" in fragment.lower() (§3.9 tag removed; the Decision priority heading still anchors the same invariant). Co-authored-by: Cursor <cursoragent@cursor.com> * prompt: rename MACRO POSTURE to CYCLE DIRECTIVE placeholder + fix cli docstring - Rename _section_macro_posture -> _section_cycle_directive with an additional cycle_directive: str = "" param. When non-empty the section renders the LLM-authored text; otherwise falls back to the standing breadth->depth arc (no behaviour change on fresh runs). - build_orchestration_prompt: add cycle_directive param, pass through. - _build_orchestration_prompt (cli): add cycle_directive param, fix stale docstring ("seeds the CYCLE DIRECTIVE section"), forward to builder. - Call site: read next_cycle_directive from coordinator.shared_state.orchestration_memory so a resumed run renders the last captured directive immediately. Co-authored-by: Cursor <cursoragent@cursor.com> * state(memory): capture next_cycle_directive from the orchestration checkpoint - CHECKPOINT_REQUEST_PROMPT: add next_cycle_directive JSON key with a 1-3 sentence instruction for the next macro-cycle focus. - _sanitize_cycle_directive: truncates to 1500 chars and rejects text containing policy-override phrases (blacklist). - parse_checkpoint_reply: captures next_cycle_directive via sanitizer; missing key -> ""; no-JSON path also sets it to "". - build_memory_record: carries next_cycle_directive with non-empty-wins semantics (inherits prior value when empty) into the persisted record and the existing orchestration_memory_history rollback ring. - test_orchestration_memory_unit: 11 new cases covering parse capture, length cap, blacklist rejection, and record carry-forward. Co-authored-by: Cursor <cursoragent@cursor.com> * state: add cycle_directive_history ring Bounded (cap 10) ring of per-macro-cycle directives injected into the orchestration system prompt; entries carry {cycle, directive, source, ts} for resume consistency and audit. Round-trips via the existing asdict / from_dict path; no migration needed. Co-authored-by: Cursor <cursoragent@cursor.com> * orchestrator(explore): rebuild orchestration prompt with cycle directive at macro-cycle boundary - cli: cache a pure _rebuild_orch_prompt partial (captures the args-only build inputs) and _orch_prompt_is_user_supplied guard on the coordinator at session start. - explore: add _cycle_directive_fallback (renders _plan_cycle_focus into a focus line) and _reseed_orch_prompt_for_cycle (pick LLM directive else fallback, rebuild the prompt, mutate system_prompt_overrides, append to the cycle_directive_history ring). Never clobbers a user --orch-prompt. - _run_cycle_soft_restart: reseed after the forced checkpoint and before the conversation reset, so the next SEED turn opens the new cycle with a re-focused system prompt on a fresh session. Co-authored-by: Cursor <cursoragent@cursor.com> * test: cover cycle-directive prompt reseed + history ring - Register _cycle_directive_fallback / _reseed_orch_prompt_for_cycle in the Coordinator phase-method delegation map so they resolve on the coordinator. - Add test_cycle_directive_reseed: fallback focus-line render, LLM directive wins, deterministic fallback when empty, user --orch-prompt skip, and the cycle_directive_history ring cap at 10. Co-authored-by: Cursor <cursoragent@cursor.com> * policy(sweep): classify conc_sweep as Coordinator-internal to fix singleton self-collision The SWEEP-entry hook (SweepPhase._on_enter_sweep) auto-enqueues the sole conc_sweep task with source="coordinator_internal" and stamps its id into phase_history[-1].evidence.auto_conc_sweep_task_id; conc_sweep is never LLM-proposed. But conc_sweep was missing from COORDINATOR_INTERNAL_ACTIONS (only roofline/profile/replay_warm_recipe/framework_agent were listed), so at dispatch re-validation PolicyGate.validate_dispatched_task fell through to the delegate-body sweep-family singleton guard. That guard keys on auto_conc_sweep_task_id — the auto-enqueued task's OWN id — so it denied the one and only conc_sweep against itself. The denial surfaced as a conc_sweep task state=failed, which the phase machine read as last_conc_sweep.status "failed" -> conc_sweep_failed -> forced SWEEP->CLOSE with a spurious failure sentinel and 0% gain, skipping the real sweep. Root-cause fix: add conc_sweep to INTERNAL_ONLY_ACTION_NAMES so it is handled uniformly as a Coordinator-internal action. This makes validate_dispatched_task short-circuit it (path checks only, no singleton re-validation), removes it from PHASE_LLM_PROPOSABLE_ACTIONS (already never advertised to the LLM), and routes any stray LLM propose through the accurate "Coordinator-managed" (phase_incompatible) denial instead of a misleading singleton denial. The LLM-facing sweep_phase_singleton / conc_sweep_phase_singleton guards at intent ingress are unchanged, so dedup protection for a full LLM-proposed sweep is intact. Also fixes the FRAMEWORK local-exploration arm, which re-dispatched unconditionally on every discovery-exhaustion tick: in a network-restricted environment (fa phase-discover always times out, every authored deliverable empty) it never exited until the max-hours backstop, starving EXPLORE/KERNEL/ SWEEP. Add _local_explore_plateaued() so the arm stops after DISCOVER_FAILURE_RETRY_LIMIT consecutive no-KEEP local_explore rows, realizing the documented "exits via plateau" contract. Adds regression tests: conc_sweep is Coordinator-internal and not LLM-proposable; the auto-enqueued conc_sweep passes dispatch re-validation against its own evidence; an LLM conc_sweep propose is denied as Coordinator-managed. Co-authored-by: Cursor <cursoragent@cursor.com> * policy(sweep): retire redundant conc_sweep singleton guard Follow-up cleanup to the previous commit, which classified conc_sweep as a Coordinator-internal action. With conc_sweep in COORDINATOR_INTERNAL_ACTIONS, an LLM-proposed conc_sweep is already rejected by _validate_phase_action as Coordinator-managed (phase_incompatible), so the per-action conc_sweep_phase_singleton guard is dead/redundant logic that only produced a less accurate denial reason. Retire it: - Remove PolicyGate._validate_conc_sweep_singleton and its two call sites (delegate + propose_action channels). - Remove the now-unused CONC_SWEEP_ACTION_NAME constant. - Inline the single-remaining-caller _validate_sweep_family_singleton helper back into _validate_sweep_singleton (the "family" parameterisation existed only to share code between the sweep and conc_sweep guards); drop the now unused typing.Callable import. - Update the SWEEP-entry evidence-stamp comment: auto_conc_sweep_task_id is now read only by sweep_phase_singleton to deny a later LLM full-workload sweep. - Retire test_conc_sweep_phase_singleton_denies_after_auto_enqueue; the replacement behaviour (LLM conc_sweep -> phase_incompatible) is covered by the tests added alongside the classification change in test_sweep_phase_auto.py. The full-workload sweep_phase_singleton guard (for the LLM-proposable sweep action) is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(source-policy): open specialist access to installed packages Allow source references and reviewed patch roots across active site and dist packages while keeping real-path containment intact. Co-authored-by: Cursor <cursoragent@cursor.com> * test(integration): cover installed package patch roots Verify reviewed integration accepts non-git package directories beneath an active installed source scope. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(framework): use official five-outcome plateau Remove the local dispatch throttle so local exploration follows the shared framework no-KEEP plateau and macro-cycle boundary semantics. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(state): scope plateau controls to each macro cycle Reset transient phase counters on reloop while retaining durable learning, and filter plateau windows to attempts recorded in the active macro cycle. Co-authored-by: Cursor <cursoragent@cursor.com> * test(policy): expect trusted source scope denials Align dispatched integration-path coverage with the broad installed package source policy. Co-authored-by: Cursor <cursoragent@cursor.com> * Merge chores fixes into long-horizon prompt (AMD-AGI#981) * fix: probe serving-slot per GPU-specialist admit to close race window The dispatcher's serving-priority gate previously computed serving_slot_busy() once per pump pass, so a serving benchmark that started mid-pass could still admit a competing GPU research specialist. Move the probe to immediately before each GPU-specialist admission (keeping the enable-check pass-level), add dispatcher-level regression tests for the defer + SQLite lane release, and document INFERENCE_OPTIMIZER_RAY_GPU_PENDING_LIMIT / _RAY_SERVING_PRIORITY. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: remove dormant/legacy mechanisms and enable framework LLM ranker by default Delete a set of hardcoded-off, deferred, env-gated, and legacy code paths end-to-end (code + tests + prompts/docs) with no fallback left behind: - EXPLORE<->KERNEL phase interleave (hardcoded return False; collapse the *_with_interleave helpers into the plain per-phase ones) - cross-framework use_llm ignored arm (kept the honored audit.py path) - HYPERLOOM_BYPASS_CRITIC operator bypass (integrate_patch always requires a permissive Critic verdict now) - INFERENCE_OPTIMIZER_SWEEP_RUN_EVAL (conc/sweep always inject RUN_EVAL=false) - HYPERLOOM_RECOVER_ALLOW_GPU_RESET (drop the whole rocm-smi --gpureset path) - retired roofline/profile CLI flags (keep the _RetiredFlag class) - flat legacy session layout (INFERENCE_OPTIMIZER_SESSION_LAYOUT) - legacy per-variant-restart conc-sweep path (CONC_SWEEP_SINGLE_SERVER) Also flip the FRAMEWORK LLM candidate ranker to default-on by reusing the orchestration backend's OpenAI-compatible client/credentials. prune_branch, INFERENCE_OPTIMIZER_STRICT_PATHS and _STRICT_PHASE were evaluated and kept. Update unactionable-methods.info.md accordingly. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(optimizer): make current_best carry the full cumulative env stack The best-recipe artifact (current_best / current_setting.sh) dropped every accepted env except the last variant's delta, so the emitted recipe could not reproduce the reported throughput. - proposals.py: seed explore tasks with base_extra_envs from current_best so the executor stacks envs the same way it already stacks server args (root cause: only base_extra_args was injected). - writeback.py: dedupe effective_extra_server_args when lifting to current_best and the stack entry, preventing doubled flag strings. - reference_script.py: allow HIP_FORCE_DEV_KERNARG and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION through the recipe env whitelist. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(report): honest stop_reason and server-boot-failure count The wind-down report rendered before the loop assigned the terminal stop_reason, and crash_count omitted server boot failures. - report.py: fall back to time_exhausted when closing_phase is active and stop_reason is still blank (closing_phase is only entered on the deadline). - report.py: surface a server_boot_failures count (warmup_failed variants from the optimization journal) so crash_count: 0 is no longer misleading. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(recipe): render current_setting.sh from trusted current_best envs The env whitelist exists to sanitize an UNTRUSTED external reference recipe on import; reusing it to gate the rendered current_setting.sh meant every newly validated env had to be added by hand or it silently vanished from the recipe. Render now emits current_best.extra_envs verbatim (skipping empty/dynamic values), and _ENV_WHITELIST is scoped to parse_reference_script only. Reverts the HIP_FORCE_DEV_KERNARG / VLLM_ROCM_QUICK_REDUCE_QUANTIZATION whitelist entries added for rendering. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(orchestrator): drop dead pause_specialist escalate lever and orphaned counter The pause_specialist_<domain> escalate hint was never advertised in any prompt and its only effect was to bump specialist_domain_empty_streak, a counter that was written but never read by any decision, prompt render, or robustness signal. Remove the hint (machine_state helpers + intent_router branch) and the orphaned counter (state field, writeback bookkeeping, gate/robustness allowlists, explore_state helper), plus the tests/golden that pinned them. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(robustness): re-emit prune_branch from the action ladder for no-lever and repeated-failure symptoms The detection signals, builder, PolicyGate allowlist, handler, and prompt surfacing for prune_branch were all intact, but the ladder's symptom->intent mapping had been severed, so pruned_families never populated. Reconnect it: _recommend now emits prune_branch for the no-lever / repeated-failure HIGH symptoms (same_payload_loop, kernel_opt_no_progress, geak_budget_starvation, amdahl_kernel_ceiling_low, and repeated_failure once it escalates to HIGH at a sustained-failure threshold), resolving the family from a fixed map or the symptom subject/evidence. Update the ladder tests, restore the integration test, and refresh the robustness prompt + SKILL docs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(optimizer): dedup vLLM server args after reference/recipe merge Reference-script and per-task extra_server_args are merged into EXTRA_VLLM_ARGS after the early single-value-flag dedup, so a flag set by both (e.g. --gpu-memory-utilization from the baseline default and a recipe override) reached vLLM duplicated. Re-run dedup_vllm_server_args on the fully merged string so the recipe/variant value wins last (no-op for sglang). Co-authored-by: Cursor <cursoragent@cursor.com> * test(optimizer): cover recipe-priority server-arg dedup Assert a single-value flag present in both reference_server_args and extra_server_args collapses to one occurrence with the extra (recipe) value winning, guarding the merged-EXTRA_VLLM_ARGS dedup path. * feat(research): wire scout output into orchestration Persist the Scout's top-level source-backed findings and executable proposals, then re-seed Orchestration with every persisted finding, proposal, and residual question so later empty rounds cannot hide viable work. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(framework): stamp cycle on audit-skip progress Include the active macro cycle on semantic audit skips so current-cycle plateau accounting retains those terminal candidate outcomes after a reloop. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: bind integrate patch benchmarks to live state Co-authored-by: Cursor <cursoragent@cursor.com> * feat: detect checkpoint precision from model path Co-authored-by: Cursor <cursoragent@cursor.com> * fix: preserve effective config and source layers Co-authored-by: Cursor <cursoragent@cursor.com> * fix: emit complete replay contracts Co-authored-by: Cursor <cursoragent@cursor.com> * fix: honor unset environments in current best Apply removals before layering winner environments so generated replay settings match the benchmarked configuration. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: checkpoint orchestration before scout reseed Force working-memory compaction before resetting the conversation so recent orchestration context survives scout updates. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
… gate, and coverage robustness (AMD-AGI#1021) * ci: expose explicit failure info for sharded test runs Add failure annotations, job summary, and shard/test-level failure listing for the sharded pytest matrix so that failed tests are surfaced directly in the CI UI instead of requiring log digging. No production code changes; workflow-only. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: address review feedback on shard-failure visibility - Upload pytest-output.log as an artifact on failure so the full test output is retrievable from the CI run - Explicitly handle the missing-log scenario to avoid failing the summary step when no log file is present - Add comments clarifying the summary ordering and the label assumptions used to attribute failures to shards Workflow-only change; no production code affected. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: address second-round review on shard-failure visibility - Cap per-test failure annotations (max 20) to avoid flooding the Actions UI when many tests fail in one shard - Warn when the tee'd pytest log write is truncated/fails instead of silently losing output - Use a set -e-safe arithmetic increment for the failure counter so the step is not aborted when the count starts at zero Workflow-only change; no production code affected. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: fix PIPESTATUS read-timing bug that failed passing shards The previous commit read PIPESTATUS[0] and PIPESTATUS[1] on separate lines. PIPESTATUS is reset by the next command, so by the time index 1 was read it was empty. Under set -e, the arithmetic test on an empty TEE_RC raised an "integer expression expected" error and aborted the step, marking shards that actually passed (e.g. 2441 passed, 0 failed) as failed and tripping the coverage gate. Snapshot the whole array in one statement immediately after the pipeline (rc=("PIPESTATUS[@]")) and guard both reads with a :-0 default so a short array can never feed an empty operand to the arithmetic test. Workflow-only change; no production code affected. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: drop tee exit-code check from the shard test step The disk-full hardening that inspected tee's exit code was the source of the earlier PIPESTATUS read-timing bug (it read a second PIPESTATUS element that had already been reset, which under set -e failed passing shards). A truncated tee write is an extremely low-probability event, and guarding it is not worth the added fragility. Simplify the step back to reading only pytest's own exit code (PYTEST_RC from PIPESTATUS[0], captured immediately after the pipeline) to drive the shard's pass/fail. All other failure-visibility behavior (per-shard annotations, aggregated job-summary table, precise failing- shard reporting, log artifact) is unchanged. Workflow-only change; no production code affected. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: verify shard completeness before combining coverage Fail-closed gate in the coverage job: read total_shards from pyproject.toml and assert every shard reported back (both .coverage.shard* and shard-status/*.outcome) before combine + fail_under. Prevents a partial suite (cancelled run, OOM/kill, or artifact upload failure) from passing the 90% gate with tests silently skipped. * ci: fail shard job on test failure so the matrix cell turns red The pytest step keeps continue-on-error so the always() reporting and artifact-upload steps still run on failure. A new final step in the test job then re-surfaces the shard's real result as the job exit code (exit 1 when steps.pytest.outcome != success), so a failing shard turns its matrix cell red instead of a deceptive green. Placed after all always() upload steps, so the coverage job still receives this shard's outcome + coverage data to aggregate. * ci: parse JUnit XML for shard failures; aggregate-before-gate for missing shards Two fixes to the sharded coverage job: - Failure extraction now parses each shard's JUnit XML instead of grepping the log, so fixture/collection ERRORs are captured (not just FAILED) and test stdout starting with "FAILED" is no longer mis-reported. pytest gets --junitxml and the report step reads <failure>/<error> per nodeid. - The completeness check is merged into an "Aggregate shard results" step that runs BEFORE any gate: it always renders the shard table (with a MISSING row) and sets tests_ok/complete/failed_shards/missing_shards on every branch. A single "Enforce shard results" gate then reds the job when tests failed OR a shard is missing, so the annotation always names the real shards instead of an empty "Failed shard(s):". Combine and fail_under are gated on complete && tests_ok so coverage is never computed on a partial or failed suite. * ci: precise shard-failure nodeids (-rfE summary) and keep coverage on test failure Two follow-up fixes to the sharded coverage job: - Failure extraction reads pytest's own "short test summary info" section (-rfE) instead of reconstructing a nodeid from JUnit's lossy dotted classname, so reported ids are the exact pytest nodeids (path/to/test_x.py::Cls::test) and ERRORs (fixture setup/teardown, collection) are captured alongside FAILEDs. Parsing is bounded to that section so unrelated stdout starting with FAILED/ERROR is not mis-read. - Combine shard coverage now only requires complete==true (all shards reported), not tests_ok. A failed assertion does not make the coverage data partial, so the combined total is still produced and shown (flagged as possibly misleading); fail_under stays gated on complete && tests_ok, and Enforce shard results still reds the job on any failure/incompleteness. * ci: recognize pytest's title-wrapped totals banner as summary section end The shard failure parser bounded the "short test summary info" section by a pure '='/'_' banner, but pytest's terminating totals line wraps a title (`==== 1 failed, N error in Xs ====`), so it was not recognized and the loop ran to EOF -- risking collection of a stray FAILED/ERROR line emitted by a plugin after the summary. is_banner now matches both banner forms (pure padding and title-wrapped) via regex, so the section ends at the totals line. Unit-tested: totals/ERROR-setup/header lines match; result lines don't; a post-banner FAILED line is no longer collected. * ci: skip e2e for workflow-only backport Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Zeng <lzeng@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Zeng, Leixin <296359813+zengleixin-amd@users.noreply.github.com>
…rkspaces - Replace per-run disposable worktrees with retained workspaces for post-mortem - Add process-group isolation with SIGTERM->SIGKILL escalation and full reaping - Recover last verified-best commit from campaign state on timeout - Simplify CLI contract: replace explicit flags with environment variables - Capture staged/unstaged diffs before forge for in-place restore verification - Generate unique branch names with UUID suffix to avoid collisions - Add comprehensive tests for lifecycle, timeout, process cleanup, and recovery Co-authored-by: Cursor <cursoragent@cursor.com>
…MEWORK outcomes across resumes, and mine verified launch recipes (AMD-AGI#1020) * fix(framework): preserve config lever channels Route specialist CLI arguments and environment overrides separately through integrate_patch so benchmark launches apply the intended server flags. Co-authored-by: Cursor <cursoragent@cursor.com> * test(framework): cover config lever routing Verify explicit server arguments remain separate from environment overrides through benchmarking and promotion. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(framework): reconcile outcomes across phase changes Record authored outcomes from task provenance, replace stale empty rows, and recover missed terminal results without fabricating empty deliveries. Co-authored-by: Cursor <cursoragent@cursor.com> * test(framework): cover cross-phase outcome recovery Protect provenance gating, stale empty replacement, persisted result recovery, and empty delivery recording after phase transitions. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(research-scout): add verified launch-recipe sites as a research source Teach the research-scout specialist to mine curated per-(model x hardware x quant x strategy) recipe pages (vLLM Recipes, LMSYS/SGLang cookbook) for verified serve flags/envs/benchmarks via WebFetch, emitting them as advisory proposal_set variants. Threads a recipe_sites field through SpecialistPromptInputs with a built-in default site list. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(research-scout): wire HYPERLOOM_RECIPE_SITES override into scout dispatch Populate the scout task's recipe_sites param from the HYPERLOOM_RECIPE_SITES env var (comma/space separated) so operators can point the scout at additional recipe sites without a code change; falls back to the built-in defaults when unset. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(grid): stop variant loop when the session budget runs out run_grid gained a session_deadline_sec argument: before each variant it checks the monotonic wall-clock deadline and marks the remaining variants skipped (error_class=session_time_exhausted) instead of running them, so a timeout halts the grid mid-way rather than draining every variant first. SharedState.grid_session_deadline_sec() derives that deadline from the remaining budget with a close-phase reserve, mirroring the conc_sweep gate. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(explore,sweep): enforce the session deadline across variants sweep forwards the session deadline into run_grid so a long conc/isl/osl grid stops when the budget expires. explore runs one variant per run_grid call, so it checks the same deadline at the top of its own variant loop and stops early; untested variants stay out of the ledger so a resume retries them. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(framework): stamp a terminal row when outcome recovery fails When an authoring or local-explore specialist returns an already-terminal task via its idempotency key but the outcome cannot be recovered from the bus, the pump previously logged a warning and returned without recording anything, so the candidate stayed unprocessed and the pump re-selected the same finished specialist every tick. Both paths now stamp a recovery_failed progress row so the candidate is marked processed and the pump advances. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(grid): skip a variant when the remaining budget cannot fit its worst case The grid deadline check now stops before a variant whose worst-case variant_timeout_sec no longer fits the remaining session budget, instead of only stopping once the deadline has already passed. This keeps the final variant from overrunning the close/report window and mirrors the conc_sweep per-variant estimate. explore applies the same guard in its own variant loop. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(research-scout): allow disabling recipe sites via a sentinel value Recipe-site guidance falls back to the built-in defaults when HYPERLOOM_RECIPE_SITES is unset, so the scout always received it. Operators can now set the variable to none/off/disable to turn recipe-site guidance off entirely. The PR notes are corrected to describe the actual default-fallback and the new disable path. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…-long-horizon-integration
…T in dotenv Add FORGE_PATH, KERNEL_AGENT_ROOT, and KERNEL_OPT_BACKEND_ORDER to DOTENV_EXACT_ALLOWLIST and KERNEL_AGENT_ENV_EXACT_ALLOWLIST so they are loaded from .env without warnings. Also add FORGE_ and KERNEL_OPT_ prefixes to DOTENV_PREFIX_ALLOWLIST for future forge/kernel-opt env vars. Co-authored-by: Cursor <cursoragent@cursor.com>
- _resolve_fp8_quant_type now accepts gpu_type and returns "bpreshuffle"
on gfx950 (MI355X/CDNA4) where aiter auto-upgrades blockscale kernels
to the bpreshuffle memory layout. Falls back to rocminfo detection when
gpu_type is "auto" or empty.
- _resolve_forge_server_log handles the nested benchmark directory layout
(runs/{phase}/{hash}/{warmup_round|measure_round}/{bench_dir}/server.log)
by searching sibling warmup_round dirs and using recursive glob in the
fallback path. Also adds "roofline" to the fallback phase list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemm): capture vLLM TunableOp shapes before tuning Record the real baseline workload when dense vLLM lacks shape inputs so Forge can tune instead of silently skipping, while preserving existing backend and explicit-input paths. * fix(gemm): capture block-FP8 runtime shapes Route vLLM block-FP8 workloads through profiled shape capture so Forge tunes the active AITER kernel path instead of skipping on empty TunableOp input. * fix(gemm): avoid TunableOp during block-FP8 profiling Keep profile capture scoped to profiler and AITER settings so unused TunableOp recording cannot add overhead or runtime side effects. * fix(gemm): harden shape capture controls Surface best-effort cleanup failures in debug logs and document and test the failure, multi-node, and operator-disable paths. --------- Co-authored-by: BaoYunkai <269084791+BaoYunkai@users.noreply.github.com>
…-long-horizon-integration
- Dispatch gate: _DEFAULT_HOT_KERNEL_MIN_GPU_PCT 3.0 -> 10.0 - KEEP gate: KEEP_THRESHOLD 1.05x -> 1.10x - Merge main: PR AMD-AGI#989 vLLM GEMM runtime shape capture Co-authored-by: Cursor <cursoragent@cursor.com>
…nalization - Cache _is_gfx950 rocminfo detection with lru_cache (one subprocess call per session instead of per-invocation) - _find_server_log_near: select newest server.log by mtime when multiple benchmark dirs exist under warmup_round, matching fallback glob behavior - Wrap _finalize_forge_workspace in try/except so inplace restore failure is logged but does not swallow the forge result dict Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Forge Fusion independently gated so disabling or completing GEMM tuning cannot bypass eligible fusion work. Co-authored-by: BaoYunkai <269084791+BaoYunkai@users.noreply.github.com>
_is_vllm_block_fp8 was missing "bpreshuffle" and "a8w8_bpreshuffle" in its quant_type match set. When _resolve_fp8_quant_type returns "bpreshuffle" on MI355X, _forge_framework_for_vllm fell through to plain "vllm" instead of "vllm-aiter", routing to TunableOp (hipblaslt) instead of the aiter CK tuner that matches the actual hot kernel. Co-authored-by: Cursor <cursoragent@cursor.com>
…I#1017) * fix(kernel): delegate Forge driver fallback to task preparer Keep Forge attempts alive when driver generation is unavailable or compile-only so the loop can build a valid measurement harness before optimization. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(kernel): persist Forge invocation specifications Capture per-operator runtime, model, source, and test evidence so Forge can rebuild valid drivers from an explicit contract instead of heuristic prompts. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): preserve grouped candidates across Forge dispatch Keep task-group workload cases intact across subprocess boundaries so one Forge run prepares and validates all shapes for a shared source function. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): validate grouped operators across all shapes Preserve exact invocation cases through Forge and scope task accounting by stable identities so grouped retries and integration cannot rotate or stall. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): recover validated Forge best after timeout Treat Forge timeout as recoverable when a validated checkpoint exists, exporting only the committed best so unfinished working-tree candidates cannot replace or discard proven improvements. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): recover E2E from stale AITER locks Safely sweep both AITER build trees and retry integration once when server logs prove a stale baton wait. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): resolve CodeQL findings Make fail-soft paths explicit and remove unreachable grouped-task logic so static analysis can verify the intended behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): stabilize task identity and KEEP integration Decouple persisted kernel tasks and pending patches from trace-local ordinals so route changes and reranking cannot lose validated artifacts. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…MD-AGI#1025) * fix(magpie-patcher): trust MI355X SGLang clients Patch both remote-direct and local-server paths in sglang_mi355x.sh so custom-tokenizer models such as Kimi receive --trust-remote-code. Keep MI300X behavior intact and add layout, idempotency, and drift tests for the MI355X script. * chore(deps): update InferenceX pin to latest main Keep installer and CLI preflight on the same reproducible upstream revision. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(magpie-patcher): extend SGLang client trust patch to mi300x local path The client trust patch only reached sglang_mi355x.sh (mi355x patcher) and sglang_mi300x.sh's remote-direct path; mi300x's local-server client path was never gated. Both scripts share byte-identical client blocks, so generalise the mi355x client-trust patcher into a script-agnostic _apply_sglang_client_trust_patch_atomic applied to both scripts. The local splice runs only when a local-server path marker is present and reports drift there as a failure, so remote-only/reduced layouts skip cleanly. Builds on meina's b97a2d6. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(breakdown): add author-time v4 operation model (AMD-AGI#1008) Capture canonical session facts through the SBD SDK at execution time, preserving legacy projections without reconstructing optimization semantics from session artifacts. Co-authored-by: haiskong-bot <haiskong-bot@local> --------- Co-authored-by: meinali-566 <289551133+meinali-566@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: fengshaoyi <shafeng@amd.com> Co-authored-by: haishuok0525 <haiskong@amd.com> Co-authored-by: haiskong-bot <haiskong-bot@local>
Two rough edges now that six dispatch sites and the explore rebind funnel through one projection: - The extra_envs normalizer called .items() on whatever current_best held, so a resumed non-dict raised inside the funnel rather than reading as no config. The guard came out in 7fb6119 as speculative; with a single shared funnel it is the same tolerance to_str_list already gives the other four fields, so restore it as a named normalizer -- inlining it costs 119 columns. - Export first_positive_tput. The explore rebind hand-rolled the same tput/output_throughput lookup with a bare float(), which raises on a non-numeric tput that this function is already written to tolerate. Co-authored-by: Cursor <cursoragent@cursor.com>
Routing _enqueue_internal_sweep_task through the shared helper widened its params from base_extra_args alone to the full set, and _build_grid assembles variants with args_mode="replace" when current_best carries it. That is the intended reading -- under replace, current_best's args are the complete effective list -- but it changes what the sweep launches, so assert it rather than leave it implied. Co-authored-by: Cursor <cursoragent@cursor.com>
…, add packaging CI setuptools ignores a package-data glob that matches nothing, so a deleted file leaves an invisible dead declaration behind. Three packaging defects had accumulated this way, none of which any existing job could catch. Fixes: * Drop `hyperloom.agents.critic` `kb_contributions/*.json`. The only file it ever matched was removed in AMD-AGI#922; the glob has matched nothing since. * Exclude test packages from distribution. `packages.find` defaults to `namespaces = true`, so all 12 test dirs were discovered and shipped: 524 of 1087 wheel entries (7.22 of 17.27 MB uncompressed). They could never run either, because none of their 9 fixture data files are package-data. Verified against an installed wheel: `test_slugify.py` ships but its `fixtures/slugify-golden.json` does not. Wheel is now 2.97 MB (was 4.74 MB). No production module imports a test package. * Migrate `project.license` to the PEP 639 SPDX form. The TOML-table form is deprecated and becomes a hard error on 2027-Feb-18; setuptools 83 already warns. This raises the build-system floor to setuptools>=77. Guards: * `test_packaging_lint.py` checks every path-bearing declaration against the working tree: package-data globs, data-files sources, console-script targets, test-package exclusion, ruff per-file-ignores, and the license form. Follows the existing `test_common_import_lint.py` lint-as-test precedent and is stdlib-only, so it needs no project extras. * `packaging.yml` runs that lint and asserts a real wheel's contents. It has no `paths-ignore` on purpose: eight package-data globs point at `*.md`, and a documentation-only PR is both the change most likely to strand a declaration and the change `lint.yml` / `tests-coverage.yml` skip. * `check_wheel_contents.py` derives its expectations from pyproject.toml rather than a hardcoded list, so it also catches the inverse defect (a declared asset missing from the wheel). Note: `scripts/build_wheel.sh` builds with `--no-build-isolation`, so it now needs setuptools>=77 in the ambient environment. It fails loudly at config validation if not. `release-provenance.yml` uses build isolation and resolves the floor automatically. Co-authored-by: Cursor <cursoragent@cursor.com>
`_export_workload_envs_for_optimize` writes TP/CONC/EP straight into
`os.environ`, and `monkeypatch.delenv(key, raising=False)` records nothing to
restore when the key was absent, so `test_multi_node_always_exports_workload_envs`
left `EP=2` behind for the rest of the pytest worker.
`materialize_config_with_envs` reads `os.environ.get("EP")` and appends
`--enable-expert-parallel` when EP > 1, which corrupted a sibling JSON-valued
flag in `test_json_serve_arg_survives_shape_capture_port_removal`:
json.loads('{"cudagraph_mode":"FULL"} --enable-expert-parallel')
JSONDecodeError: Extra data: line 1 column 27 (char 26)
The leak was masked whenever a test exporting `EP=1` happened to run earlier in
the same process, because `delenv` then had a value to restore. Only a specific
ordering exposes it, so it stayed latent until a shard boundary moved and put
both tests in one worker.
Restore the three keys with an autouse fixture, which plugs both export sites
and makes the module order-independent. Verified by an env probe: without the
fixture it reports `EP leaked as '2'`, with it the environment is clean.
Co-authored-by: Cursor <cursoragent@cursor.com>
…ed ones The existing checks only ran declaration -> tree, so an asset added without a matching `package-data` glob was silently left out of the wheel. Auditing that direction found 11 undeclared files under `src/`, three of which had to ship: * `assets/agentx/aiperf_client.sh`. `agentx/deploy.py` copies it into InferenceX `benchmarks/` at runtime, so `HYPERLOOM_AGENTX=1` raised `FileNotFoundError: AgentX asset missing from package` on every wheel install. `assets/*.sh` only matches the top level of `assets/`, and the sibling `map_aiperf.py` shipped anyway because namespace discovery picks up any directory holding a `.py`, which hid the gap. * `assets/slurm/*.sh.in`. The shipped `run_hyperloom.sbatch` copies `_incontainer.sh.in` into the container at launch; the slurm globs covered `.sh`/`.sbatch`/`.tsv`/`.template` but not `.sh.in`. * `breakdown/*.md`. The only one of the ten `SKILL.md` files under `src/` that did not ship. The remaining eight are justified in `_UNPACKAGED_ASSETS`: `.gitignore` files, `.ci-deferred/`, and `assets/quick-start/` (a container image build context whose Dockerfile clones the repo and whose scripts hardcode `/opt/Hyperloom`). Guards: * `test_no_undeclared_assets_under_src` walks `git ls-files src` and fails on any non-Python file no `package-data`/`data-files` entry ships, so a new asset cannot be added without a decision. Skips when git is unavailable. * Broaden the `packages.find` exclude to the singular/gerund spellings. A directory named `test/` or `testing/` was previously packaged, and the exclusion test now checks all three names so the two cannot drift apart. Co-authored-by: Cursor <cursoragent@cursor.com>
…re publish Excluding the test trees from `packages.find` is not sufficient on its own. `include-package-data` makes setuptools re-attach a directory it excluded from the package list as package data of the nearest ancestor package, taking the file list from `SOURCES.txt`. `pip wheel` builds in-tree and leaves `build/` and `src/*.egg-info/` behind, so a second build -- or a build after switching branches -- silently ships the test trees again. Reproduced on this branch with an egg-info generated on `main`: 1091 wheel entries, 524 of them tests, identical to an unpatched tree. CI stays green only because a fresh checkout has no egg-info to inherit. `scripts/build_wheel.sh` is the path that produces the published release asset, so it now: * Removes `build/` and `src/*.egg-info/` before building. With the same stale egg-info the wheel is 567 entries and 0 tests, byte-identical to a clean build. * Runs `check_wheel_contents.py` on the artifact before `--publish` can upload it, so a later regression in the cleanup cannot reach a release. Verified with a stub checker: the script aborts with exit 1 before calling `gh`. The verification pre-flights the TOML reader, because the checker needs tomllib (python>=3.11) or tomli and the canonical `/opt/venv` may be 3.10. Without it a missing module would surface as a traceback plus a vague check failure. Co-authored-by: Cursor <cursoragent@cursor.com>
…erns `_check_no_test_packages` matched only `tests`, while `packages.find` excludes `test` and `testing` too and `test_packaging_lint.py` checks all three. A directory named `test/` or `testing/` reaching the wheel would therefore pass the artifact check while failing the declaration lint -- the exact drift this module avoids elsewhere by deriving its expectations from `pyproject.toml`. Derive the names from the exclude patterns instead of hardcoding a second list, so the check cannot narrow while the exclude list widens. An empty exclude is now reported rather than silently disabling the check. Verified against the previous version: a `test/` or `testing/` entry was MISSED before and is CAUGHT after; a real wheel still reports 567 entries with no test packages. Co-authored-by: Cursor <cursoragent@cursor.com>
fix(packaging): stop shipping test trees, drop dead package-data glob, add packaging CI
[FIX] make the grading anchor and the config it was measured on inseparable
…ectification (AMD-AGI#1093) * feat(explore): surface real server crash cause into VariantResult Add server_log_path field to VariantResult. Import server_log_death_excerpt in _grid_runner and use it at SERVER_DEAD, no_benchmark_workspace, invalid- measurement, magpie_timeout, detokenizer_stall, killed_overtime, warmup timeout, and warmup_round_failed sites so the real traceback excerpt (e.g. 'mla_gluon requires batch_size=1') replaces the hardcoded generic string. Export server_log_death_excerpt from _subprocess_kill __all__. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(explore): stop shredding failure context in transit Widen reason truncation in explore executor from 256 to 1200 chars. Add error_class and server_log_path to tested_update, rejected_update, warmup tested_update, and per_variant_outcomes rows. Add variant_name to _common_result_fields so downstream consumers can discriminate per- variant failures. Raise _truncate_excerpt limit from 800 to 1200 chars. Raise _DEFAULT_LAST_FAILURES from 10 to 30. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(render): expose multi-line failure excerpts and log paths in prompt Replace splitlines()[0][:200] with the last 600 chars of the excerpt in _format_last_action_failures, show the last 10 failures instead of 3, and emit variant_name and stderr_log_path (log=) fields so Orchestration can Read the log on demand. Add error_class and reason to _format_variant_line so crashed variants are distinguishable from measured non-wins. Widen the rejected window in _format_search_state from 5 to 15 entries. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(gaps): make per-variant failures first-class, stop gap collapse Add _record_explore_variant_failures called from the dispatcher explore hook to record each FAILED per_variant_outcomes row into last_action_failures with variant_name, error_class, reason, workspace and server_log_path. Key _extract_gaps_from_attempts on (action, error_class, variant_name) so different crash causes no longer collapse into the same gap; build symptom from the first meaningful excerpt line instead of the generic error_class. Fall back to (action, error_class) key when variant_name is empty to preserve existing behaviour for non-explore failures. Pass reason and error_class through append_gap_attempt in _record_explore_round_gaps. Co-authored-by: Cursor <cursoragent@cursor.com> * test(explore): update locked assertions and add new failure-context coverage Update test_record_action_attempt_failed_truncates_error_excerpt and test_record_action_failure_truncates_excerpt_and_tails_subprocess to expect 1200 chars (raised from 800). Update test_to_prompt_summary_shows_last_three_ failures_with_suffix for the new 10-entry window. Update test_format_search_ state in both renderer test files to expect 'rejected (last 15)'. Update test_extract_gaps_from_attempts for per-variant gap separation; add test_extract_gaps_no_variant_collapses (backward compat) and test_extract_gaps_symptom_uses_excerpt. Add test_server_dead_surfaces_log_excerpt to verify mla_gluon excerpt flows from a seeded server.log into VariantResult. Add renderer tests for multi-line excerpt, log= path, error_class/reason in variant line, and +N earlier suffix. Add test_record_action_failure_stores_variant_name and test_record_action_failure_caps_at_new_default (cap=30). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(explore): dedupe server.log resolution and fix stale docstrings Collapse six duplicated `slot / "server.log"` locals into one per-iteration local plus a `_existing_log_path` helper, replacing seven inline exists() ternaries. Fold the death excerpt into no_benchmark_workspace's summary before the abort marker is written so abort_reason.json and state agree. Render the failure blob from stderr_tail (the actionable end) rather than the tail of the head-sliced error_excerpt, which showed the least useful middle. Hoist the render window and excerpt budget to module constants. Drop a dead symptom branch in _extract_gaps_from_attempts (an already-stripped excerpt always yields a non-empty first line) and the redundant slice over an already-capped last_action_failures. Drop redundant getattr defaults and the unreachable empty-list guard. Update stale docstrings: server_log_death_excerpt now also serves explore, and _common_result_fields / record_action_attempt gained variant_name. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(explore): remove cross-round historical eligibility gates Delete gain_unlockable, _is_blocked, seen_fps, and unlocked_reference from ExploreExecutor. Delete the ledger_dup skip branch. Only same-payload duplicates within one submitted grid (unique_in_round / round_dup) are suppressed; all historical tested/rejected/accepted fingerprints may now re-enter future grids. Simplify _framework_config_new_variants to return all valid dicts without consulting explore_search["tested"]; the tested ledger remains as context. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(explore): safe accepted reruns; remove dead _entry_fp/_entry_control_fields Remove accepted_fps_now filter so a failed/reverted rerun of an accepted fingerprint is recorded in rejected as current evidence. The accepted record retains the previously applied configuration. Extend _lift_to_current_best with fingerprint-based stack membership check alongside the existing (action, variant_name) guard; a renamed duplicate fingerprint cannot append a second optimization_stack entry. Remove the now-unused _entry_fp inner function and the _entry_control_fields module-level function. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(explore): align orchestration guidance with no-blacklist behavior Update acceptance-threshold advisory to show prior results as evidence rather than blocked/unblocked eligibility signals. Drop "KEEP'd variants stay de-duped permanently" language from the cycle strategy advisory. Clarify RULE F3 applies to baseline, not explore. Update EXPLORE grid hint and IDEA GENERATION section to state only same-grid duplicates are collapsed and any fingerprint may be re-proposed. Co-authored-by: Cursor <cursoragent@cursor.com> * test(explore): update and add coverage for blacklist removal Rewrite test_explore_executor_dedups_against_ledger as test_explore_executor_historical_fingerprint_reruns: historical REVERT now runs; only same-grid round_dup entries are collapsed. Add test_explore_executor_historical_failed_and_accepted_rerun: FAILED and accepted fingerprints may rerun; latest result overwrites the ledger entry; no ledger_dup in skipped_dup. Rename test_new_variants_filters_already_tested to test_new_variants_retains_historical_tested_fingerprints: both variants run. Update test_hold_generating_finishes_when_no_new_candidates to use an empty pending grid (the only remaining termination condition). Update test_acceptance_threshold_advisory_lists_unblocked: all entries now appear as evidence regardless of KEEP/REVERT outcome. Add test_lift_does_not_double_append_same_fingerprint and test_lift_at_or_below_anchor_does_not_modify_stack for the fingerprint-based stack dedup in _lift_to_current_best. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(explore): delete no-op new_variants wrapper and fix stale docs _framework_config_new_variants degenerated into an isinstance(dict) filter that can never fire: _build_framework_config_grid only ever appends literal dicts and already skips non-dicts. Delete it, inline both call sites, and drop its delegation-registry entry. Lane termination is unaffected; the max_rounds cap is the remaining bound. Collapse the writeback stack guard from two full set builds into the existing name set plus a short-circuiting fingerprint scan, and document the dedup keys on _lift_to_current_best. Fix stale docs that still described the removed blacklist: the explore module docstring, two section comments, the proposals ledger-injection comment, and specialist.md. Drop the duplicated "evidence not a gate" sentence from the cycle-strategy advisory (the acceptance-bar block states it with the threshold) and tighten the RULE F3 and reject-triage prompt text. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(gate): deny LLM baseline while an enablement round is in flight Add the enablement_round_in_flight rule to _validate_baseline_singleton so the Orchestration LLM cannot dispatch a new baseline while an enablement specialist is authoring a patch. The check fires before the baseline_phase_singleton guard, keeping both rules distinguishable in policy_denials. The revalidation path (Coordinator-internal, bypasses intent_router) is unaffected. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(state): collapse 34 enablement_* fields into EnablementRound Move all per-round enablement fields from SharedState top-level into a nested EnablementRound dataclass at SharedState.enablement. enablement_mode remains top-level as a session-scoped CLI admission setting. Add EnablementRound in state/_shared_state/enablement_round.py with from_dict and to_dict helpers. Bump LATEST_STATE_SCHEMA_VERSION to 4 and add a migration that lifts flat enablement_* keys from v3 state.json into the nested object. Update all non-test call sites across framework.py, writeback.py, machine_state.py, build_lifecycle.py, and gate.py. Add a nested-then-flat reader (_eg) to the breakdown collector so old and new state.json both export identical breakdown keys. Add v3→v4 and v4 round-trip migration tests. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(state): derive in-flight from task registry, delete watchdog Remove the stored dispatched bool and dispatch_tick from EnablementRound. In-flight status is now derived by querying the task registry: add _enablement_in_flight() which checks whether inflight_task_id maps to a queued or running task. The watchdog (_enablement_round_silently_finished plus _ENABLEMENT_WATCHDOG_GRACE_TICKS / _ENABLEMENT_WATCHDOG_HARD_TICKS) is deleted; its stall-accounting role is preserved by checking the registry directly and calling _maybe_rearm_enablement on a silently-finished round. Simplify _maybe_rearm_enablement to always clear inflight_task_id at the end of a rearm (no conditional guard). Drop the redundant dispatched term from enablement_engaged (inflight_task_id covers the same signal). Source the breakdown's dispatched key from bool(inflight_task_id). Register _enablement_in_flight in the coordinator delegation map; drop the deleted _enablement_round_silently_finished entry. Update gate.py to check inflight_task_id directly for the enablement_round_in_flight rule. Rewrite the two watchdog tests against task-registry semantics and add a get() method to _FakeTasks. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(enablement): tighten code after objectification Replace 49 defensive getattr(state.enablement, "X", default) calls with direct attribute access now that EnablementRound is a dataclass with guaranteed fields. Use EnablementRound() in both test fixtures instead of types.SimpleNamespace, which lacked all fields. Drop EnablementRound.to_dict() (dead; SharedState.to_dict uses asdict which recurses automatically) and its unused asdict import. Remove fake.state = SimpleNamespace(pending_proposals=...) from the _enqueue_self fixture (only the deleted watchdog used it). Fix stale enablement_dispatched references in docstrings and comments across framework.py, explore.py, intent_router.py, and four test files. Remove the dispatch_tick field from breakdown/schema.py and its emission from sessions.py (field was deleted in Stage 3 and always returned -1). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(test): use complete EnablementRound in parameter-builder fixtures Replace partial SimpleNamespace enablement stubs with EnablementRound in provision and targeted-build routing tests. Production code now directly reads required round fields such as origin, so the partial stubs no longer represent valid state. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(enablement): treat a pending integrate as part of the in-flight round _enablement_in_flight only checked the authoring specialist task state, but the specialist goes terminal a full tick before the Critic sees the integrate proposal that consumes its deliverable. On that same tick _pump_enablement_safely therefore saw a settled round, bumped the stall streak with a synthetic "round_finished_without_rearm" revert, cleared inflight_task_id, and dispatched a second specialist against the stale launch_log while the first round's integrate was still awaiting review. That breaks the serial patch chain, can reach enablement_stalled early, and drops the gate's baseline race protection. A round now stays in flight while an undecided integrate_patch proposal or a queued/running integrate_patch task carries its specialist_task_id. Keying on PendingProposal.decided bounds the deferral without reintroducing the tick counters: approve materialises the task, reject rearms directly. Add coverage for the deferral, the decided-proposal bound, the approved-task case, and a regression test asserting no false stall and no second dispatch while the proposal is unreviewed. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(enablement): tighten in-flight check and its tests Update the _maybe_enqueue_enablement_specialist guard docstring and the reconcile comment, which still described the round as just the specialist task after the check grew to cover the integrate. Drop the redundant `or {}` on PendingProposal.payload and Task.params; both are required dict fields (params is NOT NULL at the schema layer). Compress the in-flight docstring and comment to state the contract rather than narrate the timing bug. Remove the unused tmp_path parameters and a dead _FakeTasks reassignment left over from the filesystem-based watchdog, hoist PendingProposal to the existing top-level import, and drop redundant empty _FakeTasks constructions that the fixture already provides. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…AGI#1098) * fix(kernel): deploy canonical Forge artifacts for E2E validation Consume Forge publication bundles with safe path resolution and fail closed instead of benchmarking an unrelated primary-source fallback. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): defer accuracy until throughput passes Avoid spending full eval budgets on regressing kernel candidates by measuring hot throughput first and reusing the server for accuracy only after the performance gate succeeds. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): make timeout fallbacks observable Log malformed rebaseline timeout inputs and rely on the executor's timeout contract without expanding the existing import cycle. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): bind Forge targets to runtime implementation Resolve dispatch kernels to the implementation that actually ran and fail closed when source ownership or Forge edits do not match the observed runtime symbol. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): allow multi-file Forge patches and cap integrate timeout Let validated Forge patches modify supporting implementation files without semantic target rejection, while bounding inherited integrate benchmark timeouts to one hour. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): respect benchmark timeout and expose canonical fallbacks Honor the benchmark timeout contract, preserve runtime backend provenance on all dispatch outcomes, and make canonical artifact fallback reasons observable. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(kernel): preserve single-round accuracy and contain Forge files Keep eval enabled when lifecycle staging is unavailable, and reject canonical files directories that resolve outside the Forge campaign root. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* Doc clean up for crisp user experience * Minor fixes and clean-up
…#1099) * fix(kernel): parse current TraceLens input_shapes so GEMM tuning sees real M _extract_gemm_shapes_from_candidates only understood the legacy TraceLens rendering, where one input_shapes entry carried every tensor joined by "<br>". Current TraceLens emits one entry per tensor ({"call_num": N, "shape": "(M,K) fp8"}), so every entry hit the "len(parts) < 2" guard and the extractor returned no shapes at all (measured on a real session: 22 GEMM kernels, 88 entries, 0 shapes). That empty result is load-bearing: _vllm_block_fp8_profile_capture_required only runs the block-FP8 profile capture when shapes_json is empty, so the lossy capture became the shape source and recorded a single large prefill M (2095). GEMM tuning then optimized an operating point the workload barely uses: the micro benchmark won 2.89-3.29x while the throughput-dominant small-M decode GEMMs regressed, for -18.45% E2E and an immediate revert. Parse the per-tensor layout (A = entry[0] "(M,K)", B = entry[1], oriented by matching the contracted dim K) while keeping the legacy "<br>" path byte identical. Order the emitted shapes by observed call count so the decode shapes - invoked ~30x more often than prefill here - are tuned first when the tuner runs out of budget, and drop N==1 matrix-vector heads, which are not tunable GEMM tiles yet would otherwise sort first on call count. On the session that regressed this turns 0 extracted shapes into 19 real ones covering M=64 (decode) plus M=1084/3126 (prefill), with all six decode shapes ordered first; the lossy profile capture no longer runs at all. * fix(kernel): scope traced GEMM shapes to the precision being tuned A TraceLens capture records every GEMM dtype the model runs. On Qwen3.5-122B-A10B-FP8 that is 15 FP8 projections (aiter::gemm_a8w8_blockscale_ck) plus 4 BF16 router/head shapes (vllm::rocm_unquantized_gemm), and the extractor emitted all 19 regardless of which tuner would consume them. Feeding BF16 shapes to an FP8 tuner is not a miscompile - aiter looks the config up by (M,N,K) at serve time, so those rows are simply never hit - but it burns tuning budget and, worse, distorts the call-count ordering: the BF16 head is invoked 1440 times and sorted ahead of the real FP8 decode shapes it displaces. Thread the tuning precision through _resolve_forge_shapes into the extractor and keep only shapes whose traced dtype matches. Both FP8 and BF16 dense tuners consume this file, so the filter is parameterised rather than hardcoded, and an empty precision keeps every dtype (existing behaviour and callers unchanged). On the real capture: 19 shapes -> 15 with precision="fp8" (4 with "bf16"), and the four highest-ranked entries are now all FP8 decode shapes. * fix(kernel): let vLLM block-FP8 tune the traced decode shapes The block-FP8 route resolves its shapes from Kineto *op* events. Decode steps replay inside a CUDA Graph and emit no such events, so that route structurally reports prefill M only. Measured on the capture behind the -18.45% E2E regression: the decode-only trace split yields zero block-FP8 events, while both prefill-bearing splits yield M=2095 - the single shape that was then tuned. TraceLens candidates are built from the device kernel timeline, which does see through the graph: the same session recorded the decode shapes at M=64 with ~30x the call count of prefill. Consult them before falling back to the profile capture. require_fresh_profile preserves the vLLM rule that shapes must be workload-matched, and the precision scoping added in the previous commit keeps BF16 heads out of an FP8 tuner's input - the two guards that make this source trustworthy for vLLM. The gate stays surgical: only the block-FP8 branch consults the traced shapes, so the vLLM TunableOp recording pass (which needs tunableop_input, a different artifact, and is likewise short-circuited by a non-empty shapes_json) keeps its current behaviour. Adds a handler-level regression test asserting the capture pass no longer runs and the tuned shapes carry the decode M; it fails without this change. * fix(kernel): close the precision-scoping holes in traced GEMM shape resolution Artifacts bypassed scoping (high). precision only reached the candidate extraction, so a pre-rendered shapes artifact -- last_trace_analyze.shapes_json, artifact_paths.gemm_shapes_json, or the sidecar shapes.json -- was returned first whenever it merely parsed. With correct FP8 candidates and a stale BF16 artifact both present, the BF16 shapes won. Those artifacts are bare [{M,N,K}] lists carrying no dtype or provenance, so they cannot be checked against the target precision; when scoping is requested the extraction now runs first and artifacts stay the fallback for the unscoped case. Exact-string dtype matching (high). The precision and the traced token spell the same dtype differently, and the comparison demanded equality: precision=fp8 rejected fp8_e4m3 / e4m3fnuz, precision=fp16 rejected the f16 that this repo's own _TRACE_DTYPE_SUFFIX emits, and FP4/MXFP4 produced no token at all so scoping silently did nothing. Both sides are now folded onto a canonical family (fp8 / bf16 / fp16 / fp4) covering those aliases, and the token regex accepts dotted spellings so torch.float8_e4m3fn is captured whole. Duplicate shapes kept the first weight (medium). A repeated (M,N,K) discarded later sightings, so a call count of 10 outranked a later 1000 for the same shape and pushed the real decode hotspot down the list. Weights are now merged with max() before ordering. Legacy separator (medium). Only lowercase "<br>" split; "<br/>" and "<BR/>", both of which appear elsewhere in the tree, fell through and yielded no shapes. Splitting is case-insensitive and tolerates the self-closing forms. Adds cover for the artifact-vs-candidates precision conflict (and that an unscoped call still prefers the artifact), the dtype aliases above, max-weight merging, and every separator spelling. 260 passed. --------- Co-authored-by: xiaofei-zheng <XiaoFei.Zheng@amd.com>
…9000 Pro support
Blocker fixes (per reviewer checklist 5168506779):
- Fix B: add rx9070xt, rx9070, rx9060xt, r9000 to --gpu-type choices (parser.py)
so the CLI accepts the RDNA4/GFX12 SKUs.
- Fix C2: populate _AMD_GPU_DISPATCH_IDENTITIES for all four
RX 9000-series SKUs (gfx1201/gfx1203/gfx1206/gfx1207) with AMD-verified
CU counts (9070xt=64, r9000=64, 9070=56, 9060xt=32) plus the R9000 Pro
(gfx1203) — previously returned None for every RDNA4 GPU type.
- Fix C4: rewrite the component compatibility grid-table as a robust
list-table, add the RX 9070 XT / RX 9070 / RX 9060 XT / R9000 Pro SKUs
to the GEAK and Magpie component rows, and note the RDNA4 ROCm 7.3+
Windows requirement. Fixes the broken border formatting (trailing-space
+ inline separator misalignment) that caused RST warnings.
- Fix E: add the Windows hipConfig --show-device probe path (was
Linux/rocm-smi-only) with tags for bare 'RX 9070'/'RX 9060' (no ' XT'
suffix) and 'R9000' so the bare non-XT Pro SKU autodetects correctly,
plus a regex fallback for full product strings ('AMD Radeon RX 9070',
'AMD Radeon Pro R9000'). Also normalise mi308x/mi325x to the mi300x
Magpie runner label in _gpu_runner_type (was inconsistent).
- Fix F: split the single shared _RDNA4_PEAK_TFLOPS_RXP table into
per-SKU peak and achievable tables scaled by AMD's CU-count x game-clock
ratios, so FP32/FP16/FP8 ceilings are distinct and correctly ranked
9070xt > 9000xt > 9070 > 9060xt. The R9000 Pro (gfx1203, 64 CU) shares
the RX 9070 XT die and gets the same peak with 32 GB GDDR6.
Notes:
- 'R9000' = Radeon Pro R9000 (workstation AI card, Navi 48 / gfx1203,
64 CU, 32 GB) — NOT a consumer 'RX 9000 XT' (no such product exists).
- Fix A (duplicate def in test_coverage_margin3_unit.py) is already
resolved in upstream main; no action needed here.
- README already ends with a trailing newline (verified).
Signed-off-by: maxritz <the.nair@outlook.com>
Author
|
Summary All reviewer blocker items are now resolved in commit 7a0b4c0, which has been force-pushed to Maxritz/Hyperloom-RDNA:main. The fork's broken empty-tree merge (41b0323) has been replaced with a clean chain: upstream 4cfe232 → fix commit 7a0b4c0. Reviewer Blockers — All Resolved
Key Naming Correction rx9000xt → r9000 everywhere. Per Wikipedia's "Radeon Pro R9000" article, gfx1203 maps to the Radeon Pro R9000 workstation AI card (Navi 48 die, 64 CU, 32GB GDDR6) — the same die as the RX 9070 XT but with double the VRAM. There is no consumer "RX 9000 XT" SKU. Smoke-Tested Verification mi300x (gfx942, 304 CU) — mi308x/mi325x collapse to this runner R9000 shares the RX 9070 XT's die — same peak (14.6 FP32) and sustained (11.0 FP32), but 32GB GDDR6 vs 16GB. |
Collaborator
|
Thanks for your contribution! Please resolve conflict, we are about to merge this pr. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # MI308X/MI325X are mapped to the MI300X runner by _gpu_runner_type(); every | ||
| # RDNA4 SKU resolves to itself. The matching rx9xxx runner scripts are shipped | ||
| # by the AMD-AGI/Magpie package (see docs/components/magpie.md). | ||
| _SHIPPED_MAGPIE_RUNNERS: frozenset[str] = frozenset({ |
Collaborator
|
Two blockers from a code read. 1. Branch is 699 commits behind merge-base is 2. Linux RDNA4 detection never matches The Linux branch matches space-free tags against for tag in ("MI355X", "MI325X", "MI308X", "MI300X", "RX9070XT", "RX9070", "RX9060XT", "R9000"):
The Windows branch already uses spaced tags ( Suggest one normalizing matcher shared by both platforms (strip spaces, lowercase, then look up a single table). That also removes the duplicated tag lists and the |
xiaofei-zheng
requested review from
Ahmedhasssan-aig,
devalshahamd and
tsrikris
as code owners
August 25, 2026 10:22
Collaborator
|
@Maxritz I would urge you to discuss this with module owners. TraceLens support for RDNA is still in discussions so pushing in such a large feature will bloat the codebase meanwhile. @xiaofei-zheng I would hold off against merging this code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…or Windows ROCm 7.3+
Windows hipConfig autodetection in gpu_types.py
RDNA4 HW specs (16GB GDDR6, 640 GB/s BW, WMMA FP8) in roofline_ceiling.py
Optional fcntl import for Windows compatibility in local_store.py
hasattr(os, 'getuid') guard for Windows in state_paths.py
POSIX-only test guards in test files
Update README and compatibility.rst with RDNA4 GPU table
Description: what and why
Linked issue(s): close/fix refs
Tests: added/updated? commands run?
Breaking changes: yes/no (details if yes)