[draft]Dev.yihou.aiopt.task package.concat - #157
Draft
dorado269 wants to merge 852 commits into
Draft
Conversation
…ee instances across two people checkpoint's two plus my amplification of the second. With theirs alone it reads as one person being careless; with mine it is a bias with a direction. Also the audit form of an existing rule: a forward-only check against an absence-defect reports the file as sound rather than staying silent. Signed-off-by: yihou <yihou@amd.com>
…and leaves it empty Corrected against a real operator_identity, run 20260905T074905-9ec798 on 287, which had been sealed twenty minutes when I filed the wrong cause. T69 said 'the tags identify writes are bare language names', inferred from one workset's Definitions. identify.py:547 writes a SUFFIXED fellow and scaffold.py:264 copies it into the Definition tags, so the original _fellow matching '*-fellow' was correct by design. What actually happens: identify resolves the language for 2 of 5 operators and leaves the other three empty (lang='unknown'), and scaffold's list comprehension drops the empty value. The fallback fired because the value was ABSENT, not because the spelling disagreed. The 'two producers disagreed' half stands, with different producers: 91ea967b's bare-language tags are not scaffold's shape, and build_workset is kind: ai -- the agent wrote them. scaffold suffixed, agent bare, nothing reconciles. Third vocabulary, and the origin of the bad name: mock_adapt.py:451 hard-codes tags including the literal 'generic-fellow'. The mock corpus is where that string comes from. 62032fc/f92e42b stand and are better than the wrong diagnosis deserved -- they accept both vocabularies and refuse on absence, so they were correct under either cause, which is why the wrong one survived a landing. Revised close: find why identify reports lang='unknown' for two Triton files it resolved to a path; one vocabulary across identify/scaffold/agent/mock_adapt; delete generic-fellow from mock_adapt. Signed-off-by: yihou <yihou@amd.com>
…real time is 08:19Z Signed-off-by: yihou <yihou@amd.com>
…ted in the record that says what is true Signed-off-by: yihou <yihou@amd.com>
checkpoint wrote nine last-verified stamps by extrapolation, none read, one 66 minutes into the future -- while adopting the dating convention in that same file's header. Their own diagnosis is the generalisation: nothing was contesting the number, so nothing prompted the one command that would have settled it. Signed-off-by: yihou <yihou@amd.com>
…bvious fix misclassifies 47% of the table Answers T69's revised close (a): why identify reports lang='unknown' for Triton files it resolved to a source path. Measured by running taxonomy.fellow_of against the five device symbols in operator_identity, run 20260905T074905-9ec798, then against the whole 124-row kernel_table from the same run. Defect A: the triton-fellow pattern '^_[a-z0-9_]+_kernel$' requires a leading underscore AND a terminal _kernel. Real fla kernels are <op>_kernel_<suffix> and miss on two independent counts. k024 matched only because it happens to satisfy both -- one of three, and the one that passed is why the rule looked like it worked. Defect B: nothing matches _ZN7sgl_hip; the hip patterns are all aiter. The trap, and why the population mattered: relaxing to the substring '_kernel' costs one misclassification in the five-operator sample and 58 in the 124-row table -- ATen, rocprim, sgl_hip -- all handed to triton-fellow because it is first and first match wins. 47% of the table. In the sample it looked like a 20% edge case. Validated candidate '^(?!_Z)[A-Za-z_][A-Za-z0-9_]*_kernel([_A-Za-z0-9]*)$' moves 11 kernels from unclassified to triton (9 -> 20) and changes nothing else. Full anchoring excludes anything with a space, :: or <; (?!_Z) covers mangled names that are otherwise valid identifiers. Not landed: m1's full real chain is on 217 and m3 is downstream. The change turns refusals into campaigns, which is more work on a widened matcher. Same call as readme.md:98. Separately measured and left open: 79 of 124 still have no fellow. The taxonomy classifies about a third of what the profiler sees, and the practical exposure is smaller because rank promotes few of them -- by how much is unmeasured. Signed-off-by: yihou <yihou@amd.com>
…the one nobody can see Signed-off-by: yihou <yihou@amd.com>
…-m4-mocked is unreachable Signed-off-by: yihou <yihou@amd.com>
Both were queued behind '217 clearing m3'. Verified there is no such chain: four agent-sys run processes are live, all --var node=crsuse2-m2m-287 or -088; hold 112699 on 217 has been idle 35 minutes. The gate had no subject. kernel_taxonomy.yaml, fellows: -> triton-fellow. The pattern needed a leading underscore AND a terminal _kernel; real fla kernels are <op>_kernel_<suffix> and missed on two counts. Replaced with the anchored form validated in T72. Re-measured against the live 124-row kernel table after the edit: 9 -> 20 triton, hip 23 and ck 2 unchanged, 79 still unclassified. k004 and k014 now resolve; k015 and k018 correctly do not. The comment carries the trap rather than just the fix: a bare _kernel substring would take 58 more symbols -- ATen, rocprim, sgl_hip -- and first-match-wins hands every one to Triton before hip-fellow is consulted. buckets: carries the same old pattern at line 96 and is deliberately NOT changed -- it decides which kernels rank promotes, so widening it changes what the pipeline works on, not which fellow a campaign asks for. Noted in the file and in T72. build_workset.task/readme.md: the system prompt told the agent that ../../../../../rank0/definitions/ holds two worked examples to imitate. It is not in this repository and resolved nowhere from the zone. Replaced with the keys check_workset_shape actually grades, and the removed pointer written out so the agent is told the examples do not exist rather than left to look. show rc=0 after both. Signed-off-by: yihou <yihou@amd.com>
m2's sharpening. Zero containers is the same reading for never-started and started-and-torn-down; a cross-node control rules out a namespace artefact and cannot touch the temporal ambiguity. What resolved it was started_at in deploy.log -- an artefact carrying a timestamp of activity rather than state. Signed-off-by: yihou <yihou@amd.com>
…umber T72 -> T73 kernel_taxonomy.yaml, collective bucket: the comment said the aiter cross_device_reduce 'fell through to unknown, was marked routable, and rank selected it as candidate #1'. taxonomy.py:43-48 now returns routable: False for an unmatched symbol, so falling through today excludes rather than promotes. Written out rather than deleted -- the rule below exists for that episode, and under the current behaviour the same miss would have been a quiet omission instead of a bad candidate, which is not better. Third comment in m3's files today describing a prior state as a live one, after rank0 and forge_fellow. Comment-only: fellows 23/20/2/79 and routable 18 both unchanged after the edit. T73 carries the buckets: measurement the leader asked for. The asymmetry is the finding -- a fellows miss mislabels and is visible; a buckets miss lands in unknown with routable: False and is excluded, so rank never sees it. Seven kernels would move to routable, 3.220% of GPU time, pool 18 -> 25. The largest, fused_recurrent_gated_delta_rule_packed_decode_kernel at 2.350%, would rank #2 -- one kernel that should have been a top-two candidate has been invisible to rank for this entire effort. Existing worksets are not invalidated; the distortion is an omission at rank 2, not a wrong ordering. Scope limit recorded: one profile, one workload, one node; 2.350% for a decode kernel is a property of this trace's prefill/decode mix, and I have not measured a decode-heavier one. HELD by the leader for the round's goal (跑通即可), not the measurement -- expanding the pool changes which operator gets optimised and would discard the day's characterisation of attention_chunk_fwd_o. Cost of holding stated: the first green chain will be green over a distorted pool, so it will not have validated the selection. Renumbered T72 -> T73: readme-cn's expect_ranks entry took T72 at 08:04 (3d7a05e) and mine landed 08:24 (f834fcf), so mine is the duplicate. I reused a number from a heading listing taken before theirs existed -- the stale-pointer class, on an identifier this time. Both references in kernel_taxonomy.yaml updated in this commit. Signed-off-by: yihou <yihou@amd.com>
… (m3's measurement, leader's hold) Signed-off-by: yihou <yihou@amd.com>
… empty
Crossed the 73 ${X:?} sites in assets/ against the 9 package vars that default
to empty. Exactly three abort on empty: E2E_AIPERF_TRACE at
assets/load/aiperf_replay.sh:38 (m2's profiling bench), E2E_GSM8K_DATA at
assets/accept/measure.sh:190 (stage 5's correctness arm), and E2E_MEASURE_GPU,
which this block already passes.
The first ended two rung-5 attempts on 287. The engine came up READY and the
bench aborted with "AIPERF_TRACE: parameter null or not set"; the undelivered
output escalated, nothing answered, and the run was torn down -- which reads as
a stall and is not one. The second would have aborted the last stage of a run
that had by then spent four hours getting there.
Both paths verified readable before use.
Signed-off-by: yihou <yihou@amd.com>
…llision hypothesis died on the node Signed-off-by: yihou <yihou@amd.com>
…es nothing The other arm of main.py:1015. The companion 2026-09-03 record is about a run cut while a leaf works; this is a run that is genuinely hung and never cut. Measured on run 20260905T081811 (p4_d, real deploy on 088): 43 minutes with no write of any kind, zero escalations so blocked was empty, --stall-after 900 never fired. _is_running is truthful -- runner.py:420 documents it as whether a thread is carrying the attempt, and one was -- so holding stays non-empty and the elapsed-time test is unreachable at any threshold. Notes that the companion file's recommended fix, making the code implement its own docstring, makes this case strictly worse. Signed-off-by: yihou <yihou@amd.com>
readme-cn's third axis. Drift misdirects one reader and self-corrects; a collided T-number puts two findings under one name in an append-only file and stops being cheap the moment anything cites it. Signed-off-by: yihou <yihou@amd.com>
… by composing the value before the clock returned Signed-off-by: yihou <yihou@amd.com>
… looked checkpoint printed 09:28:36Z and hand-wrote 09:05Z into the file in the same command. Not the instrument answering a different question, which is what ast.parse and SAVE_OK are -- the instrument present, correct, and unused. Read-then-use: NOW=$(date -u) then reference it. Signed-off-by: yihou <yihou@amd.com>
…e census behind each claim Signed-off-by: yihou <yihou@amd.com>
… code Signed-off-by: yihou <yihou@amd.com>
…passed The process-table check is not redundant with rocm-smi and that is the point: a line that has launched and not brought up holds no VRAM. m1 asserted zero on 217 at 09:12 and m4's chain, launched 08:47, came up at ~09:13 -- one minute after the precondition passed. Only argv sees that window. Two working forms, m1's literal grep and m3's cross-node loop, checked against the same five live lines on 2026-09-05 and returning the same five PIDs. Three patterns that silently passed, all tried the same morning: pgrep -af 'cli.main run' (0 of 3 -- every line goes through run_with_long_stall.py so cli.main never appears in argv), pgrep -af 'python.*cli\.main run' (0 of 4, and it looks stricter), and a loop whose body contained the node name matching its own shell. Recorded why m3's form works, because it was not chosen for it: the agent.sys alternative matches --package agent_sys/... via regex . against the underscore, not cli.main. Sound reason, unchosen property. The failure mode is the load-bearing part -- a pgrep that matches nothing does not error, it reports the node free. All three returned 'free' about a busy machine, and each was caught only because the reader already knew the answer. Hence the rule: a check that says free when you are hoping for free is the one to re-run. Also: take distinct port_router, container and work_root even when the check is clean. Signed-off-by: yihou <yihou@amd.com>
…t from the canonical block is the only one with no outlier property Signed-off-by: yihou <yihou@amd.com>
…es, and one row is pending evidence Signed-off-by: yihou <yihou@amd.com>
…ll detector, none died Signed-off-by: yihou <yihou@amd.com>
checkpoint found all four 'unexplained deaths' were the 900s detector firing as designed, and the line saying so was second-to-last in logs readable all day. My poll grepped for 'phase', matched that line, and tail -1 discarded it. Signed-off-by: yihou <yihou@amd.com>
Signed-off-by: yihou <yihou@amd.com>
m4's 64-minute 'stall' was an agent that had finished and refused, with the complete reason in its transcript the whole time. We diagnosed it from mtimes and the process table, and I sent them to wchan and stack. Signed-off-by: yihou <yihou@amd.com>
…ence -- replacing both looser claims Signed-off-by: yihou <yihou@amd.com>
…ntrolled and load-bearing (3885050) Signed-off-by: yihou <yihou@amd.com>
readme-cn hit the doubled-pathspec failure a second time, having written the lesson after the first. What saved it both times was reading the command output, not the rule. So: a rule that needs remembering fails; a rule that changes the command works. Placed at the head of the debugging section because it governs whether anything below it does anything. Signed-off-by: yihou <yihou@amd.com>
…correctly - the natural hypothesis is dead Signed-off-by: yihou <yihou@amd.com>
…r, and the abundant-precondition result is the real finding Signed-off-by: yihou <yihou@amd.com>
…omes not code Signed-off-by: yihou <yihou@amd.com>
… across containers; under-refusing is the dangerous direction Signed-off-by: yihou <yihou@amd.com>
… the rule - and an unowned actor is creating yihou_m3_* containers on two nodes Signed-off-by: yihou <yihou@amd.com>
… the name Signed-off-by: yihou <yihou@amd.com>
…t is established, from a 149-file zone Signed-off-by: yihou <yihou@amd.com>
…ion, and counting handoff states overcounts refusals Signed-off-by: yihou <yihou@amd.com>
… -11.2% remains and its cause is unread Signed-off-by: yihou <yihou@amd.com>
…l is real; stopping GPU rounds because no launch variable addresses it Signed-off-by: yihou <yihou@amd.com>
… result asymmetrically; and stopping before stage 5 is a safety requirement Signed-off-by: yihou <yihou@amd.com>
…t - a collision costs a run and reads healthy Signed-off-by: yihou <yihou@amd.com>
… nobody said Measured 2026-09-05: every workset in circulation records commit 'unknown', including the 11:19:40 artefact grafted into the mock corpus and consumed by two downstream stages. E2E_PACKAGE_COMMIT has a --var and was passed zero times in five launches -- by me, after I wrote 'declaring it does not fix that; supplying it does' into the yaml. The declaration was not the fix. scaffold.py now falls back to git rev-parse HEAD on the package tree. assets/serve/round.sh:272 already does this; copied, not invented. 'unknown' remains the fallback when the tree is not a checkout, so this can only add information. Second reason from tonight: a run died because its package copy predated the fix it was meant to test by 33 minutes, and nothing in the run said which code it had. The surprise is documented in the schema rather than left to be found: it records the STAGED COPY's commit, and a run re-stages per task, so a chain spanning a landing records different commits at different stages. That answers 'which code produced this artefact', not 'which commit was the run launched at'. There is no field for the second and a reader assuming one value per run will read the variation as a bug. Also noted there: 'unknown' is exactly 7 characters and cleared the existing minLength: 7 floor, which is why the schema never caught it. Verified: real SHA on a git tree, 'unknown' on a non-git dir and on a missing path; py_compile; a real workset still validates through the package's own schema loader; show rc=0. Schema edit redone as a text edit after a json round-trip with ensure_ascii=False silently rewrote three unrelated descriptions -- reverted, diff is now 2+/1- and touches only this field. Signed-off-by: yihou <yihou@amd.com>
…alising a shared file rewrites other people's lines Signed-off-by: yihou <yihou@amd.com>
…ming answer, unopposed source, decider benefits Signed-off-by: yihou <yihou@amd.com>
Signed-off-by: yihou <yihou@amd.com>
…ally - it was never a 1-in-13 rate Signed-off-by: yihou <yihou@amd.com>
…mpaign
The user's standing task is now defect discovery rather than a green board, so
every run is a sample and needs a record. A run does not keep the line that
launched it -- measured: the staged step still reads expect_ranks:
'${expect_ranks:-8}', so whether a run passed 4 or fell back to 8 is
unrecoverable, and at least four incidents traced to a launch variable nobody
could check afterwards.
Records: node/jobid/cards/mocked-stages as their own fields (with two hosts the
node is a variable, and if the empty-zone rate differs between them that is the
first structural clue about the mechanism); the launch line sorted so two runs
diff cleanly; package provenance; zone counts; verdicts tallied BY VERDICT
rather than by handoff state, because a failing sibling invalidates its passing
siblings and counting handoff states overcounts refusals; and the last three log
lines rather than the last one, because the terminal line reads like an ordinary
stop and the cause is above it.
Two defects found by testing it against a run whose answer was already known:
it resolved a relative --package against cwd and silently dropped the whole
provenance section, and it read GENERATED.txt as of collection time rather than
run time -- reporting 6cee278 for a run that used c870c5d. That is the same
staleness trap one level up, inside the tool built to detect staleness. It now
prefers the launcher's own "package built from ..." line, recorded at launch.
Signed-off-by: yihou <yihou@amd.com>
… can start loading, and a card reading expires in minutes Signed-off-by: yihou <yihou@amd.com>
…run survives, the location does not Signed-off-by: yihou <yihou@amd.com>
Six separate reports of the empty-materials fault gave a COUNT -- "1 of 13" --
and stopped there. A count invites a rate, and the rate we all wrote down
("non-deterministic, validator-agnostic, roughly 1 in 13") was wrong: it is
m2_profiling's output-validation zone every time. Found by team-lead resolving
five runs by hand; independently reproduced here across six, including r7m1a,
which is a configuration never previously run (m2 real, m5 mocked, other half of
the node) and which is m2_profiling as well.
The identity was always one lookup away: the zone directory is
validation.<TASK-ID>.<phase>.<hash> and the store maps task id -> closure. The
tool now does that lookup and prints closure=<name> on every empty zone, so the
report is an identity rather than a number.
My own "six runs, exactly one empty each" was the refutation of the rate reading
and I did not read it that way -- an independent per-zone probability puts some
runs at zero and some at two. Hand over the table, not the summary statistic.
Signed-off-by: yihou <yihou@amd.com>
…read the parent task, not the zone's own Signed-off-by: yihou <yihou@amd.com>
…he closure On 2026-09-06 two of us read the same empty zone and reached opposite conclusions: "parent closure = main, kinds = ['e2e_packup']" and "closure = m2_profiling". Both readings were of the zone's identity by inference -- one from directory nesting, one from mapping the zone name's task id through the store -- and a correct finding was withdrawn on the strength of the first. The unambiguous discriminator is the handoff the zone lists in inputs.json and that handoff's kind. Measured that way across eight runs: seven keep-package runs profiling_evidence v0, 0 files EMPTY one noval run (p6m1) profiling_evidence v1, 45 files populated So it is profiling_evidence every time, and the one run where it was populated staged v1 instead of v0. The tool now prints kind= and the staged versions alongside the closure, so the identity does not have to be inferred again. Signed-off-by: yihou <yihou@amd.com>
… a rate - and resolve the identity from the zone name Signed-off-by: yihou <yihou@amd.com>
…efusing validator is not exercised Signed-off-by: yihou <yihou@amd.com>
…cannot expose it - and it made me retract a true finding Signed-off-by: yihou <yihou@amd.com>
…ol, and it reproduces GPU-free in 18 minutes Signed-off-by: yihou <yihou@amd.com>
…actly like a hang; and compare the strength of a refuting check against the convincing one Signed-off-by: yihou <yihou@amd.com>
…ckage generator, not the framework Signed-off-by: yihou <yihou@amd.com>
…est the repository package before budgeting for it Signed-off-by: yihou <yihou@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please include a brief summary of the changes, relevant motivation and context.
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: