[sai-miduthuri]

Description

When a release test fails on master, #65906 asks the Anyscale observability
agent why, #65910 prints the answer in its own buildkite group, and #66006
attaches it to the job as an annotation. All three reach whoever opens the
build. None of them reaches the person who owns the failing test, who is
watching its GitHub issue.

This posts the analysis there. When the state machine is already tracking an
open issue for the test, the reporter comments the agent's summary on it, with
the failing build in the opening line and a link to the slack thread holding the
full report.

Nothing new is created: the issue is the one ReleaseTestStateMachine opened
when the test started failing, and the comment only appears while that issue is
open. A test with no tracked issue is skipped before the repo handle is built,
so the common case does not pay for the AWS Secrets Manager fetch that building
it costs.

Test.get_open_github_issue is extracted from is_jailed_with_open_issue,
which now delegates to it via has_open_github_issue. Checking that the issue is
open is required rather than defensive: _close_github_issue closes the issue
but leaves KEY_GITHUB_ISSUE_NUMBER on the test, so a recovered test points at a
closed issue indefinitely.

Related issues

None.

Additional information

What the comment looks like

Latest run: https://buildkite.com/ray-project/release/builds/108659#01a0e9e5-…

Observability Agent RCA:

<the agent's summary>

[Full report and feedback](<slack thread>) — rate it with the 'All good' or 'Needs correction' buttons in the thread.

Laid out like the buildkite annotation, and the test is not named because the
comment is on that test's own issue. Each part is dropped rather than left empty
when the agent did not supply it; a response with neither a summary nor a thread
is not posted at all.

One comment per build, not one per job

A test with repeated_run gets one job per repeat — microbenchmark.aws has
five — and a manual retry adds another. All of them are separate processes in
one build, so the only state they share is build-scoped agent meta-data. Each
job claims obs-agent-commented-<test> before posting; a job that finds it
claimed skips.

The claim is written and then read back, because buildkite-agent meta-data
has set/get/exists/keys and no compare-and-set. set is
last-writer-wins, which is what makes the read-back work: when two jobs both
find the key unset and both write, the one whose write landed last reads its own
id back and owns the claim, and the other reads a stranger's and stands down.
Two details make that safe:

• The claimant is BUILDKITE_JOB_ID, and the protocol is skipped entirely when
  there is none — two jobs writing the same constant would each read it back and
  both proceed.
• A read-back returning nothing means the agent call failed, not that somebody
  else won: the key was just written. Standing down there would let a broken
  agent silently drop the analysis, so it stands down only on positively reading
  another job's id.

It narrows the race rather than closing it: both jobs still post if one reads
back before the other writes at all, which needs the second write to land in the
gap between one job's two consecutive local calls.

A failed post hands the claim back, by writing an empty value since the agent
cannot delete a key — but only when retrying could succeed (>= 500, 429,
timeouts). Releasing on a permanent failure would make every remaining job fail
the same way.

The buildkite annotation is deliberately not deduped this way. One agent
report per test job is the point of it, and test_the_annotation_is_not_deduped_by_build
pins that.

Everything interpolated into the comment is defused first

The body is markdown and the summary is free-form text from an LLM, so three
things in it would be side effects rather than intentions:

• @name notifies a real person and #123 posts a backlink on that issue. Both
  autolinks are broken with an empty html comment, which renders as nothing.
• Anything github reads as an html tag is dropped by its sanitizer — <lambda>
  and <module> are routine in a traceback — so the markup is escaped.
• Code spans and fenced blocks are left alone entirely: github autolinks neither
  and renders neither as html there, and rewriting inside one would show the
  escapes and the comment marker literally in a quoted log line.

The slack thread url is another value the agent's response decides, so only a
plain http url becomes a link, in the <...> destination form; anything else is
shown as code rather than given the chance to close the link early.

Oversized summaries are trimmed, not dropped

Github rejects a body over 65536 characters. The rest of the body is measured
first and the summary given what is left, so the failing build and the slack link
always survive. The trim cuts the raw summary and re-sanitizes, because
sanitizing expands — < becomes four characters and @ eight — so a budget
spent on raw characters would still overflow, and cutting sanitized text would
cut through a half-written entity. Each pass shortens in proportion to the
overshoot rather than by it; subtracting it outright discarded the whole analysis
for a summary full of @ or <.

This matters beyond tidiness: the 422 an over-long body produces is exactly the
permanent failure the claim above does not release on, so without the trim one
oversized analysis meant no comment from any job in the build.

Nothing here can change a test result

glue.py runs the reporting loop without guarding reporter.report_result(...),
so every failure in this path is caught and logged. GitHubClient also gained a
30s timeout on every request: it had none, and requests waits forever, so a
github that accepted the connection and never answered would have held the step
open until buildkite killed it.

Not duplicating existing work

gh pr list --repo ray-project/ray --state open --search "observability agent github issue comment"
gh pr list --repo ray-project/ray --state open --search "ray_release reporter github issue"

Neither returns anything in this area. The three PRs this builds on (#65906,
#65910, #66006) are all merged.

Tests

//release/... cannot be analysed on current master — //release:requirements_py310_test
references @rules_python//python/pip_install:pip_compile.py, which no longer
exists in the pinned rules_python. That reproduces on a clean master checkout
with no changes, so the release_unit targets were run explicitly instead:

$ bazel test $(bazel query 'attr(tags, release_unit, tests(//release:all))') --test_output=errors
Executed 29 out of 29 tests: 29 tests pass.

$ pre-commit run
# all hooks passed

New coverage in test_observability_agent_reporter.py and test_test.py:

Behavior	Test
Comments on an open issue, fetching it exactly once	test_comments_on_an_open_issue
Silent on a closed issue, and without a tracked issue	test_does_not_comment_on_a_closed_issue, test_does_not_reach_github_without_a_tracked_issue
A github failure never propagates	test_a_github_failure_does_not_propagate
The agent's prose cannot mention people or link issues	test_the_comment_cannot_mention_people_or_link_issues
Markup is escaped, but not inside code	test_the_comment_escapes_what_the_agent_sent, test_the_comment_leaves_the_agents_code_spans_alone
The slack url cannot escape its link	test_the_slack_link_target_cannot_escape_the_link, test_the_slack_link_uses_a_bounded_destination
One comment per build across five jobs	test_only_one_job_per_build_comments
A simultaneous claim is settled by the read-back	test_the_loser_of_a_simultaneous_claim_does_not_comment
An unverifiable claim still comments; no job id means no claim	test_an_unverifiable_claim_still_comments, test_no_claim_without_a_job_id_to_claim_with
Transient failures release the claim, permanent ones keep it	test_a_transient_failure_hands_the_claim_back, test_a_permanent_failure_keeps_the_claim
The annotation is not deduped	test_the_annotation_is_not_deduped_by_build
Oversized summaries are trimmed and still fit	test_an_oversized_summary_is_trimmed_to_fit
With no summary the thread is the body; with neither, no comment	test_the_comment_points_at_the_thread_when_there_is_no_summary, test_no_comment_when_the_agent_returned_nothing
get_open_github_issue over open / closed / unreachable	test_get_open_github_issue

The dedupe and response-shape tests were checked against the bug rather than
only the fix: with them in place, removing the claim fails five of them, and
removing just the read-back fails the simultaneous-claim one.

Note that bazel test //release/... cannot be analysed on current master —
//release:requirements_py310_test references
@rules_python//python/pip_install:pip_compile.py, which no longer exists in the
pinned rules_python. That reproduces on a clean master checkout, hence the
explicit target list above.

Exercised end to end on a real build

The reporter is gated to AUTOMATIC=1 runs on master, so a PR pipeline would
normally run none of this. Four temporary commits opened that path — a canned
agent response, a stand-in anyscale job id and issue number, a forced failure of
hello_world.aws before anything reached anyscale, and an unconditional export
of the flag — and all four are reverted in this PR (73cc06406a,
07d9850788, 34c63a2014, 3eb60acc13). The canned summary was written to
carry exactly what the comment path defuses.

Verified on builds 108426 and 108659, commenting on a scratch issue:

Behaviour
Comment posted on the tracked issue	verified
Latest run: names the failing job	verified
<class 'ValueError'> and <lambda> render as text, not eaten by the sanitizer	verified
@someone and #1234 render plainly — no notification, no issue backlink	verified
A fenced code block passes through verbatim	verified
A url #fragment in the prose survives	verified
Slack link in the bounded <...> destination form	verified
One comment per build, across the final automatic retry and each manual one	verified
The annotation is not deduped — one per job, per attempt	verified

One defect surfaced and was fixed: the comment came out with no Latest run:
line, because result.buildkite_url is filled in by _load_test_configuration
and the forced failure raised above the call to it. That was the scaffolding
rather than the feature — a real failure cannot reach the reporters that way,
since that call is the first thing in the try.

AI assistance

AI assistance (Claude Code) was used for this change.

🤖 Generated with Claude Code

[sai-miduthuri]

GitHub Issue comment example created at anyscale#2040 through release pipeline build buildkite/release for commit 224a50a.

[gemini-code-assist]

Code Review

This pull request enables the Observability Agent to post its Root Cause Analysis (RCA) directly as a comment on a test's open GitHub issue. It includes logic to prevent duplicate comments from multiple jobs in the same Buildkite build, sanitizes agent output to avoid unwanted mentions or autolinks, and handles truncation to respect GitHub's comment size limits. Additionally, a default timeout is introduced for GitHub API requests. The feedback suggests adding a timeout to the buildkite-agent subprocess call to prevent potential hangs during metadata operations.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3eb60ac. Configure here.}

[elliot-barn]

theres an argument to be made where we should point users to the agent's findings in slack if we want users to comment their responses on the threads. I'm open to approving this for now but lets assess if this increases engagement with the observability agent findings

[elliot-barn]

*Point users to the slack message instead of outputting the content as a github comment

[sai-miduthuri]

This is for a hypothetical, unlikely case where the "summary" (which isn't the full root cause analysis by the agent, and should never actually be 65536 chars long) created by the agent is just so large that it reaches the GITHUB_COMMENT_LIMIT. When this happens, the summary is truncated with enough room to accommodate the Slack message link at the end.

The Slack message link is guaranteed to always be printed.