[release] add an observability agent reporter for failed release tests

[sai-miduthuri]

Description

Adds ObservabilityAgentReporter, a release test reporter that asks the Anyscale
observability agent why a failed release test job failed, and logs the answer at the
end of the buildkite step.

When a release test run ends in an application-level failure, the reporter creates an
observability agent debug session for the test's Anyscale job, asks it "Why did this
job fail?", and logs the summary of the analysis together with a link to the slack
thread that holds the full report. The intent is that whoever triages a red release
test gets a first read on the failure without leaving the buildkite log.

It triggers only on the RUNTIME_ERROR, ERROR and UNKNOWN result statuses, and is
a no-op for everything else, so that infra failures the agent cannot explain
(INFRA_ERROR, INFRA_TIMEOUT, TRANSIENT_INFRA_ERROR) are left alone. It is also a
no-op for failures that never got as far as creating an Anyscale job.

Related issues

None.

Additional information

Where the job id comes from. AnyscaleJobManager already captures the Anyscale
production job id when it submits the job, AnyscaleJobRunner.job_id() exposes it, and
glue.py stores it on result.job_id before the reporting loop runs. The reporter
reads it from there rather than making a second lookup.

Reporting never changes a test outcome. run_release_test_anyscale runs the
reporting loop without guarding reporter.report_result(...), so every failure in this
reporter is caught and logged. An observability agent outage cannot turn a passing test
red, or add noise to the traceback of a failing one.

Log volume. Only the summary and the slack thread link are logged at INFO. The full
response, which also carries the findings, issues and next steps, goes to logger.debug
so it stays available without cluttering the step output. A response with no
metadata.slack_thread is logged as an error, since every response is expected to carry
one.

SKIP_COMMAND_FAILURES ships off. The reporter carries a gated skip for failures
raised by the test command itself (COMMAND_ERROR, COMMAND_ALERT, COMMAND_TIMEOUT,
PREPARE_ERROR), disabled by default and pending a decision in review. Worth knowing
before flipping it: a result with the ERROR status always carries one of those return
codes, so enabling the skip leaves only RUNTIME_ERROR and UNKNOWN triggering the
agent.

Sample output, from a run against a real failed staging job:

[INFO] observability_agent.py: 130  Observability agent analysis of job prodjob_5cjsupu5jvs67f3s1z7v6j8zgi:
The job failed because the job's container entrypoint/driver process exited with an
entrypoint command error (RAY_JOB_ENTRYPOINT_COMMAND_ERROR). Platform events show the
job transitioned to ERRORED and then OUT_OF_RETRIES.
>>> Only the summary is logged here. The full report, with the evidence and
>>> next steps behind it, is in the slack thread below.
>>> The observability agent is under active development: please rate that
>>> report with the 'All good' or 'Needs correction' buttons in the thread.
>>> Full report and feedback: https://anyscaleteam.slack.com/archives/...

Not a duplicate

Searched the open PRs for work in this area before starting; there is no open PR adding
an observability agent reporter, or otherwise touching
release/ray_release/reporter/:

gh pr list --repo ray-project/ray --state open --search "observability agent release test"
gh pr list --repo ray-project/ray --state open --search "ray_release reporter"
gh pr list --repo ray-project/ray --state open --search "obs_agent"

Tests

$ bazel test //release:test_observability_agent_reporter
//release:test_observability_agent_reporter                              PASSED in 3.8s
# 12 tests: trigger statuses, no-op statuses, missing job id, logged summary and slack
# thread, missing slack thread logged as an error, 422 and malformed responses swallowed,
# and the gated command-failure skip in both positions.

$ bazel test //release:test_glue //release:test_run_script //release:test_result
//release:test_glue                                                      PASSED in 12.7s
//release:test_run_script                                                PASSED in 4.7s
//release:test_result                                                    PASSED in 2.4s

$ pre-commit run --files release/ray_release/reporter/observability_agent.py \
    release/ray_release/tests/test_observability_agent_reporter.py \
    release/ray_release/scripts/run_release_test.py release/BUILD.bazel
# all hooks passed

The reporter was also exercised end-to-end against the staging observability agent with
a real failed job id, before the logging was narrowed to the summary: the debug session
was created, the query returned an analysis, and the no-op paths made no HTTP calls at
all. The current code is covered by the unit tests above.

AI assistance

AI assistance (Claude Code) was used to write this change.

[gemini-code-assist]

Code Review

This pull request introduces the ObservabilityAgentReporter to query the Anyscale observability agent for failed release test jobs and log their analysis. It also adds corresponding unit tests and a temporary forced failure in hello_world.py for pipeline verification. The review feedback is highly actionable, pointing out the need to revert the temporary test failure before merging, recommending an early exit when ANYSCALE_CLI_TOKEN is missing to prevent noisy tracebacks, and suggesting safer dictionary lookups to avoid potential AttributeError crashes when handling null values in API responses.

I am having trouble creating individual review comments. Click here to see my feedback.

release/hello_world_tests/hello_world.py (6-10)

This temporary change forces a failure for testing. It must be reverted before merging this pull request to avoid breaking the hello_world release test.

release/ray_release/reporter/observability_agent.py (93-99)

Since this reporter is appended unconditionally to the reporters list, it will run in environments (such as OSS CI or local runs) where ANYSCALE_CLI_TOKEN is not set. Currently, a missing token causes _post to raise a RuntimeError, which is caught and logged as a full traceback via logger.exception. To avoid misleading and noisy tracebacks in the logs of these environments, consider adding an early exit check for the token in report_result.

        job_id = result.job_id
        if not job_id:
            logger.info(
                f"Skip triggering the observability agent for test "
                f"{test.get_name()}; the test run has no Anyscale job id"
            )
            return

        if not os.environ.get("ANYSCALE_CLI_TOKEN"):
            logger.info(
                f"Skip triggering the observability agent for test "
                f"{test.get_name()}; ANYSCALE_CLI_TOKEN is not set"
            )
            return

release/ray_release/reporter/observability_agent.py (120-122)

If the response contains "result": null, response.get("result", {}) will return None instead of {} because the key exists. This will cause an AttributeError when calling .get() on query_result. Similarly, if analysis or metadata are null in the JSON response, calling .get() on them will also raise an AttributeError. Using or {} handles these cases safely.

        query_result = response.get("result") or {}
        summary = (query_result.get("analysis") or {}).get("summary")
        slack_thread = (query_result.get("metadata") or {}).get("slack_thread")

release/ray_release/reporter/observability_agent.py (144)

If "result" is null in the response, response.get("result", {}) will return None, causing an AttributeError when calling .get("debug_session_id"). Using or {} prevents this potential crash.

        debug_session_id = (response.get("result") or {}).get("debug_session_id")

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 006cdad. Configure here.}

[sai-miduthuri]

The Observability Agent trigger behavior is showcased in the buildkite/release CI test for commit 6f88ae3.

The failing hello_world.aws test triggered a debug session because of a (forced) failure in the Ray job during the test, which was classified as a COMMAND_ERROR, and was thus allowed to trigger the agent in this PR.

The failing train_benchmark-qwen3_06b_deepspeed test did not trigger a debug session because it was already identified to be an INFRA_ERROR caused by insufficient instance capacity.