[release] print the observability agent analysis in its own buildkite group

[sai-miduthuri]

Description

Stacked on #65906, which adds the observability agent reporter. Review that one
first; this PR's diff is only the change on top of it.

That reporter logs its analysis with logger.info, which puts it in the middle of
the +++ :memo: Reporting results group — between the other reporters' output and
the traceback of the failure it is explaining. This moves it to the end of the
step, under its own group:

+++ :memo: Reporting results
    reporters ... traceback ... "Script failed on try 1/1 ..." epilogue
+++ :robot_face: Observability agent analysis
    summary, feedback ask, slack thread link

The reporter writes its message to the file named by RELEASE_TEST_OBS_AGENT_FILE
and run_release_test.sh prints that file as its last action. Buildkite log groups
are sequential, not nested — a group runs until the next header — so a group
opened by the reporter would file the traceback, the shell epilogue and the exit
status under the analysis heading. Printing last is what avoids that, and it is why
this is not simply a buildkite_group() call inside the reporter.

When no file is configured or the write fails, the reporter logs the message inline
exactly as before, so the analysis cannot be lost, and a failure to write it cannot
change the outcome of the test run.

Related issues

None.

Additional information

Two decisions worth a reviewer's opinion, both one-line changes:

• When the file is written, the reporter logs a one-line pointer to it rather than
  the message, so the analysis is not duplicated in one log while still leaving a
  breadcrumb if the group never appears.
• The default path is /tmp/obs_agent_analysis.txt, outside RELEASE_RESULTS_DIR,
  so it is not uploaded to the Artifacts tab. Putting it under RELEASE_RESULTS_DIR
  would make it retrievable as an artifact at the cost of one more artifact per
  failing test.

Hardening that came out of self-review, each with a regression test that fails
when the fix is reverted:

• The analysis is written as UTF-8. open() without an explicit encoding uses the
  container locale, and real agent summaries contain em dashes, so an ascii locale
  raised UnicodeEncodeError — not an OSError, so it escaped the write guard.
  glue.py does not guard the reporting loop, so that would have failed the test.
• Null fields in the agent's response no longer raise. response.get("result", {})
  returns None when the key is present and null, and the next .get raised an
  AttributeError out of the reporter.
• The analysis body is indented when printed. It is prose written by an LLM, and a
  line starting with ---, +++ or ~~~ would otherwise open a buildkite group of
  its own and file the rest of the analysis under it.
• The unit tests no longer inherit RELEASE_TEST_OBS_AGENT_FILE from the
  environment, which this PR makes run_release_test.sh export.

Not duplicating existing work

gh pr list --repo ray-project/ray --state open --search "observability agent buildkite group"
gh pr list --repo ray-project/ray --state open --search "run_release_test.sh"

Neither returns anything touching this area; the only related PR is #65906, which
this one is stacked on.

Tests

$ bazel test --test_tag_filters=release_unit //release/... --test_output=errors
Executed 29 out of 29 tests: 29 tests pass.

$ bazel test //release:test_observability_agent_reporter //release:test_run_script
//release:test_observability_agent_reporter                              PASSED
//release:test_run_script                                                PASSED

$ pre-commit run --files release/ray_release/reporter/observability_agent.py \
    release/ray_release/tests/test_observability_agent_reporter.py \
    release/ray_release/tests/test_run_script.py release/run_release_test.sh
# all hooks passed, shellcheck included

New coverage:

Behavior	Test
The analysis is written to the configured file	test_analysis_written_to_file
It is logged inline when no file is configured	test_analysis_logged_when_no_file_is_configured
A write failure falls back to logging, and never propagates	test_analysis_logged_when_the_file_cannot_be_written, test_write_failures_never_propagate
Non-ascii summaries round-trip	test_analysis_file_handles_non_ascii
Null result / analysis / metadata do not raise	test_null_fields_in_the_response_do_not_raise
The group is printed, and printed last	test_obs_agent_analysis_is_printed_in_its_own_group
No group when there is no analysis	test_no_group_when_there_is_no_analysis
Markup in the summary cannot open a group	test_analysis_cannot_open_a_buildkite_group_of_its_own

The buildkite rendering itself is not verifiable locally; the group markers are
asserted as text.

AI assistance

AI assistance (Claude Code) was used for this change.

[gemini-code-assist]

Code Review

This pull request updates the observability agent reporter to write its analysis to a file instead of logging it inline, and configures run_release_test.sh to print this analysis at the end of the test run under its own Buildkite group. Feedback on these changes includes adding || true to the rm -f cleanup command to prevent script crashes under set -e, ensuring the analysis path is a regular file before reading it, and providing a fallback string in the reporter if the agent returns a null summary.

[gemini-code-assist]

If the observability agent returns a response where analysis is present but summary is None or null, summary will be None. This results in the message containing the literal string "None" (e.g., "Observability agent analysis of job <job_id>:\nNone"). Providing a fallback string like "No summary provided." when summary is empty or None improves readability and prevents logging "None".

Suggested change

	summary = (query_result.get("analysis") or {}).get("summary")
	summary = (query_result.get("analysis") or {}).get("summary") or "No summary provided."

[sai-miduthuri]

Fixed, but not with a placeholder string — or "No summary provided." would make the problem more visible rather than less. The file stays non-empty either way, so the step still ends with a prominent +++ :robot_face: Observability agent analysis group whose entire content is that sentence, and it reads as an intended state, so an API regression would look designed rather than broken. None is at least obviously wrong.

Instead every field the agent leaves out is named where the group is read: it states that the summary is missing and gives the debug session id, states when the slack thread is missing too, and is still emitted when the response is empty — so an agent that returned nothing does not look like an agent that never ran. That also matches how a missing slack thread was already handled, with a logger.error rather than a substituted value.

With both fields missing the group now reads:

Observability agent analysis of job prodjob_x:
>>> The agent returned no summary for this job.
>>> Debug session: oasess_y
>>> The agent returned no slack thread, so the full report
>>> and its feedback buttons cannot be reached from here.

Covered by test_missing_summary_is_named_in_the_analysis and test_empty_analysis_still_reports_the_failure, both asserting no "None" reaches the output.

[sai-miduthuri]

The Observability Agent response can be viewed in the logs of the forced hello_world.aws test failure on buildkite/release on commit b8d3428.

The agent's summary and thread URL are printed onto the logs more prominently, and given their own Observability agent analysis group in the Buildkite job logs.