[sai-miduthuri]

Description

Stacked on #65910, which is stacked on #65906. Review those first; this PR's
diff is only the annotation on top of them.

#65910 puts the observability agent's analysis in its own buildkite group at the
end of the failed step — but that only reaches someone who opens the step. This
adds a buildkite annotation, so a red nightly shows its analyses without
digging through step output.

The annotation is job-scoped (--scope=job), so it renders on the job it
came from; buildkite also surfaces job-scoped annotations on the build page, so
one call reaches both places. Scope is what decides this: --scope defaults to
build, and --job alone only records which job an annotation came from — the
first run on this branch passed --job without --scope and the annotation
landed at the build level, which is how this was found.

The full command:

buildkite-agent annotate --style=info --scope=job --context=obs-agent-<test name> --append --job <job id>

Each attempt of a retried test gets its own annotation, because a buildkite
retry is a new job with a new id — the API shows retry_source.job_id differing
from id — and a job-scoped annotation is identified by its context within its
job. So every attempt's report is kept, attributed to the attempt that produced
it, and buildkite shows them together on the build page.

The context is obs-agent-<test name> because an annotation needs one and
buildkite defaults it to "default"; the test name makes it identifiable in the
UI and keeps it clear of anything else annotating the same job. It is not what
separates the attempts — scope already does that. --append is passed for the
case of a job annotating twice under this context, which does not happen today
(one release test job runs one test once), but where replacing would be wrong.

Each block is headed with the test name, the attempt number, and a link to the
Anyscale job:

hello_world.aws — attempt 1 (BUILDKITE_RETRY_COUNT=0) — prodjob_w9q946e1l8xvk6p6bdywmzh8li

<the agent's summary>

Full report and feedback — rate it with the 'All good' or 'Needs correction' buttons in the thread.

Both numbers are rendered because they disagree by one: buildkite labels the
first try "Retry 1 of N", while BUILDKITE_RETRY_COUNT counts retries after
it and is 0 there. Rendering the raw value alone put "attempt 3" on the attempt
the UI calls "Retry 4 of 5"; rendering both reconciles the annotation with the
label it hangs off and with the job log, which prints the raw value.

Guarded on BUILDKITE, and wrapped so that neither a missing binary nor a
non-zero exit can change the outcome of the test run — the same rule the rest of
this reporter follows. Everything interpolated into the body is HTML-escaped:
buildkite renders the body as markup, and the agent's summary is free-form prose
where an unescaped < would be rendered rather than shown.

This is not new ground for the repo. buildkite-agent annotate already has
two call sites: release/ray_release/byod/build.py:71-81, from Python inside
ray_release, and .buildkite/hooks/post-command:19-20, from the host hook.

It also needs a change to the release test step config

The first CI run on this branch produced no annotation:

Could not annotate the buildkite job: [Errno 13] Permission denied: 'buildkite-agent'

Two separate causes, both in the step's docker plugin config, and both fixed in
f51462c6f4:

• The step hand-mounted /usr/local/bin/buildkite-agent, but the agent on these
  hosts is /usr/bin/buildkite-agent-stable. The source path does not exist, so
  docker creates an empty directory at the destination, and executing a
  directory is EACCES — the error above, rather than the ENOENT a missing
  file would give. That volume had to be removed, not supplemented:
  /usr/local/bin precedes /usr/bin on PATH in the python image, so the
  stale directory shadows any working binary mounted elsewhere.
• BUILDKITE_AGENT_ACCESS_TOKEN is not among the variables reaching the
  container — propagate-environment does not carry it — so even a working
  binary could not authenticate.

mount-buildkite-agent: true handles both. Note that the rayci keys
(allow_mount_buildkite_agent in .buildkite/release/config.yml,
mount_buildkite_agent in build.rayci.yml) do not apply here: release test
steps are written to release_tests.json and uploaded straight to buildkite by
custom_image_build_and_test_init, never generated by rayci, so the docker
plugin's own hyphenated option is the reachable lever.

Worth a reviewer's attention: this grants every release test container a
working buildkite-agent and the agent access token, so test code could
annotate the build or upload artifacts as the agent. It is the same trust level
the release build steps already run at, and release tests are internal code,
but it is a change in posture on a shared file rather than a change local to
this reporter.

Related issues

None.

Additional information

Verified on real builds

Three temporary commits carried this PR through CI and have all been reverted;
each pair is on the branch so the history shows what was run:

Temporary commit	Revert	What it was for
131706a3e3 serve a canned agent response	98c8f488b3	Iterating without creating a real debug session and slack thread on every run
e5f741e939 force hello_world to fail	ab9f02ebf3	Getting a release test run to actually reach the agent
e456942239 export TRIGGER_OBSERVABILITY_AGENT unconditionally	cb49f5edf4	#65906 gates the reporter to AUTOMATIC master runs, so a PR pipeline would otherwise run none of this

No temporary commit remains in effect. The final run was against a genuinely
failing test with the real agent, and produced the reports as intended.

What the builds settled

A job-scoped annotation does not accumulate across retries — each attempt
annotates its own job. The docs were not explicit about whether --append
with a stable context across a new --job grows one annotation or opens
several. Build 106662 answers it: the two attempts of hello_world.aws that
reached the agent produced two annotations under the identical context
obs-agent-hello_world.aws, one block each — "attempt 3" on one job, "attempt 4"
on the next. Scope, not the context, is what separates them.

That is the behaviour this PR wants: every attempt's analysis is kept and
attributed to the attempt that produced it, rather than concatenated into one
block where it is unclear which run said what.

Two related notes for anyone reading these programmatically: Build.annotations
returns only build-scoped annotations over both the REST and GraphQL APIs — the
job-scoped ones are reachable under Job.annotations — and the build page
showing them is a UI behaviour, not API-level promotion.

Automatic retries produce exactly one trigger per build. Since #65907 landed, the step publishes
BUILDKITE_MAX_RETRIES=3 for hello_world (from its num_retries: 3) and the
script honours it, so _is_transient_error rewrites every attempt except the
last to TRANSIENT_INFRA_ERROR, which is not a trigger status:

attempt	BUILDKITE_RETRY_COUNT	retry_count >= max_retry	status	agent
0	0	false → transient	TRANSIENT_INFRA_ERROR	skipped
1, 2	1, 2	false → transient	TRANSIENT_INFRA_ERROR	skipped
3	3	true	ERROR	triggers

That is #65907 working as intended — one recorded result, and one analysis, per
build. No num_retries value changes it. The multiple-attempt case above was
therefore reached with a manual retry: once the automatic retries are exhausted at
retry_count=3, a manual retry gives retry_count=4, so 4 >= 3 holds and the
attempt reports a real error and triggers. That is also the realistic case, since
a manual retry is what a human does while debugging a failure.

Not duplicating existing work

gh pr list --repo ray-project/ray --state open --search "buildkite annotation observability agent"
gh pr list --repo ray-project/ray --state open --search "buildkite-agent annotate"

Neither returns anything in this area beyond the two PRs this one is stacked on.

Tests

$ bazel test --test_tag_filters=release_unit //release/... --test_output=errors
Executed 29 out of 29 tests: 29 tests pass.

$ bazel test //release:test_observability_agent_reporter
//release:test_observability_agent_reporter                              PASSED
# 31 tests, 6 new for the annotation

New coverage:

Behavior	Test
argv: job scope, context keyed on the test name, style, append, --job for attribution	test_the_annotation_is_job_scoped_and_keyed_on_the_test
Attempt number matches buildkite's own label	test_the_attempt_number_matches_the_buildkite_label
The agent's prose cannot inject markup into the body	test_the_annotation_escapes_what_the_agent_sent
No annotation off a buildkite agent	test_no_annotation_outside_buildkite
A missing buildkite-agent binary is swallowed	test_annotation_failures_never_propagate
A non-zero exit is logged, not raised	test_a_non_zero_annotate_exit_is_logged_not_raised

Two bugs the tests caught while writing them: --job was appended after the
body, leaving the job id as the positional argument rather than the annotation
text; and the guard was "BUILDKITE" not in os.environ, which treats
BUILDKITE="" as being on an agent. The guard now matches byod/build.py.

AI assistance

AI assistance (Claude Code) was used for this change.

[gemini-code-assist]

Code Review

This pull request integrates the observability agent with Buildkite annotations, allowing test failure summaries to be appended directly to Buildkite jobs. It configures the Docker plugin to mount the buildkite-agent binary and implements the annotation logic in the reporter, along with temporary mock responses and forced failures for testing. The review feedback suggests avoiding the use of !r (repr) inside HTML attribute values within the annotation formatting, as it can lead to fragile HTML rendering, and recommends using explicit double quotes instead.

[sai-miduthuri]

The Observability Agent response can be viewed in the logs of the forced hello_world.aws test failure on buildkite/release on commit 3820a26, and buildkite/release on commit 79ff865.

The agent's summary and thread URL are printed in the buildkite job-level annotations and the build-level annotation, and also given their own Observability agent analysis group, in the Buildkite job logs on the 4th and 5th test attempts.

The 4th attempt is the final automated retry of the test, and the 5th attempt is a manual retry of the test. Attempts 1-3 do not query the Observability Agent because they are treated as having failed due to transient errors.

[sai-miduthuri]

Actual Observability Agent reports are visible at buildkite/release for commit 4792b54.

[elliot-barn]

Approved! But can you update the pr description that mentions the annotation showing up on the build page

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cb49f5e. Configure here.}