Skip to content

fix(rocProfiler-SDK): buffered-tracing race + thread_trace API sync - #514

Open
zichguan-amd wants to merge 3 commits into
amd-stagingfrom
zichguan/rocprofiler-buffered-tracing-race
Open

fix(rocProfiler-SDK): buffered-tracing race + thread_trace API sync#514
zichguan-amd wants to merge 3 commits into
amd-stagingfrom
zichguan/rocprofiler-buffered-tracing-race

Conversation

@zichguan-amd

@zichguan-amd zichguan-amd commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two related fixes bringing the rocProfiler-SDK examples back in line with the upstream samples (ROCm/rocm-systems, projects/rocprofiler-sdk/samples), which had diverged.

1. Fix buffered-tracing data race + sync examples with upstream

  • api_buffered_tracing: guard the shared call_stack_t (tool_data) vector with a mutex. It is written concurrently from the async buffer-flush thread and the thread_precreate/thread_postcreate callbacks with no synchronization, so overlapping reallocations corrupt the heap and abort with "double free or corruption" on the first kernel launch. This was the root cause of the timing-dependent CI aborts of rocprofiler-sdk_api_buffered_tracing. Verified on a repro host: original crashed 5/5, fixed passed 10/10.
  • Common/rocprofiler_utils.hpp: make ROCPROFILER_CHECK/ROCPROFILER_CALL use a __LINE__-uniqued internal variable (ROCPROFILER_VARIABLE(CHECKSTATUS, __LINE__)), matching upstream's defines.hpp and the existing ROCPROFILER_WARN. The previous plain status collided with a caller variable named status.
  • pc_sampling: check the real status from rocprofiler_destroy_buffer instead of the constant ROCPROFILER_STATUS_SUCCESS (which made the check a no-op); fix "availabe" typo.
  • api_callback_tracing, openmp_target: wrap rocprofiler_is_initialized() in ROCPROFILER_CALL so a failed query is not silently ignored (relies on the macro-hygiene fix above).

2. Update thread_trace to current SDK thread-trace API

  • The experimental thread-trace shader-data callback contract changed between ROCm releases: rocprofiler_thread_trace_shader_data_callback_t now takes a single rocprofiler_thread_trace_shader_data_t struct. This struct-based form is available since ROCm 10; the pinned 7.14 stable image still ships the older multi-argument callback (agent, shader_engine_id, data, data_size, flags, userdata) and has no such struct, so the updated example does not compile there.
  • shader_data_callback: take the struct; decode from shader_data.data + read_offset for shader_data.data_size - read_offset bytes.
  • tool_codeobj_tracing_callback: only handle ROCPROFILER_CALLBACK_PHASE_LOAD.
  • Brace-init the thread-trace parameter union values.
  • Enable thread_trace in the suite CMake (add_subdirectory). It was previously commented out ("Disabled until feature is released"), which is why the API break escaped CI. It now builds on the nightly TheRock channel (which carries the new API).
  • Skip thread_trace on the pinned 7.14 stable image via a stable-channel build entry in the skip manifest (.github/build_tools/skip_manifest.py), since that image predates the new struct API. Nightly builds are unaffected and still compile it.

Test plan

  • Full-suite cmake build with -DROCM_EXAMPLES_ENABLE_OPENMP=ON — configure + build exit 0, all targets including rocprofiler-sdk_thread_trace produced.
  • Skip manifest: --channel stable lists Libraries/rocProfiler-SDK/thread_trace in skip_build.txt and the CMake override drops it (SkipExamples: skipping ...); --channel nightly does not skip it.
  • api_buffered_tracing race repro: 5/5 crash before, 10/10 pass after.
  • CI green: nightly builds thread_trace, 7.14 stable skips it.

🤖 Generated with Claude Code

zichguan-amd and others added 2 commits September 3, 2026 11:08
…ng data race)

Bring several rocProfiler-SDK examples back in line with the upstream samples in
ROCm/rocm-systems (projects/rocprofiler-sdk/samples); the copies here predated
these fixes.

- api_buffered_tracing: guard the shared call_stack_t (tool_data) vector with a
  mutex. It is written concurrently from the asynchronous buffer-flush thread and
  from the internal thread-creation callbacks (thread_precreate/thread_postcreate)
  with no synchronization, so overlapping reallocations corrupt the heap and abort
  with "double free or corruption" during the first kernel launch. This was the
  root cause of the scattered, timing-dependent CI aborts of
  rocprofiler-sdk_api_buffered_tracing. Verified on a repro host: original crashed
  5/5, fixed passed 10/10.
- Common/rocprofiler_utils.hpp: make ROCPROFILER_CHECK/ROCPROFILER_CALL use a
  __LINE__-uniqued internal variable (ROCPROFILER_VARIABLE(CHECKSTATUS, __LINE__)),
  matching upstream's defines.hpp and the existing ROCPROFILER_WARN. The previous
  plain `status` collided with a caller variable named status, so wrapping
  rocprofiler_is_initialized(&status) would not compile / self-initialized.
- pc_sampling: check the real status from rocprofiler_destroy_buffer instead of
  the constant ROCPROFILER_STATUS_SUCCESS (which made the check a no-op that
  swallowed non-busy errors); fix "availabe" typo.
- api_callback_tracing, openmp_target: wrap rocprofiler_is_initialized() in
  ROCPROFILER_CALL so a failed query is not silently ignored.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
… API

The rocprofiler-sdk experimental thread-trace API changed the shader-data callback
contract without a version bump: rocprofiler_thread_trace_shader_data_callback_t
now takes a single rocprofiler_thread_trace_shader_data_t struct. The example's old
5-argument callback no longer compiles against ROCm's headers. Sync with the
upstream sample (ROCm/rocm-systems agent.cpp):

- shader_data_callback: take rocprofiler_thread_trace_shader_data_t; decode from
  shader_data.data + read_offset for shader_data.data_size - read_offset bytes.
- tool_codeobj_tracing_callback: only handle ROCPROFILER_CALLBACK_PHASE_LOAD.
- brace-init the thread-trace parameter union values.

thread_trace was previously commented out of the suite CMake ("Disabled until
feature is released"), so the API break was not caught by CI. Now that the example
builds against the current SDK, enable it in the suite (add_subdirectory) so it is
covered going forward. Verified with a full suite cmake build.
@zichguan-amd
zichguan-amd requested a review from a team as a code owner September 3, 2026 15:24
The updated thread_trace example targets the new single-struct thread-trace
shader-data callback (rocprofiler_thread_trace_shader_data_t), which is only
available since ROCm 10. The pinned 7.14 stable CI image still ships the older
multi-arg callback and has no such struct, so the example fails to compile there.

Add a stable-channel build skip in the skip manifest so the pinned 7.14 native
workflows drop thread_trace, while nightly TheRock builds (which carry the new
API) still build it. Annotate the CMake add_subdirectory with the availability.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant