fix(rocProfiler-SDK): buffered-tracing race + thread_trace API sync - #514
Open
zichguan-amd wants to merge 3 commits into
Open
fix(rocProfiler-SDK): buffered-tracing race + thread_trace API sync#514zichguan-amd wants to merge 3 commits into
zichguan-amd wants to merge 3 commits into
Conversation
…ng data race) Bring several rocProfiler-SDK examples back in line with the upstream samples in ROCm/rocm-systems (projects/rocprofiler-sdk/samples); the copies here predated these fixes. - api_buffered_tracing: guard the shared call_stack_t (tool_data) vector with a mutex. It is written concurrently from the asynchronous buffer-flush thread and from the internal thread-creation callbacks (thread_precreate/thread_postcreate) with no synchronization, so overlapping reallocations corrupt the heap and abort with "double free or corruption" during the first kernel launch. This was the root cause of the scattered, timing-dependent CI aborts of rocprofiler-sdk_api_buffered_tracing. Verified on a repro host: original crashed 5/5, fixed passed 10/10. - Common/rocprofiler_utils.hpp: make ROCPROFILER_CHECK/ROCPROFILER_CALL use a __LINE__-uniqued internal variable (ROCPROFILER_VARIABLE(CHECKSTATUS, __LINE__)), matching upstream's defines.hpp and the existing ROCPROFILER_WARN. The previous plain `status` collided with a caller variable named status, so wrapping rocprofiler_is_initialized(&status) would not compile / self-initialized. - pc_sampling: check the real status from rocprofiler_destroy_buffer instead of the constant ROCPROFILER_STATUS_SUCCESS (which made the check a no-op that swallowed non-busy errors); fix "availabe" typo. - api_callback_tracing, openmp_target: wrap rocprofiler_is_initialized() in ROCPROFILER_CALL so a failed query is not silently ignored. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
… API
The rocprofiler-sdk experimental thread-trace API changed the shader-data callback
contract without a version bump: rocprofiler_thread_trace_shader_data_callback_t
now takes a single rocprofiler_thread_trace_shader_data_t struct. The example's old
5-argument callback no longer compiles against ROCm's headers. Sync with the
upstream sample (ROCm/rocm-systems agent.cpp):
- shader_data_callback: take rocprofiler_thread_trace_shader_data_t; decode from
shader_data.data + read_offset for shader_data.data_size - read_offset bytes.
- tool_codeobj_tracing_callback: only handle ROCPROFILER_CALLBACK_PHASE_LOAD.
- brace-init the thread-trace parameter union values.
thread_trace was previously commented out of the suite CMake ("Disabled until
feature is released"), so the API break was not caught by CI. Now that the example
builds against the current SDK, enable it in the suite (add_subdirectory) so it is
covered going forward. Verified with a full suite cmake build.
The updated thread_trace example targets the new single-struct thread-trace shader-data callback (rocprofiler_thread_trace_shader_data_t), which is only available since ROCm 10. The pinned 7.14 stable CI image still ships the older multi-arg callback and has no such struct, so the example fails to compile there. Add a stable-channel build skip in the skip manifest so the pinned 7.14 native workflows drop thread_trace, while nightly TheRock builds (which carry the new API) still build it. Annotate the CMake add_subdirectory with the availability. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes bringing the rocProfiler-SDK examples back in line with the upstream samples (ROCm/rocm-systems,
projects/rocprofiler-sdk/samples), which had diverged.1. Fix buffered-tracing data race + sync examples with upstream
call_stack_t(tool_data) vector with a mutex. It is written concurrently from the async buffer-flush thread and thethread_precreate/thread_postcreatecallbacks with no synchronization, so overlapping reallocations corrupt the heap and abort with "double free or corruption" on the first kernel launch. This was the root cause of the timing-dependent CI aborts ofrocprofiler-sdk_api_buffered_tracing. Verified on a repro host: original crashed 5/5, fixed passed 10/10.ROCPROFILER_CHECK/ROCPROFILER_CALLuse a__LINE__-uniqued internal variable (ROCPROFILER_VARIABLE(CHECKSTATUS, __LINE__)), matching upstream'sdefines.hppand the existingROCPROFILER_WARN. The previous plainstatuscollided with a caller variable namedstatus.rocprofiler_destroy_bufferinstead of the constantROCPROFILER_STATUS_SUCCESS(which made the check a no-op); fix "availabe" typo.rocprofiler_is_initialized()inROCPROFILER_CALLso a failed query is not silently ignored (relies on the macro-hygiene fix above).2. Update thread_trace to current SDK thread-trace API
rocprofiler_thread_trace_shader_data_callback_tnow takes a singlerocprofiler_thread_trace_shader_data_tstruct. This struct-based form is available since ROCm 10; the pinned 7.14 stable image still ships the older multi-argument callback (agent, shader_engine_id, data, data_size, flags, userdata) and has no such struct, so the updated example does not compile there.shader_data_callback: take the struct; decode fromshader_data.data + read_offsetforshader_data.data_size - read_offsetbytes.tool_codeobj_tracing_callback: only handleROCPROFILER_CALLBACK_PHASE_LOAD.thread_tracein the suite CMake (add_subdirectory). It was previously commented out ("Disabled until feature is released"), which is why the API break escaped CI. It now builds on the nightly TheRock channel (which carries the new API).thread_traceon the pinned 7.14 stable image via a stable-channel build entry in the skip manifest (.github/build_tools/skip_manifest.py), since that image predates the new struct API. Nightly builds are unaffected and still compile it.Test plan
-DROCM_EXAMPLES_ENABLE_OPENMP=ON— configure + build exit 0, all targets includingrocprofiler-sdk_thread_traceproduced.--channel stablelistsLibraries/rocProfiler-SDK/thread_traceinskip_build.txtand the CMake override drops it (SkipExamples: skipping ...);--channel nightlydoes not skip it.🤖 Generated with Claude Code