Skip to content

Proposal: optional phase annotations for shared test items - #44

Open
Azzaare wants to merge 1 commit into
julia-testitems:mainfrom
Azzaare:feature/test-item-phases
Open

Azzaare wants to merge 1 commit into
julia-testitems:mainfrom
Azzaare:feature/test-item-phases

Conversation

@Azzaare

@Azzaare Azzaare commented Oct 17, 2026

Copy link
Copy Markdown

A test item can already be shared across functional tests and performance tooling, but the tool cannot distinguish imports/input construction, explicit warm-up, and the operation to measure. Timing the entire item includes setup and assertions; importing a measurement backend inside the item couples otherwise ordinary tests to that backend.

Would lightweight phase annotations belong in TestItems? This PR proposes @prepare, @warmup, and @measure inside an item:

using TestItems

@testitem "Sort a vector" begin
    using TestItems: @prepare, @warmup, @measure

    @prepare begin
        values = [3, 1, 2]
    end
    @warmup sort(values)
    @measure sorted = sort(values)

    @test sorted == [1, 2, 3]
    @test values == [3, 1, 2]
end

During ordinary execution, each macro expands to its original body. All three bodies run once in source order, including explicit warm-up. Bindings, imports, return values, exceptions and control flow retain their ordinary semantics. There is no injected runtime mode check, timer, closure, repetition policy or backend loading. Loading the package and expanding macros still have their normal costs.

For external tools, with_phase_transform(transform) do ... end installs a task-local expansion-time callback receiving (phase, body, source, caller) and returning replacement syntax. A consumer can wrap the existing TestItemRunner.run_tests call and instrument freshly evaluated item bodies. No TestItemRunner changes or new package dependencies are included. Contexts are nested and restored on exceptions; they are not automatically propagated to child tasks/processes.

The hook does not instrument already compiled code, and annotating an expression directly inside the do block is too early: it has already expanded before the context is installed. The README and docstrings explain this boundary. Measurement policy and rejecting invalid/incomplete results remain the consumer's responsibility.

The implementation comes from a small, local, dependency-free TestItemPhases prototype. That prototype has not been registered in General; the intention is to discuss incorporating the neutral annotations here rather than introduce a competing registered package. There is no registration or release request in this PR.

Validation:

  • Local Pkg.test("TestItems") on Linux x86_64: 75/75 passing on Julia 1.12.7 and 1.13.1, each with the unchanged TestItemRunner 1.3.2.
  • Existing declaration-macro tests are retained; Project.toml, runner code and CI configuration are unchanged.
  • git diff --check passes. The full supported Julia-version matrix has not been run locally.

The tests cover transparent expansion, scope/imports, values, exceptions and nonlocal control flow; task isolation, nested/disabled contexts and exception restoration; and discovery plus ordinary/instrumented execution through an unmodified TestItemRunner.

I would welcome feedback on whether these annotations fit TestItems' scope, their names, and the expansion-time consumer interface. This proposes the neutral boundary only, without committing TestItems to a measurement backend.

@Azzaare Azzaare closed this Oct 17, 2026
@Azzaare Azzaare reopened this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant