Proposal: optional phase annotations for shared test items

LICENSE

@@ -1,6 +1,7 @@
 MIT License
 
 Copyright (c) 2022 David Anthoff
+Copyright (c) 2026 Jean-Francois Baffier
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -8,3 +8,90 @@
 
 This package provides the `@testitem` macro for the test runner feature in VS Code. You can learn more over at the [documentation](https://www.julia-vscode.org/docs/stable/userguide/testitems/) for the entire test item ecosystem. Note that while the documentation is hosted as part of the Julia VS Code extension, the test item framework is not a VS Code extension specific technology and can be used entirely without VS Code.
 
+## Preparation, warm-up and measurement regions
+
+Optional `@prepare`, `@warmup` and `@measure` annotations let a test item share
+its correctness checks with external performance tools, without depending on a
+particular measurement package:
+
+```julia
+using TestItems
+
+@testitem "Sort a vector" begin
+    using TestItems: @prepare, @warmup, @measure
+
+    @prepare begin
+        # Imports, input construction and other preparation.
+        values = [3, 1, 2]
+    end
+
+    @warmup sort(values)
+
+    @measure begin
+        sorted = sort(values)
+    end
+
+    @test sorted == [1, 2, 3]
+    @test values == [3, 1, 2]
+end
+```
+
+In a normal functional run, **every body executes once in source order**.
+Explicit warm-up code is not skipped, and no hidden repetitions are added. The
+annotations expand to their original expression: they add no closure, scope,
+timer, runtime callback or runtime mode check. Bindings, imports, values,
+exceptions, `return`, `break` and `continue` retain their ordinary behavior.
+Package loading and macro expansion still have their normal Julia costs.
+
+Each annotation takes one expression, including a `begin ... end` block.
+An external tool can use the annotations to exclude preparation and warm-up
+from measurement. Keep assertions outside `@measure` if their cost should be excluded.
+These annotations do not add any package dependencies or require changes to
+TestItemRunner. They must be in scope in the item body, for example through the
+normal package imports or an explicit `using TestItems`.
+
+### Interface for external tools
+
+`with_phase_transform` allows a consumer to replace phase syntax while fresh
+item code is being expanded. For example, this observes the annotations without
+changing their execution:
+
+```julia
+using TestItems: with_phase_transform
+using TestItemRunner
+
+regions = Symbol[]
+transform = function (phase, body, source, caller)
+    push!(regions, phase)  # Expansion-time observation, not runtime timing.
+    return body
+end
+
+with_phase_transform(transform) do
+    TestItemRunner.run_tests("test")
+end
+```
+
+The callback receives the phase (`:prepare`, `:warmup` or `:measure`), raw body
+expression, call-site `LineNumberNode` and caller `Module`. It returns replacement
+syntax **without** pre-escaping it; the phase macro escapes it into the caller's
+scope. A consumer inserting calls to its own helpers should use `GlobalRef` and
+preserve user bindings, control flow, values and exceptions. Measurement policy,
+including handling failed or incomplete measurements, belongs to that consumer.
+TestItems itself supplies no timer, repetitions or performance verdict.
+
+The context is task-local and restored on return or error. Nested contexts are
+supported, and `with_phase_transform(nothing) do ... end` temporarily disables a
+transform. The context is not automatically propagated to child tasks or other
+processes. TestItemRunner's discovery does not expand or execute item bodies.
+
+This is an **expansion-time** interface. Wrap `include`, `include_string`,
+`macroexpand` or a runner evaluating fresh bodies; wrapping an already compiled
+function does not instrument it. In particular, an annotation written directly
+inside the `do` block is expanded before the context is installed:
+
+```julia
+with_phase_transform(transform) do
+    @measure sort([3, 1, 2])  # Already expanded; remains transparent.
+end
+```
+

src/TestItems.jl

@@ -1,6 +1,9 @@
 module TestItems
 
 export @testitem, @testmodule, @testsnippet
+export @prepare, @warmup, @measure, with_phase_transform
+
+include("phases.jl")
 
 """
     @testitem "name" [keyword_args...] begin

src/phases.jl

@@ -0,0 +1,85 @@
+const _PHASE_TRANSFORM_KEY = gensym(:TestItems_phase_transform)
+
+"""
+    with_phase_transform(f, transform)
+    with_phase_transform(transform) do
+        # Expand and evaluate fresh test-item code here.
+    end
+
+Call `f()` with an expansion-time phase transform in the current task. The
+previous transform is restored on normal return and on exceptions. Pass `nothing`
+to temporarily restore transparent expansion. The return value is that of `f()`.
+
+When a phase macro expands, `transform(phase, body, source, caller)` receives:
+
+- `phase`: one of `:prepare`, `:warmup` or `:measure`;
+- `body`: the unevaluated expression supplied to the macro;
+- `source::LineNumberNode`: the macro's call site;
+- `caller::Module`: the module in which it expands.
+
+Return replacement Julia syntax, not a pre-escaped expression. The macro escapes
+the replacement into the caller's scope. A consumer must preserve the intended
+scope, values, control flow and exception behavior of the user code.
+
+The transform runs during **macro expansion**, never as an implicit runtime
+callback. Wrap `include`, `include_string`, `macroexpand`, or a runner that
+evaluates fresh item bodies. Already expanded or compiled code is unchanged;
+wrapping a call to such a function does not instrument it. The transform is
+task-local, is not installed globally, and is not automatically propagated to
+other tasks or processes.
+
+The package supplies no timer, repetition policy or performance backend.
+"""
+function with_phase_transform(f::Function, transform)
+    task_local_storage(f, _PHASE_TRANSFORM_KEY, transform)
+end
+
+function _expand_phase(phase::Symbol, body, source::LineNumberNode, caller::Module)
+    transform = get(task_local_storage(), _PHASE_TRANSFORM_KEY, nothing)
+    replacement = transform === nothing ? body : transform(phase, body, source, caller)
+    esc(replacement)
+end
+
+"""
+    @prepare expression
+    @prepare begin
+        # Imports, input construction and other preparation.
+    end
+
+Mark preparation code. Normally this expands directly to `expression`, evaluated
+once in the caller's existing scope, with its original value and exceptions.
+Top-level imports and bindings remain usable by later phases and assertions.
+No closure, local scope, timer or runtime callback is added.
+"""
+macro prepare(body)
+    _expand_phase(:prepare, body, __source__, __module__)
+end
+
+"""
+    @warmup expression
+    @warmup begin
+        # Explicit warm-up operations.
+    end
+
+Mark explicit warm-up code. Normally its body runs once, in source order, just
+as if the marker were absent. It is not skipped and no hidden repetitions are
+added. A performance consumer can interpret this region through an explicit
+expansion-time transform; functional runs keep the ordinary code semantics.
+"""
+macro warmup(body)
+    _expand_phase(:warmup, body, __source__, __module__)
+end
+
+"""
+    @measure expression
+    @measure begin
+        # The operation selected for measurement by an external tool.
+    end
+
+Mark a measurement region. Without an explicit transform this is only an
+annotation: it executes the original body once and returns its value, without
+measuring anything. Bindings remain visible to subsequent correctness checks.
+"""
+macro measure(body)
+    _expand_phase(:measure, body, __source__, __module__)
+end