Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A test item can already be shared across functional tests and performance tooling, but the tool cannot distinguish imports/input construction, explicit warm-up, and the operation to measure. Timing the entire item includes setup and assertions; importing a measurement backend inside the item couples otherwise ordinary tests to that backend.
Would lightweight phase annotations belong in TestItems? This PR proposes
@prepare,@warmup, and@measureinside an item:During ordinary execution, each macro expands to its original body. All three bodies run once in source order, including explicit warm-up. Bindings, imports, return values, exceptions and control flow retain their ordinary semantics. There is no injected runtime mode check, timer, closure, repetition policy or backend loading. Loading the package and expanding macros still have their normal costs.
For external tools,
with_phase_transform(transform) do ... endinstalls a task-local expansion-time callback receiving(phase, body, source, caller)and returning replacement syntax. A consumer can wrap the existingTestItemRunner.run_testscall and instrument freshly evaluated item bodies. No TestItemRunner changes or new package dependencies are included. Contexts are nested and restored on exceptions; they are not automatically propagated to child tasks/processes.The hook does not instrument already compiled code, and annotating an expression directly inside the
doblock is too early: it has already expanded before the context is installed. The README and docstrings explain this boundary. Measurement policy and rejecting invalid/incomplete results remain the consumer's responsibility.The implementation comes from a small, local, dependency-free TestItemPhases prototype. That prototype has not been registered in General; the intention is to discuss incorporating the neutral annotations here rather than introduce a competing registered package. There is no registration or release request in this PR.
Validation:
Pkg.test("TestItems")on Linux x86_64: 75/75 passing on Julia 1.12.7 and 1.13.1, each with the unchanged TestItemRunner 1.3.2.Project.toml, runner code and CI configuration are unchanged.git diff --checkpasses. The full supported Julia-version matrix has not been run locally.The tests cover transparent expansion, scope/imports, values, exceptions and nonlocal control flow; task isolation, nested/disabled contexts and exception restoration; and discovery plus ordinary/instrumented execution through an unmodified TestItemRunner.
I would welcome feedback on whether these annotations fit TestItems' scope, their names, and the expansion-time consumer interface. This proposes the neutral boundary only, without committing TestItems to a measurement backend.