Skip to content

fix: preserve symbolic external units across precompile - #215

Open
MilesCranmerBot wants to merge 17 commits into
JuliaPhysics:mainfrom
MilesCranmerBot:fix/issue-212-precompile
Open

fix: preserve symbolic external units across precompile#215
MilesCranmerBot wants to merge 17 commits into
JuliaPhysics:mainfrom
MilesCranmerBot:fix/issue-212-precompile

Conversation

@MilesCranmerBot

Copy link
Copy Markdown
Contributor

Summary

Preserve external symbolic unit registrations across precompile and import boundaries.

Changes

  • thread the caller module through us"..." symbolic macro expansion
  • lazily re-register external bindings when a symbolic unit exists in the caller module but the symbolic registry has not been rebuilt yet
  • add a regression test covering the precompile/import case for external units

Testing

  • /usr/local/bin/julia --project -e 'using Pkg; Pkg.test()'

Fixes #212

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@github-actions

github-actions Bot commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (Julia v1.10)

Time benchmarks
main dbe1403... main / dbe1403...
Quantity/creation/Quantity(x) 3.13 ± 0.01 ns 3.54 ± 0.01 ns 0.887 ± 0.0038
Quantity/creation/Quantity(x, length=y) 3.48 ± 0.01 ns 4.64 ± 0.039 ns 0.752 ± 0.0067
Quantity/with_numbers/*real 3.14 ± 0.01 ns 3.48 ± 0.01 ns 0.902 ± 0.0039
Quantity/with_numbers/^int 8.39 ± 2.4 ns 8.39 ± 2.4 ns 1 ± 0.41
Quantity/with_numbers/^int * real 8.39 ± 2.8 ns 8.39 ± 2.4 ns 1 ± 0.44
Quantity/with_quantity/+y 4.19 ± 0.11 ns 4.15 ± 0.07 ns 1.01 ± 0.032
Quantity/with_quantity//y 3.5 ± 0.01 ns 3.5 ± 0.01 ns 1 ± 0.004
Quantity/with_self/dimension 3.13 ± 0.01 ns 3.48 ± 0.01 ns 0.9 ± 0.0039
Quantity/with_self/inv 3.48 ± 0.01 ns 3.48 ± 0.01 ns 1 ± 0.0041
Quantity/with_self/ustrip 3.13 ± 0.001 ns 3.48 ± 0.01 ns 0.9 ± 0.0026
QuantityArray/broadcasting/multi_array_of_quantities 0.153 ± 0.01 ms 0.149 ± 0.012 ms 1.02 ± 0.1
QuantityArray/broadcasting/multi_normal_array 0.0598 ± 0.0035 ms 0.06 ± 0.0035 ms 0.997 ± 0.082
QuantityArray/broadcasting/multi_quantity_array 0.173 ± 0.0044 ms 0.173 ± 0.003 ms 0.999 ± 0.031
QuantityArray/broadcasting/x^2_array_of_quantities 29.8 ± 2.1 μs 30.1 ± 2.3 μs 0.988 ± 0.1
QuantityArray/broadcasting/x^2_normal_array 4.91 ± 0.56 μs 4.89 ± 0.6 μs 1 ± 0.17
QuantityArray/broadcasting/x^2_quantity_array 9.08 ± 0.52 μs 8.96 ± 0.73 μs 1.01 ± 0.1
QuantityArray/broadcasting/x^4_array_of_quantities 0.0902 ± 0.0021 ms 0.0907 ± 0.0022 ms 0.995 ± 0.033
QuantityArray/broadcasting/x^4_normal_array 0.0561 ± 0.0035 ms 0.0562 ± 0.00035 ms 0.998 ± 0.062
QuantityArray/broadcasting/x^4_quantity_array 0.0532 ± 0.0035 ms 0.0562 ± 0.0035 ms 0.946 ± 0.085
time_to_load 0.216 ± 0.0027 s 0.215 ± 0.0024 s 1 ± 0.017
Memory benchmarks
main dbe1403... main / dbe1403...
Quantity/creation/Quantity(x) 0 allocs: 0 B 0 allocs: 0 B
Quantity/creation/Quantity(x, length=y) 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_numbers/*real 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_numbers/^int 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_numbers/^int * real 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_quantity/+y 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_quantity//y 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_self/dimension 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_self/inv 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_self/ustrip 0 allocs: 0 B 0 allocs: 0 B
QuantityArray/broadcasting/multi_array_of_quantities 2 allocs: 0.382 MB 2 allocs: 0.382 MB 1
QuantityArray/broadcasting/multi_normal_array 2 allocs: 0.0763 MB 2 allocs: 0.0763 MB 1
QuantityArray/broadcasting/multi_quantity_array 2 allocs: 0.0763 MB 2 allocs: 0.0763 MB 1
QuantityArray/broadcasting/x^2_array_of_quantities 2 allocs: 0.382 MB 2 allocs: 0.382 MB 1
QuantityArray/broadcasting/x^2_normal_array 2 allocs: 0.0763 MB 2 allocs: 0.0763 MB 1
QuantityArray/broadcasting/x^2_quantity_array 2 allocs: 0.0763 MB 2 allocs: 0.0763 MB 1
QuantityArray/broadcasting/x^4_array_of_quantities 2 allocs: 0.382 MB 2 allocs: 0.382 MB 1
QuantityArray/broadcasting/x^4_normal_array 2 allocs: 0.0763 MB 2 allocs: 0.0763 MB 1
QuantityArray/broadcasting/x^4_quantity_array 2 allocs: 0.0763 MB 2 allocs: 0.0763 MB 1
time_to_load 0.153 k allocs: 14.5 kB 0.153 k allocs: 14.5 kB 1

@github-actions

github-actions Bot commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (Julia v1)

Time benchmarks
main dbe1403... main / dbe1403...
Quantity/creation/Quantity(x) 3.13 ± 0.01 ns 3.13 ± 0.009 ns 1 ± 0.0043
Quantity/creation/Quantity(x, length=y) 3.48 ± 0.01 ns 3.48 ± 0.01 ns 1 ± 0.0041
Quantity/with_numbers/*real 3.49 ± 0.01 ns 3.48 ± 0.001 ns 1 ± 0.0029
Quantity/with_numbers/^int 13.6 ± 1 ns 13.6 ± 1 ns 1 ± 0.11
Quantity/with_numbers/^int * real 13.3 ± 1.1 ns 13.3 ± 1.1 ns 1 ± 0.11
Quantity/with_quantity/+y 4.44 ± 0.031 ns 4.41 ± 0.02 ns 1.01 ± 0.0084
Quantity/with_quantity//y 3.5 ± 0.01 ns 3.5 ± 0.01 ns 1 ± 0.004
Quantity/with_self/dimension 3.13 ± 0.01 ns 3.48 ± 0 ns 0.9 ± 0.0029
Quantity/with_self/inv 3.48 ± 0.01 ns 4.52 ± 1 ns 0.772 ± 0.18
Quantity/with_self/ustrip 3.13 ± 0.001 ns 3.13 ± 0.001 ns 1 ± 0.00045
QuantityArray/broadcasting/multi_array_of_quantities 0.118 ± 0.002 ms 0.118 ± 0.005 ms 0.999 ± 0.045
QuantityArray/broadcasting/multi_normal_array 0.0525 ± 0.0033 ms 0.0526 ± 0.00026 ms 0.999 ± 0.063
QuantityArray/broadcasting/multi_quantity_array 0.0528 ± 0.00024 ms 0.0563 ± 0.00033 ms 0.938 ± 0.007
QuantityArray/broadcasting/x^2_array_of_quantities 29.5 ± 12 μs 29.1 ± 7.2 μs 1.01 ± 0.48
QuantityArray/broadcasting/x^2_normal_array 2.76 ± 1.7 μs 2.8 ± 1.9 μs 0.986 ± 0.89
QuantityArray/broadcasting/x^2_quantity_array 4.13 ± 1.4 μs 7.37 ± 1 μs 0.56 ± 0.2
QuantityArray/broadcasting/x^4_array_of_quantities 0.0895 ± 0.0045 ms 0.0899 ± 0.0049 ms 0.996 ± 0.074
QuantityArray/broadcasting/x^4_normal_array 0.0456 ± 0.00015 ms 0.0491 ± 0.00022 ms 0.929 ± 0.0052
QuantityArray/broadcasting/x^4_quantity_array 0.0527 ± 0.00017 ms 0.0493 ± 0.00019 ms 1.07 ± 0.0054
time_to_load 0.19 ± 0.0013 s 0.189 ± 0.0017 s 1 ± 0.011
Memory benchmarks
main dbe1403... main / dbe1403...
Quantity/creation/Quantity(x) 0 allocs: 0 B 0 allocs: 0 B
Quantity/creation/Quantity(x, length=y) 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_numbers/*real 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_numbers/^int 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_numbers/^int * real 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_quantity/+y 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_quantity//y 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_self/dimension 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_self/inv 0 allocs: 0 B 0 allocs: 0 B
Quantity/with_self/ustrip 0 allocs: 0 B 0 allocs: 0 B
QuantityArray/broadcasting/multi_array_of_quantities 3 allocs: 0.382 MB 3 allocs: 0.382 MB 1
QuantityArray/broadcasting/multi_normal_array 3 allocs: 0.0764 MB 3 allocs: 0.0764 MB 1
QuantityArray/broadcasting/multi_quantity_array 3 allocs: 0.0764 MB 3 allocs: 0.0764 MB 1
QuantityArray/broadcasting/x^2_array_of_quantities 3 allocs: 0.382 MB 3 allocs: 0.382 MB 1
QuantityArray/broadcasting/x^2_normal_array 3 allocs: 0.0764 MB 3 allocs: 0.0764 MB 1
QuantityArray/broadcasting/x^2_quantity_array 3 allocs: 0.0764 MB 3 allocs: 0.0764 MB 1
QuantityArray/broadcasting/x^4_array_of_quantities 3 allocs: 0.382 MB 3 allocs: 0.382 MB 1
QuantityArray/broadcasting/x^4_normal_array 3 allocs: 0.0764 MB 3 allocs: 0.0764 MB 1
QuantityArray/broadcasting/x^4_quantity_array 3 allocs: 0.0764 MB 3 allocs: 0.0764 MB 1
time_to_load 0.145 k allocs: 11 kB 0.145 k allocs: 11 kB 1

@codecov

codecov Bot commented Apr 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.98%. Comparing base (c2098b5) to head (dbe1403).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #215      +/-   ##
==========================================
- Coverage   99.15%   98.98%   -0.18%     
==========================================
  Files          23       23              
  Lines        1305     1373      +68     
==========================================
+ Hits         1294     1359      +65     
- Misses         11       14       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MilesCranmer

Copy link
Copy Markdown
Member

@MilesCranmerBot we need to get to 100% coverage of the diff please

Comment thread test/precompile_test/ExternalUnitRegistration.jl Outdated
@MilesCranmer

Copy link
Copy Markdown
Member

@MilesCranmerBot code coverage is not yet 100% for the diff. And also see comment about making breaking changes please.

MilesCranmerBot and others added 2 commits April 22, 2026 18:32
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmer

Copy link
Copy Markdown
Member

@MilesCranmerBot needs to add a test where we have a fake package that registers units, and that has a function which uses those units. See the recent issue for the workflow I'd like to enable.

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmerBot

Copy link
Copy Markdown
Contributor Author

Done. I pushed this to the PR branch in commit 7645686.

What changed:

  • replaced the bare module test with a fake package layout under test/precompile_test/ExternalUnitRegistration/src/ExternalUnitRegistration.jl
  • kept the registered external unit plus a function that uses us"MyWb" from inside that package
  • updated the regression to check the imported package entrypoints after precompilation/import

Validation:

  • /usr/local/bin/julia --project -e 'using Pkg; Pkg.test()'

@MilesCranmer

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7645686b00

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/symbolic_dimensions.jl Outdated
Comment thread src/symbolic_dimensions.jl Outdated
@MilesCranmer

Copy link
Copy Markdown
Member

@MilesCranmerBot Please address the above feedback

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmerBot

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback in 524cb15.

What changed:

  • restored the Constants. guard so module-local quantity bindings no longer shadow symbolic constants like c
  • made lazy external-unit registration use the existing unit-update lock, avoiding duplicate first-use registration races
  • added regression tests for both cases

Validation:

  • /usr/local/bin/julia --project=. -e 'using Pkg; Pkg.test(test_args=["unittests"])'

Comment thread src/symbolic_dimensions.jl Outdated
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Comment thread src/symbolic_dimensions.jl Outdated
MilesCranmerBot and others added 2 commits April 23, 2026 13:42
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmer

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@MilesCranmer

Copy link
Copy Markdown
Member

Hm, I still think this feels too hacky... @MilesCranmerBot could you please change the PR in the following way? Here's a detailed description:

This can be fixed in a cleaner and more symmetric way.

The current PR appears to solve the immediate us"..." failure across precompile/import boundaries, but it does so by making only the symbolic parser caller-module-aware. That leaves u"..." and us"..." on different resolution paths:

  • u_str still expands through map_to_scope(Meta.parse(s))
  • us_str is changed to expand through map_to_scope(__module__, Meta.parse(s))

That asymmetry is hard to justify at the parser API level. External units should have one consistent resolution story, regardless of whether the target is ordinary or symbolic.

A cleaner fix would be:

  1. Make both macros caller-module-aware.

    • Change @u_str in src/uparse.jl to call map_to_scope(__module__, Meta.parse(s))
    • Keep the same for @us_str
  2. Add matching module-aware map_to_scope(mod::Module, ...) overloads on both the ordinary and symbolic sides.

    • map_to_scope(ex::Expr) = map_to_scope(@__MODULE__, ex)
    • map_to_scope(sym::Symbol) = map_to_scope(@__MODULE__, sym)
    • map_to_scope(mod::Module, ex::Expr) for recursive descent
    • map_to_scope(mod::Module, sym::Symbol) for symbol resolution
  3. Share the external-binding detection and lazy registration logic instead of implementing it only for symbolic parsing.

    • The external binding check should live in shared internal code
    • The lazy registration path should also be shared
    • Both u"..." and us"..." should be able to recover an external unit from the caller module when the binding exists but the registration state has not yet been rebuilt in the current import/precompile lifecycle
  4. Keep symbolic lookup independent of SYMBOLIC_UNIT_VALUES[i] cache alignment for correctness.

    • Reconstructing the symbolic quantity from the symbol is a reasonable approach
    • That part of the current PR looks directionally good
    • However, it should be part of a symmetric design rather than paired with a parser-only special case for us_str
  5. Add regression tests for both ordinary and symbolic macros through an external module boundary.

    • u"MyExternalUnit"
    • us"MyExternalUnit"
    • direct use
    • imported helper function use
    • precompile/import style loading
    • threaded first-use if concurrency is relevant

The key point is that the problem does not appear to be that symbolic registration should stop being updated. The issue is that symbolic parsing currently has a more fragile lifecycle around external registrations, especially across precompile/import boundaries. The fix should therefore make the registration and recovery story more robust, but without introducing a macro-level asymmetry between u_str and us_str.

In short, the suggested direction is:

  • make u_str and us_str symmetric
  • share external-unit recovery logic
  • keep symbolic lookup robust to cache rebuild timing
  • test both macros the same way

That should produce a clearer and more maintainable design than fixing only us_str in isolation.

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmer

Copy link
Copy Markdown
Member

@MilesCranmerBot CI is failing, please fix.

@MilesCranmer

Copy link
Copy Markdown
Member

@MilesCranmerBot code coverage of the diff is under 100%. Are there methods which aren't possible to hit in a realistic scenario? If so then please remove them. If it is possible to hit them, then ensure 100% diff coverage with testing.

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmerBot

Copy link
Copy Markdown
Contributor Author

Addressed this in a8912ae.

The uncovered diff lines were fallback branches in map_to_scope for u"..." / us"..." that looked reachable on paper, but they are not hit in the realistic external-unit flow this PR is fixing. I verified that trying to exercise them without a module binding still errors instead of going through that path, so I removed those dead branches rather than adding a contrived test.

I reran /usr/local/bin/julia --project=. -e 'import Pkg; Pkg.test(coverage=true)', and the full suite passed locally with the current diff lines covered.

Comment thread src/symbolic_dimensions.jl Outdated

@MilesCranmer MilesCranmer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MilesCranmerBot see new comments

Comment thread src/symbolic_dimensions.jl Outdated
Comment thread src/uparse.jl Outdated
Comment thread src/uparse.jl Outdated
Comment thread src/uparse.jl Outdated
MilesCranmerBot and others added 2 commits April 23, 2026 22:48
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmerBot

Copy link
Copy Markdown
Contributor Author

Addressed in 6efcab1.

What changed:

  • moved the external-unit registration helpers out of uparse.jl into shared internal code, so the parse paths stop reaching back into the parent module with reflective getfield(...) lookups
  • cleaned up the u"..." and us"..." symbol-resolution branches so the error conditions live together and the success path is the shared return block
  • kept builtin unit precedence over similarly named constants like h, while still preserving the external-unit recovery path across precompile/import boundaries

Validation:

  • /usr/local/bin/julia --project=. -e 'using Pkg; Pkg.test(test_args=["unittests"])'\n- /usr/local/bin/julia --project=. -e 'using Pkg; Pkg.test()'\n- /usr/local/bin/julia --project=. -e 'using Pkg; Pkg.test(coverage=true)'

@MilesCranmer

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6efcab19a9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/uparse.jl Outdated
Comment thread src/symbolic_dimensions.jl Outdated

@MilesCranmer MilesCranmer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MilesCranmerBot Please address these comments + those of codex bot + ensure that code coverage for the diff is 100% (failing CI at the moment)

Comment thread src/external_unit_lookup.jl Outdated
Comment thread src/uparse.jl Outdated
Comment thread src/uparse.jl Outdated
Comment thread src/symbolic_dimensions.jl Outdated
Comment thread src/symbolic_dimensions.jl
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmerBot

Copy link
Copy Markdown
Contributor Author

Addressed the open review comments in 9f886d9.

What changed:

  • removed the extra external_unit_lookup.jl helper file
  • restored uparse / sym_uparse support for registered units while keeping the external-unit precompile fix
  • simplified the parser branches and removed the GlobalRef(mod, sym) lookup pattern from the external-unit path
  • added explicit regression tests for registered-unit parsing so the changed lines are exercised directly

Local verification:

  • /usr/local/bin/julia --project=. -e 'using Pkg; Pkg.test()'
  • focused external-unit smoke test for uparse / sym_uparse and precompile-style module lookup ✅

I also resolved the review threads that this commit addresses. CI is running on the updated branch now, including the codecov path.

@MilesCranmer

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f886d9c55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/uparse.jl Outdated
MilesCranmerBot and others added 3 commits June 8, 2026 19:49
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proper way to support "g" (acceleration) unit

2 participants