Skip to content

perf: remove global scans and quadratic temporal processing - #10

Merged
script3r merged 2 commits into
mainfrom
codex/algorithmic-performance
Sep 6, 2026
Merged

script3r merged 2 commits into
mainfrom
codex/algorithmic-performance

Conversation

@script3r

@script3r script3r commented Sep 5, 2026 •

Copy link
Copy Markdown
Owner

Persistent ingestion repeatedly scanned unrelated global-ID and boundary state on ordinary duplicate merges, while temporal history processing performed several Cartesian scans. This change removes that repeated work while preserving entity resolution, temporal component guards, canonical IDs and synchronous durable acknowledgements.

  • Skip provably unchanged global-ID reconciliation work and incrementally merge normalized strong-ID histories.
  • Enumerate exact cross-shard temporal overlaps with a sweep and a common-intersection fast path.
  • Sweep golden-value boundaries and use normalized two-pointer interval intersection for queries and constraint overlap reporting.
  • Preserve legacy malformed-public-interval behavior, with persistent differential and recovery regressions.
  • Add a reproducible persistent benchmark runner, raw measurements and a detailed analysis of algorithmic bounds and remaining bottlenecks.

On five local persistent shards, three paired one-million-record runs per revision improved median throughput by 45.8% with 10% generated overlap and 46.9% with 0% generated overlap. A single two-million-record comparison improved throughput by 84.7%. All 16 million comparative records were acknowledged with zero failed batches. Focused long-history golden queries improved up to 261×; that ratio is specific to the diagnostic workload. No measured workload regressed, but these results do not establish universal capacity or linear engine scaling.

See the analysis and raw results for workload details, methods and limitations.

Local validation passed: 371 tests, zero failures (7 intentionally ignored diagnostics/examples); focused release timing diagnostics; Rust 1.88 all-target/all-feature compatibility; Rust 1.98 strict Clippy; formatting; strict Rustdoc; release-package verification; and relative documentation links. All GitHub CI checks passed: Linux build/tests, strict documentation/lints, package and deployment-manifest validation, MSRV, and the production-container build.

@script3r
script3r merged commit 361874e into main Sep 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant