Repository navigation
Streamed //x record paths and live-count limits (v2.3.0) - #37
Merged
Merged
Conversation
…dcards The burster keeps, per open spine element, the set of pattern steps its ancestry has matched; an element that completes the pattern is a burst and holds any match nested in it. Reached through internal/xdmburst, so xdm gains no exported API; ParseBursts takes at most 63 names.
… outermost() //rec, doc//rec, descendant::rec, /*/*/rec, *:rec, p:* and outermost() or innermost() of such a path now cut bursts at the outermost match; a match nested in it is evaluated inside it, in document order, so each burst holds all of the path's items that lie in it and no other.
xpath's per-transform step memo keyed //x on each burst's document node and so kept every burst tree to the end: count(//rec) over 100 MB peaked at 1.8 GB heap against the tree's 0.8 GB. The two forms select the same nodes, since a record path's predicates do not depend on position.
The result-node and byte budgets were cumulative, so a correct stream with flat state was refused after a million records. An item of a burst-fed for-each or iterate that writes no output and forces no global now refunds its charges beyond what xsl:next-iteration carries on, at most 4,096 nodes and 64 KiB per item unless the caller bounds the total. fn:copy-of and fn:snapshot are charged like xsl:copy-of. New TransformOptions MaxNodes, MaxBytes, MaxNodesBuilt, MaxBytesBuilt.
Each burst's tree copied every ancestor, so a count(//rec) under a 900-deep spine took 37.5 s against the tree's 1.6 s. Small bursts are now copied into a batch tree, under ancestor copies made once per spine element opened and once per batch, and delivered together; xslt evaluates a select once per tree. The parse defers taking back its stores while a batch shares them, bounded per spine level.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streamed record paths with
//, wildcards andoutermost(), and limits that count what a streamed loop holds. The full list is in CHANGELOG.md under "Unreleased".Streaming
//rec,doc//rec,descendant::rec,/*/*/rec,*:rec,p:*, default-namespace names, andoutermost()/innermost()of these. Both streamed shapes take them: thexsl:for-each/xsl:iterateloop, and a singlevalue-of/copy-of/sequence.count(//rec)under a 900-deep spineEvery admitted shape uses at most the tree's memory. The folded shapes stay near 3 MB heap at 1 GB.
Limits
TransformOptionsfields:MaxNodesandMaxBytes(held bounds);MaxNodesBuiltandMaxBytesBuilt(total-work bounds). Zero keeps the defaults.fn:copy-ofandfn:snapshotcopies are now charged to the result-node budget, asxsl:copy-ofalready was.Reviews
Verification
sh tests/check.shat a6db8ec: every lane that ran passed. UBL and CII are skipped (licensed corpora).Race, 32-bit vet and the
w3cschemasmodule pass.