diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7af4283d..0977c8b8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,22 @@ Notable changes, newest first. Versions follow [semantic
versioning](https://semver.org): from 1.0.0 the exported API is stable, and a
breaking change means 2.0 with a new module path. See *Stability* below.
+## Unreleased
+
+### Added
+
+| Change | What it does | Commit |
+|---|---|---|
+| Streamed record paths with `//`, wildcards and `outermost()` | `//rec`, `doc//rec`, `/*/*/rec`, `*:rec`, `p:*` stream in both shapes; nested matches ride inside their outermost record. Small records share one tree: 900-deep `count(//rec)` 2.0 s CPU. | [`df9677e`][df9677e], [`fd31bb3`][fd31bb3], [`382983c`][382983c], [`22af5f4`][22af5f4] |
+| `TransformOptions` `MaxNodes`, `MaxBytes`, `MaxNodesBuilt`, `MaxBytesBuilt` | Held and total-work bounds for result and temporary trees and strings. Zero keeps the defaults; negative removes the bound. | [`294b965`][294b965] |
+
+### Changed
+
+| Change | Problem → solution | Commit |
+|---|---|---|
+| Streamed loops were refused after ~1M records of flat state | The node and byte budgets counted every temporary ever built. A loop item that writes no output now hands back what it dropped. | [`294b965`][294b965] |
+| `fn:copy-of` and `fn:snapshot` copies were not charged to the result-node budget | Unlike `xsl:copy-of`. They are charged before the copy, so a transform making over 2M such copies is now refused. | [`294b965`][294b965] |
+
## v2.2.0 — 2026-11-01
Memory footprint work: what a tree holds and what a transform peaks at. Output
@@ -1890,3 +1906,8 @@ here so every entry in this file sits under a release.
[48b1a38]: https://github.com/knroy/go-xml/commit/48b1a38
[3157566]: https://github.com/knroy/go-xml/commit/3157566
[4944511]: https://github.com/knroy/go-xml/commit/4944511
+[df9677e]: https://github.com/knroy/go-xml/commit/df9677e
+[fd31bb3]: https://github.com/knroy/go-xml/commit/fd31bb3
+[382983c]: https://github.com/knroy/go-xml/commit/382983c
+[22af5f4]: https://github.com/knroy/go-xml/commit/22af5f4
+[294b965]: https://github.com/knroy/go-xml/commit/294b965
diff --git a/README.md b/README.md
index bd38a2b0..a85ece01 100644
--- a/README.md
+++ b/README.md
@@ -139,7 +139,7 @@ and maintains it as a project of his own.
| **XSD 1.1** | 99.98% instance (26,217 of 26,222); **99.97%** schema-validity (15,350 of 15,354); opt-in via `Version11` |
| **RELAX NG** | 100.00% of James Clark's spectest (965 of 965 assertions); XML and compact syntax, from Go and from `go-xml validate -rng` |
| **DTD** | content models, attribute defaults, enumerations, `ID`/`IDREF`; external subset, parameter entities across both subsets, conditional sections — via `dtd.Load` with a caller-supplied resolver, nothing fetched by default |
-| **Tests** | 2,988 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
+| **Tests** | 2,999 `func Test` declarations, clean under `-race` (a few subtests skip without the corpora below) |
| **Production schemas** | UBL 2.1, UN/CEFACT CII, Factur-X/ZUGFeRD, Peppol BIS 3.0 — 88 schemas load, instances validate clean |
| **API** | 1.2; the exported surface is stable and additive over 1.1, and a breaking change means 2.0 with a new module path |
@@ -180,7 +180,7 @@ true here:
**Streamed execution is implemented only for two narrow shapes** (an
`xsl:source-document streamable="yes"` whose body is one `xsl:for-each` or
- `xsl:iterate` over a child path, or one `xsl:value-of`/`xsl:copy-of`/
+ `xsl:iterate` over a record path such as `//rec` or `doc/rec`, or one `xsl:value-of`/`xsl:copy-of`/
`xsl:sequence` over one record path such as `sum(doc/rec/@v)`, read one
record at a time; see
[known gaps](docs/known-gaps.md)). Otherwise the streaming cases are in
@@ -1393,7 +1393,7 @@ back, is in [docs/testing.md](docs/testing.md).
| method | what it catches | what it misses |
|---|---|---|
-| **Unit tests** (2,988 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** (2,999 `func Test` declarations) | places where a plausible implementation is quietly wrong | anything nobody thought to write a test for |
| **Spec inventories** | features absent entirely | features present but behaving wrongly |
| **Saxon differential** | subtle behavioural divergence on real stylesheets | constructs the corpora do not use |
| **W3C QT3 suite** | systematic conformance across 22,054 XPath and 30,517 XQuery cases | XSLT (it is an XPath suite) |
diff --git a/docs/conformance-gaps.md b/docs/conformance-gaps.md
index 62814bd0..f5d574c4 100644
--- a/docs/conformance-gaps.md
+++ b/docs/conformance-gaps.md
@@ -49,7 +49,7 @@ W3C disagreements: 0 + 0 + 0 + 1 + 8 + 23 + 30 + 31 + 0 = 93. Measured 2026-09-1
-The unit-test suite is 2,988 tests.
+The unit-test suite is 2,999 tests.
The last two rows are not W3C suites but real-world corpora — DocBook xslTNG's
diff --git a/docs/known-gaps.md b/docs/known-gaps.md
index d45ddc5a..59700fdb 100644
--- a/docs/known-gaps.md
+++ b/docs/known-gaps.md
@@ -181,12 +181,18 @@ Measured on branches in 2026-11 and not adopted, each with the reason:
Since v2.2.0 an `xsl:source-document streamable="yes"` whose body is one
`xsl:for-each` or `xsl:iterate` (optionally inside constant literal result
-elements) over a child path of plain names, with non-positional predicates on
-the last step only, and a body the gate recognises with a known and verified
+elements) over a record path, with non-positional predicates on the last step
+only, and a body the gate recognises with a known and verified
§19.8 verdict, runs record by record
through `xdm.ParseBursts`: a 1 GB file totalled by `xsl:iterate` peaks at
27 MB RSS against 3.7 GB, at about 2× the CPU. Everything else builds the tree,
-as before.
+as before. A record path is child or descendant steps whose tests are names or
+wildcards (`doc/rec`, `//rec`, `doc//rec`, `/*/*/rec`, `*:rec`, `p:*`, names
+resolved through `xpath-default-namespace`), or `outermost()`/`innermost()` of
+one (since v2.3.0). A record is cut at its outermost match, so a match nested
+in it is delivered inside its tree, in document order. Small records share one
+tree, so ancestors are copied once per element opened: `count(//rec)` under a
+900-deep spine takes 2.0 s of CPU against the tree's 1.6 s.
The second shape is a body that is one `xsl:value-of`, `xsl:copy-of` or
`xsl:sequence` (again optionally inside constant literal result elements) whose
@@ -201,7 +207,8 @@ still growing with the input. A select that would keep the records' nodes or
copies (`copy-of(doc/rec)`, filters returning nodes, `head`/`tail` passed on)
is refused and runs on the tree, because a kept record costs more there than in
a whole tree (945 B against 208 B). Also refused: `distinct-values`, `reverse`,
-`sort`, `last()`, union/intersect/except, `let`, user functions, wildcard steps.
+`sort`, `last()`, union/intersect/except, `let`, user functions, and paths to
+non-elements (`//text()`, `//@x`).
In the W3C XSLT 3.0 run 364 cases take the burst path (9 before the second
shape). Of the 786 single-instruction bodies over one record path with a known
@@ -209,7 +216,16 @@ and verified verdict, 294 are admitted (261 bounded reducers, 33 pipelined).
Also:
- **The result is still built in memory**, so a large filtered copy grows with
- its output and stops at the result-node cap.
+ its output and stops at the result-node cap. Temporary work per record is not
+ charged cumulatively since v2.3.0: a loop item that writes nothing to the
+ output hands back up to 4,096 nodes and 64 KiB it dropped
+ (`TransformOptions.MaxNodesBuilt`/`MaxBytesBuilt` lift that cap). An item that
+ writes output keeps all it built charged, so a stream that both writes and
+ builds temporaries per record reaches the cap sooner than its output alone.
+- **Growing state is bounded only per value or per expression.** A carried
+ integer multiplied each record (`MaxDigits`), a sequence appended to or
+ passed through `distinct-values` each record (`MaxItems`) run quadratic time
+ and are stopped by the context deadline, before and after v2.3.0.
- **Error order can differ from tree mode, for the loop shape.** (The single-instruction
shape re-runs on the tree after any error, so its errors are the tree's; the
cost is a second read of the document, and the 1 GiB string budget charges the
diff --git a/docs/options.md b/docs/options.md
index 92a947bc..7f6f37ff 100644
--- a/docs/options.md
+++ b/docs/options.md
@@ -521,6 +521,10 @@ res, err := sty.Transform(ctx, doc.Root, xslt.TransformOptions{
| `Collections` | `xpath.CollectionResolver` | disabled | Resolves `fn:collection`. **Nil disables it**, and setting `Documents` does not set this — the two are separate switches on purpose. |
| `MaxDepth` | `int` | `DefaultMaxDepth` = 1000 | Template recursion limit, and the bound on `fn:transform` nesting. Catches a stylesheet with no base case. |
| `MaxItems` | `int` | `xpath.MaxItems` = 5,000,000 | Items one transformation may materialise, `XPDY0130` past it. Negative removes the bound; nested `fn:transform` and `xsl:evaluate` keep the caller's. |
+| `MaxNodes` | `int` | `xpath.MaxNodes` = 2,000,000 | Result- and temporary-tree nodes the transform may hold, `XPDY0130` past it. Negative removes the bound. In a streamed `xsl:for-each`/`xsl:iterate` an item that writes no output hands back what it dropped, so only what it carries on counts; elsewhere every node built counts. |
+| `MaxBytes` | `int64` | `xpath.MaxBytes` = 1 GiB | String content held, on `MaxNodes`' terms. |
+| `MaxNodesBuilt` | `int64` | 0: a streamed item hands back at most 4,096 nodes | Every node built, refunds notwithstanding. Positive bounds the total and lifts the per-item cap; negative removes both, so only what is held is bounded. |
+| `MaxBytesBuilt` | `int64` | 0: at most 65,536 bytes per item | `MaxNodesBuilt` for string content. |
| `DisableAssertions` | `bool` | `false` — assertions enabled | Turns off `xsl:assert` checking for the whole transformation. XSLT 3.0 §22.2: "By default, assertions are enabled." |
| `InitialMode` | `string` | default mode | Mode for the initial `apply-templates`. |
| `InitialTemplate` | `string` | match the root | Invokes a named template instead, which is how a stylesheet of only named templates is entered. |
diff --git a/docs/security.md b/docs/security.md
index 2295c02d..71ccb867 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -1774,7 +1774,9 @@ No `unsafe`, no `cgo`, no `reflect` in any non-test file.
If you know your documents are smaller, lower them: they are the bound on
what one request can cost you. Those bound what a parse **reads**;
`xpath.MaxNodes` (2 million) separately bounds what a transform or query
- **constructs**, which no ingress limit sees.
+ **constructs**, which no ingress limit sees — in a streamed loop, what it
+ holds; `TransformOptions.MaxNodes` and `MaxNodesBuilt` set the held and total
+ bounds.
2. **Leave `AllowDOCTYPE` off** unless a schema you control needs it. Turning it
on does not reopen XXE, but it is still the wider setting.
3. **Sanitise URLs** if you serve transform output as HTML. XSLT does not, and
@@ -1875,7 +1877,7 @@ reject* refused a legal one, and *cost* produced the right answer too slowly.
- **RELAX NG rejected a valid document** — correctness. `(a?){13} a{13}` against 13 `` crossed the 100,000-node derivative cap, and the message named a `oneOrMore` the schema does not have; `xmllint` accepts it. Choice chains now merge alternative by alternative, under a 1,024-pair budget per merge so a hostile schema stays bounded. Pinned by `TestOptionalsBeforeRequiredStayValid`. Larger N still reaches the cap (docs/known-gaps.md).
- **Serializing atomic values was quadratic** — cost, driven by result size. `xslt/seqnorm.go` joined adjacent atomics by repeated concatenation, so `1 to 100000` took 21.6 s and 6.95 GB for 588 KB of output. Each run is joined once. Pinned by `TestJoinAdjacentAtomicsIsLinear`.
- **Several producers were uncharged or unbounded** — cost, driven by document or query data. `unparsed-text-lines`, `parse-json` and `json-doc` built tens of millions of items past `MaxItems` (5.2 GB from a 67 MB file); DTD default values were copied into every element unseen by `MaxNodes` or `MaxBytes` (2 MB retained 1.97 GB); an `xs:integer` had no size bound (a 4M-digit cast allocated 34 GB); a sequence-valued accumulator copied its value into every node (200 KB held 12.4 GB). Lines and JSON values are charged, defaults are stored once, `Env.MaxDigits` (10^6) refuses with FOCA0003 or FOAR0002, and accumulator values share their prefix and are charged.
-- **`xsl:copy-of` of a document bypassed `MaxNodes`** — the inconsistency, not data loss: it copied 5.7M nodes and exited 0 where shallow-copy is refused. It is charged before the copy.
+- **`xsl:copy-of` of a document bypassed `MaxNodes`** — the inconsistency, not data loss: it copied 5.7M nodes and exited 0 where shallow-copy is refused. It is charged before the copy. `fn:copy-of` and `fn:snapshot` were the remaining uncharged route until v2.3.0; they are charged before the copy too (the subtree; a snapshot adds its ancestors).
**Eighteenth pass — a memory audit, 2026-10-31.** The rest of that audit was retention and allocation with no hostile driver; it is in CHANGELOG under *Changed — memory*.
@@ -1950,7 +1952,7 @@ than as a description.
**Eleventh pass — the result tree was bounded by nothing, 2026-09-14.**
-- **A transform could build an unbounded result tree from a small input** — availability, and the last of the three dimensions an evaluation can grow in. `MaxItems` bounds the items an expression materialises and `MaxBytes` the string content it builds; a result tree is neither. It is not an intermediate sequence — it outlives the expression that contributed to it, which is what makes it a *result* — and its size is in nodes rather than characters, so a stylesheet whose output is the CROSS PRODUCT of its input was charged nothing at all. Two nested `xsl:for-each` over `//i` is a three-line stylesheet that squares the input's node count: 811 bytes of input built 10,000 nodes, 6.4 kB built 640,000, and 24 kB built 9,000,000 and allocated 44 GB over 49 s before returning a result. Each loop is individually a few thousand items, far under `MaxItems`, and the limit that could have seen the product does not measure trees. This memory also differs from the other two in being **retained** rather than transient: the tree is the caller's result and is live until the caller discards it, so collection reclaims none of it while it is being built. A context deadline did fire — a 20 s timeout returned `context deadline exceeded` — but only after the allocation had already happened, because cancellation was checked between instructions while the growth was inside them. The realistic case is not an attacker but a trusted report stylesheet run over a customer document larger than expected. `xpath.MaxNodes` now charges every node construction, refusing with `XPDY0130` as XPath 3.1 §2.3.1 sanctions. The charge reaches `xdmbuild` — which names neither host language and imports nothing but `xdm` — through `xdmbuild.Policy`, already the seam for everything the builder cannot know by itself; a refusal is latched on the builder and returned by `execSequence`, because the constructing calls return no error and threading one through all eighty-seven call sites would be a far larger change than the bound is worth. It is held across one transform and never reset, because the runaway is a *loop* and every narrower boundary is one the loop resets — the same reasoning `MaxBytes` follows, with `entities` as the precedent for the absent reset. Temporary trees are charged too: a runaway written into an `xsl:variable` is allocated for real, and a bound that saw only the principal result tree would miss it. `xsl:copy-of` is charged for the whole subtree it copies rather than for one node, or copying a large element once per node of a large document would reach the same product uncharged. **The default is drawn from the memory it caps, not from a margin over real work**, and getting that backwards is the instructive part: a constructed node costs a few hundred bytes of live heap — measured independently at **328 bytes** on darwin/arm64, stable to a tenth of a byte between 160,000 and 640,000 nodes — so 50,000,000, a far larger multiple of anything legitimate, still allowed tens of GB and minutes of work before refusing, and 10,000,000 allowed several GB. Two million caps a tree near 0.6 GB. (An earlier draft of this entry recorded 672 bytes and a 1.3 GB cap, from the implementing agent's own measurement; it did not reproduce and no test pinned it. The error is in the safe direction — the real ceiling is half the claim, so the bound binds sooner than advertised — but per-node cost varies with node kind, name length and platform, so it is an order of magnitude to re-measure, not a constant to compute against.), and the measured case now refuses in 1.26 s having allocated 5.1 GB, flat in the input size rather than growing with it. It remains far above real work, which is the constraint that matters on the other side: instrumenting the charge point across the suites and corpora, the largest legitimate result tree was 274,719 nodes (XSLT suites), with XSpec at 52,607, the XQuery QT3 lane at 35,328, and DocBook xslTNG at 35,104 over 577 real documents — so the bound is seven times the largest measured anywhere and thirty-eight times the largest real-world one. XQuery is charged through the same seam, a direct element constructor inside a FLWOR having the same shape. Pinned by `xslt/nodebudget_test.go`, which asserts the refusal, the code and sentinel, that a 640,000-node tree still builds, that the refusal's allocation does not grow with the input, that the budget neither leaks across transforms nor is reset so often that it never binds, and the variable and `xsl:copy-of` routes. See CHANGELOG.
+- **A transform could build an unbounded result tree from a small input** — availability, and the last of the three dimensions an evaluation can grow in. `MaxItems` bounds the items an expression materialises and `MaxBytes` the string content it builds; a result tree is neither. It is not an intermediate sequence — it outlives the expression that contributed to it, which is what makes it a *result* — and its size is in nodes rather than characters, so a stylesheet whose output is the CROSS PRODUCT of its input was charged nothing at all. Two nested `xsl:for-each` over `//i` is a three-line stylesheet that squares the input's node count: 811 bytes of input built 10,000 nodes, 6.4 kB built 640,000, and 24 kB built 9,000,000 and allocated 44 GB over 49 s before returning a result. Each loop is individually a few thousand items, far under `MaxItems`, and the limit that could have seen the product does not measure trees. This memory also differs from the other two in being **retained** rather than transient: the tree is the caller's result and is live until the caller discards it, so collection reclaims none of it while it is being built. A context deadline did fire — a 20 s timeout returned `context deadline exceeded` — but only after the allocation had already happened, because cancellation was checked between instructions while the growth was inside them. The realistic case is not an attacker but a trusted report stylesheet run over a customer document larger than expected. `xpath.MaxNodes` now charges every node construction, refusing with `XPDY0130` as XPath 3.1 §2.3.1 sanctions. The charge reaches `xdmbuild` — which names neither host language and imports nothing but `xdm` — through `xdmbuild.Policy`, already the seam for everything the builder cannot know by itself; a refusal is latched on the builder and returned by `execSequence`, because the constructing calls return no error and threading one through all eighty-seven call sites would be a far larger change than the bound is worth. It is held across one transform and never reset (since v2.3.0 a streamed loop item that writes no output and forces no global hands back what it built beyond its carried parameters, at most 4,096 nodes and 64 KiB per item by default, so total work stays at most the bound plus that allowance per record), because the runaway is a *loop* and every narrower boundary is one the loop resets — the same reasoning `MaxBytes` follows, with `entities` as the precedent for the absent reset. Temporary trees are charged too: a runaway written into an `xsl:variable` is allocated for real, and a bound that saw only the principal result tree would miss it. `xsl:copy-of` is charged for the whole subtree it copies rather than for one node, or copying a large element once per node of a large document would reach the same product uncharged. **The default is drawn from the memory it caps, not from a margin over real work**, and getting that backwards is the instructive part: a constructed node costs a few hundred bytes of live heap — measured independently at **328 bytes** on darwin/arm64, stable to a tenth of a byte between 160,000 and 640,000 nodes — so 50,000,000, a far larger multiple of anything legitimate, still allowed tens of GB and minutes of work before refusing, and 10,000,000 allowed several GB. Two million caps a tree near 0.6 GB. (An earlier draft of this entry recorded 672 bytes and a 1.3 GB cap, from the implementing agent's own measurement; it did not reproduce and no test pinned it. The error is in the safe direction — the real ceiling is half the claim, so the bound binds sooner than advertised — but per-node cost varies with node kind, name length and platform, so it is an order of magnitude to re-measure, not a constant to compute against.), and the measured case now refuses in 1.26 s having allocated 5.1 GB, flat in the input size rather than growing with it. It remains far above real work, which is the constraint that matters on the other side: instrumenting the charge point across the suites and corpora, the largest legitimate result tree was 274,719 nodes (XSLT suites), with XSpec at 52,607, the XQuery QT3 lane at 35,328, and DocBook xslTNG at 35,104 over 577 real documents — so the bound is seven times the largest measured anywhere and thirty-eight times the largest real-world one. XQuery is charged through the same seam, a direct element constructor inside a FLWOR having the same shape. Pinned by `xslt/nodebudget_test.go`, which asserts the refusal, the code and sentinel, that a 640,000-node tree still builds, that the refusal's allocation does not grow with the input, that the budget neither leaks across transforms nor is reset so often that it never binds, and the variable and `xsl:copy-of` routes. See CHANGELOG.
**Tenth pass — allocation-local resource accounting, 2026-09-11.**
diff --git a/docs/stats.md b/docs/stats.md
index b6f5c8a7..4aade744 100644
--- a/docs/stats.md
+++ b/docs/stats.md
@@ -23,7 +23,7 @@ claim is only as good as the command behind it.
| figure | count | counted by |
|---|---:|---|
-| Unit tests | 2,988 | `grep -rn '^func Test' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
+| Unit tests | 2,999 | `grep -rn '^func Test' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
| Fuzz targets | 12 | `grep -rn '^func Fuzz' --include='*_test.go' . \| grep -v '/\.claude/worktrees/' \| grep -vc '^\./bench/'` |
| Limit boundary tests | 14 | `grep -hc '^func Test' ./*/limits_boundary*_test.go \| awk '{n += $1} END {print n + 0}'` |
diff --git a/docs/testing.md b/docs/testing.md
index 92a5ff35..04cd329a 100644
--- a/docs/testing.md
+++ b/docs/testing.md
@@ -21,7 +21,7 @@ let something through, and the column that matters is the last one.
| layer | count | catches | misses |
|---|---:|---|---|
-| **Unit tests** | 2,988 | a plausible implementation that is quietly wrong | anything nobody thought to write a test for |
+| **Unit tests** | 2,999 | a plausible implementation that is quietly wrong | anything nobody thought to write a test for |
| **Limit boundary tests** | 14 tests | an off-by-one or an overflow at the edge of a configurable limit | a limit nobody added to the inventory |
| **Race detector** | same tests | shared state a single-goroutine run never reveals | a data race on a path no test walks |
| **W3C conformance suites** | 152,241 cases | systematic divergence from the specification | what the suites do not ask about — see below |
diff --git a/docs/todo.md b/docs/todo.md
index af9334dc..eff76619 100644
--- a/docs/todo.md
+++ b/docs/todo.md
@@ -23,7 +23,7 @@ Current position:
| XSLT 3.0 | 99.80% — 11,495 of 11,518 in scope (23 failing); 3 of those need more of the §19.8 streamability analysis |
| RELAX NG | 100.00% — 965 of 965 |
| Schemas wrongly refused | 7 — 6 on XSD 1.0, 1 on 1.1 |
-| Tests | 2,988 `func Test` declarations, clean under `-race` |
+| Tests | 2,999 `func Test` declarations, clean under `-race` |
Every one of those failures, and why it is still open, is catalogued in
[known-gaps.md](known-gaps.md). This file is the forward-looking half — what
diff --git a/internal/xdmburst/xdmburst.go b/internal/xdmburst/xdmburst.go
new file mode 100644
index 00000000..e3520151
--- /dev/null
+++ b/internal/xdmburst/xdmburst.go
@@ -0,0 +1,25 @@
+// Package xdmburst lets streamed XSLT cut bursts at a pattern with wildcards
+// and descendant steps without adding exported API to xdm. Package xdm sets
+// Parse in its init. The arguments typed any are xdm types: this package
+// cannot import xdm, which imports it.
+package xdmburst
+
+import "io"
+
+// Step is one step of a burst pattern: an element name test, reached from
+// the step before it (or the document node) as a child or, with Descendant,
+// as any descendant.
+type Step struct {
+ URI, Local string
+ AnyURI, AnyLocal bool
+ Descendant bool
+}
+
+// MaxSteps is the longest pattern Parse takes.
+const MaxSteps = 63
+
+// Parse is xdm.ParseBursts over a pattern of 1 to MaxSteps steps: opts is an
+// xdm.ParseOptions, and fn is called with the *xdm.Node of each outermost
+// element the pattern matches -- an element matched inside it is delivered
+// within it, not again.
+var Parse func(r io.Reader, opts any, steps []Step, fn func(any) error) error
diff --git a/internal/xpathleaf/host.go b/internal/xpathleaf/host.go
index 601dd9a1..370baf62 100644
--- a/internal/xpathleaf/host.go
+++ b/internal/xpathleaf/host.go
@@ -84,3 +84,26 @@ var (
// select expression in parts this way (xslt/streamburst.go). Package xpath
// sets it in its init.
var WithExpr func(c, e any) any
+
+// Limits are the bounds an evaluation's result-node and string-byte counters
+// are checked against, for a host that offers its caller more than the
+// package defaults. Nodes and Bytes bound the counters, which a host may
+// refund (Refund); zero is xpath.MaxNodes or xpath.MaxBytes and a negative
+// value is no bound. NodesBuilt and BytesBuilt, when positive, bound every
+// node and byte charged, refunds notwithstanding; zero or negative is none.
+type Limits struct {
+ Nodes, Bytes int64
+ NodesBuilt, BytesBuilt int64
+}
+
+// Package xpath sets these in its init. ctx is a *xpath.Context. SetLimits
+// writes the limits of a context the caller has just made with NewContext and
+// not yet shared; a nested evaluation that adopts the budget takes them with
+// it. Charged reports the result-node and byte counters, and Refund lowers
+// them by what the host knows is no longer held (never below zero). Neither
+// touches the NodesBuilt and BytesBuilt totals.
+var (
+ SetLimits func(ctx any, l Limits)
+ Charged func(ctx any) (nodes, bytes int64)
+ Refund func(ctx any, nodes, bytes int64)
+)
diff --git a/xdm/bursts.go b/xdm/bursts.go
index 87275180..3d5dc1e4 100644
--- a/xdm/bursts.go
+++ b/xdm/bursts.go
@@ -3,6 +3,7 @@ package xdm
import (
"io"
+ "github.com/knroy/go-xml/v2/internal/xdmburst"
"github.com/knroy/go-xml/v2/internal/xdmclone"
)
@@ -29,30 +30,64 @@ import (
// document order; a node fn keeps stays valid after fn returns.
// ParseBursts parses r and calls fn with each element whose ancestry of
-// element names, from the document element down, is path. An error from fn
-// stops the parse and is returned.
+// element names, from the document element down, is path, of at most
+// xdmburst.MaxSteps names. An error from fn stops the parse and is returned.
func ParseBursts(r io.Reader, opts ParseOptions, path []QName, fn func(*Node) error) error {
- if len(path) == 0 {
- panic("xdm: ParseBursts needs a non-empty path")
+ steps := make([]xdmburst.Step, len(path))
+ for i, q := range path {
+ steps[i] = xdmburst.Step{URI: q.URI, Local: q.Local}
}
- opts.bursts = &burster{path: path, fn: fn}
+ return parseBursts(r, opts, steps, fn)
+}
+
+func init() {
+ xdmburst.Parse = func(r io.Reader, opts any, steps []xdmburst.Step, fn func(any) error) error {
+ return parseBursts(r, opts.(ParseOptions), steps, func(n *Node) error { return fn(n) })
+ }
+}
+
+// parseBursts is ParseBursts over a pattern that may have wildcards and
+// descendant steps. An element whose ancestry matches it is a burst; one
+// whose ancestry could still lead to a match below it is kept on the spine;
+// any other is dropped. A match inside a burst is delivered with it, so the
+// bursts are the outermost matches, in document order.
+func parseBursts(r io.Reader, opts ParseOptions, steps []xdmburst.Step, fn func(*Node) error) error {
+ if len(steps) == 0 || len(steps) > xdmburst.MaxSteps {
+ panic("xdm: ParseBursts needs a path of 1 to 63 names")
+ }
+ opts.bursts = &burster{steps: steps, fn: fn, states: []uint64{1}, gens: []uint64{0}}
_, err := parse(r, "", opts)
return err
}
// burster is ParseBursts' state inside parse.
type burster struct {
- path []QName
- fn func(*Node) error
- // spine is how many levels of the open chain match path and are kept.
+ steps []xdmburst.Step
+ fn func(*Node) error
+ // spine is how many levels of the open chain could lead to a match and
+ // are kept.
spine int
// drop is the depth of the element being built only to be dropped (a
- // burst, or an element off the path), or 0.
- drop int
+ // burst, or an element off the path), or 0; burst says it is a burst.
+ drop int
+ burst bool
+ // states[d] is the set of pattern steps the spine element at depth d
+ // (0: the document node) has matched, as bits: bit k means the first k
+ // steps are matched by it and its ancestors.
+ states []uint64
// marks[d] is the tree's state once the spine element at depth d (0:
// the document node) had stored its attributes. Everything stored after
// it belongs to content that is dropped.
marks []burstMark
+ // gens[d] tells the spine element at depth d from the others that have
+ // been there: gen counts the spine elements opened.
+ gens []uint64
+ gen uint64
+ // batch holds the small bursts not yet delivered (burstBatch), or is nil.
+ batch *burstBatch
+ // spineCopies counts the records copied as ancestors of a burst, for
+ // the test that keeps it in proportion to the elements read.
+ spineCopies int
// moved is the first of the parse's chunks the last burst's tree took
// over, or 0.
moved int
@@ -75,19 +110,37 @@ func (b *burster) start(el *Node, d int) {
if b.drop != 0 {
return
}
- if d == b.spine+1 && b.matches(el, d) {
- if d < len(b.path) {
- b.spine = d
- b.marks = append(b.marks, markOf(el.tree))
- return
- }
+ n := b.advance(b.states[b.spine], el.Name())
+ switch {
+ case n&(1< 0:
b.spine--
b.marks = b.marks[:len(b.marks)-1]
+ b.states = b.states[:len(b.states)-1]
+ b.gens = b.gens[:len(b.gens)-1]
default:
return nil
}
+ if d == 1 {
+ // The document element is done: so is the last batch.
+ if err := b.flush(); err != nil {
+ return err
+ }
+ }
b.discardLast(el.Parent(), spaces, names)
return nil
}
@@ -114,13 +176,34 @@ func (b *burster) end(el *Node, d int, spaces *spaceTable, names *nameCache) err
// trimLeaves drops the leaves completed directly inside the spine element
// cur, at depth d: text, comments and processing instructions no path step
// selects.
-func (b *burster) trimLeaves(cur *Node, d int, spaces *spaceTable, names *nameCache) {
+//
+// It is also where a batch is delivered early: what the parse stored since
+// cur's mark is all garbage now, and once a child of cur opens it cannot be
+// taken back until cur closes, so it is not let grow past burstFrozen.
+func (b *burster) trimLeaves(cur *Node, d int, spaces *spaceTable, names *nameCache) error {
if b.drop != 0 || d == 0 || d != b.spine {
- return
+ return nil
}
for last := cur.LastChild(); last != nil && last.isLeaf(); last = cur.LastChild() {
b.discardLast(cur, spaces, names)
}
+ if b.batch != nil && textSince(cur.tree, b.marks[len(b.marks)-1].text.blocks) >= burstFrozen {
+ if err := b.flush(); err != nil {
+ return err
+ }
+ b.rewind(cur.tree, spaces, names)
+ }
+ return nil
+}
+
+// textSince is how many bytes t's text store holds in blocks from the
+// block numbered from on.
+func textSince(t *Tree, from int) int {
+ n := 0
+ for _, blk := range t.text.blocks[from:] {
+ n += len(blk)
+ }
+ return n
}
// discardLast removes p's last child and the text, names and namespace
@@ -133,15 +216,235 @@ func (b *burster) discardLast(p *Node, spaces *spaceTable, names *nameCache) {
t.chunks = t.chunks[:b.moved]
b.moved = 0
}
+ if b.batch != nil {
+ // The batch's records point into the stores: they are taken back
+ // once it has been delivered.
+ return
+ }
+ b.rewind(p.tree, spaces, names)
+}
+
+// rewind takes back what the parse's tree t stored since the deepest
+// spine element's mark.
+func (b *burster) rewind(t *Tree, spaces *spaceTable, names *nameCache) {
m := b.marks[len(b.marks)-1]
- p.tree.text.rewind(m.text)
- rewindNames(p.tree, m.names, names)
- p.tree.rewindFrames(m.frames, m.frameData)
+ t.text.rewind(m.text)
+ rewindNames(t, m.names, names)
+ t.rewindFrames(m.frames, m.frameData)
// The whitespace table remembers ranges in the store, and some of them
// were just released. Its map is not rebuilt for the next burst.
*spaces = spaceTable{noMap: true}
}
+// take delivers the burst el: a large one at once, in a tree of its own
+// (deliver), and a small one with the bursts before it (burstBatch).
+func (b *burster) take(el *Node) error {
+ t := el.tree
+ if t.endOf(el)-el.self > burstSmall {
+ if err := b.flush(); err != nil {
+ return err
+ }
+ return b.deliver(el)
+ }
+ b.add(el)
+ if bt := b.batch; int(bt.t.n) >= max(burstBatchRecords, 4*len(b.gens)) ||
+ textSince(t, bt.blocks) >= burstBatchText {
+ return b.flush()
+ }
+ return nil
+}
+
+// A burst of more than burstSmall records is delivered on its own; smaller
+// ones are batched until the batch holds burstBatchRecords records (or four
+// times the spine's depth), or the parse has stored burstBatchText bytes of
+// text since the batch began. The text a spine element's content stored
+// before a child of it opened waits for the element to close: burstFrozen
+// bounds it.
+//
+// ponytail: per level, so a batch open under a deep spine can leave up to
+// depth x burstFrozen (64 MB at MaxDepth 1000) waiting; a total would need
+// the levels' shares tracked.
+const (
+ burstSmall = chunkMax
+ burstBatchRecords = 4096
+ burstBatchText = 1 << 20
+ burstFrozen = 64 << 10
+)
+
+// burstBatch is a tree that small bursts are copied into, each under copies
+// of its ancestors, until it is delivered: its bursts are handed to fn one
+// after another, all in the one tree. A burst under the same ancestors as
+// the one before it shares their copies, so the ancestors are copied once
+// for each that is opened, and once more for each batch, rather than once
+// for every burst: a burst under a deep spine cost the spine's depth.
+//
+// Its records are copied verbatim, sharing the parse's names, frames and
+// text, which the parse therefore does not take back (discardLast) until
+// the batch has been delivered. Bursts in one tree are still disjoint
+// subtrees in document order, each with its whole ancestry, and nothing a
+// burst may read tells it from a tree of its own (see the xslt gate).
+type burstBatch struct {
+ t *Tree
+ // path[d] is the copy of the spine element at depth d (0: the document
+ // node), and gens[d] that element's gens entry.
+ path []uint32
+ gens []uint64
+ // els are the bursts, in order.
+ els []uint32
+ // blocks is how many text blocks the parse had when the batch began.
+ blocks int
+ src *Tree
+}
+
+// add copies el, a small burst, into the batch, under copies of the spine
+// elements it does not share with the burst before it.
+func (b *burster) add(el *Node) {
+ t := el.tree
+ bt := b.batch
+ if bt == nil {
+ bt = &burstBatch{t: &Tree{}, blocks: len(t.text.blocks), src: t}
+ b.batch = bt
+ bt.t.rec(bt.copyRange(t, 0, 1, noIdx, noIdx)).v1 = noIdx
+ bt.path, bt.gens = append(bt.path, 0), append(bt.gens, 0)
+ }
+ d := len(b.gens) - 1 // el's parent
+ k := 1
+ for k < len(bt.path) && k <= d && bt.gens[k] == b.gens[k] {
+ k++
+ }
+ bt.closeTo(k)
+ anc := make([]*Node, 0, d-k+1)
+ for p := el.Parent(); len(anc) < d-k+1; p = p.Parent() {
+ anc = append(anc, p)
+ }
+ for i := len(anc) - 1; i >= 0; i-- {
+ s := anc[i]
+ j := bt.appendUnder(t, s.self, s.self+1+s.attrCount())
+ e := bt.t.rec(j)
+ e.end, e.v1 = openEnd, noIdx
+ bt.path, bt.gens = append(bt.path, j), append(bt.gens, b.gens[k])
+ b.spineCopies += 1 + int(s.attrCount())
+ k++
+ }
+ bt.els = append(bt.els, bt.appendUnder(t, el.self, t.endOf(el)))
+}
+
+// appendUnder copies the records [lo, hi) of t, a node's own and those after
+// it, as the last child of the batch's deepest copy, and returns its index.
+func (bt *burstBatch) appendUnder(t *Tree, lo, hi uint32) uint32 {
+ p := bt.t.rec(bt.path[len(bt.path)-1])
+ j := bt.copyRange(t, lo, hi, p.self, p.v1)
+ p.v1 = j
+ return j
+}
+
+// closeTo ends the copies at depths k and deeper.
+func (bt *burstBatch) closeTo(k int) {
+ for i := len(bt.path) - 1; i >= k; i-- {
+ bt.t.rec(bt.path[i]).end = bt.t.n
+ }
+ bt.path, bt.gens = bt.path[:k], bt.gens[:k]
+}
+
+// copyRange appends copies of the records [lo, hi) of t, whose links within
+// the range are renumbered; the first's parent and previous sibling are
+// parent and prev. It returns the first's index.
+func (bt *burstBatch) copyRange(t *Tree, lo, hi, parent, prev uint32) uint32 {
+ c := bt.t
+ base := c.n
+ remap := func(x uint32) uint32 {
+ if x >= lo && x <= hi {
+ return x - lo + base
+ }
+ return x
+ }
+ for i := lo; i < hi; i++ {
+ s := t.rec(i)
+ r := *s
+ d := c.alloc()
+ j := d.self
+ r.tree, r.self = c, j
+ if i == lo {
+ r.parent, r.prev = parent, prev
+ } else {
+ r.parent = remap(r.parent)
+ if r.prev != noIdx {
+ r.prev = remap(r.prev)
+ }
+ }
+ if r.isLeaf() {
+ r.end = j + 1
+ } else {
+ r.end = remap(t.endOf(s))
+ if r.v1 != noIdx {
+ r.v1 = remap(r.v1)
+ }
+ }
+ *d = r
+ if s.flags&fTyped != 0 {
+ *d.ownTyping() = *s.typ() // the names are shared, not copied
+ }
+ if s.flags&fBase != 0 {
+ if c.bases == nil {
+ c.bases = map[uint32]string{}
+ }
+ c.bases[j] = t.bases[i]
+ }
+ if s.flags&fManyAttrs != 0 {
+ if src := c.ownSource(); src.attrCounts == nil {
+ src.attrCounts = map[uint32]uint32{}
+ }
+ c.source.attrCounts[j] = t.source.attrCounts[i]
+ }
+ if s.kind == uint8(KindDocument) {
+ if u := t.docURIs[i]; u != "" {
+ if c.docURIs == nil {
+ c.docURIs = map[uint32]string{}
+ }
+ c.docURIs[j] = u
+ }
+ }
+ }
+ return base
+}
+
+// flush delivers the batch, if there is one: it becomes a document as a
+// burst's tree is (deliver), sharing the parse's stores as they now are,
+// and fn is called with each of its bursts in order.
+func (b *burster) flush() error {
+ bt := b.batch
+ if bt == nil {
+ return nil
+ }
+ b.batch = nil
+ bt.closeTo(0)
+ c, t := bt.t, bt.src
+ c.rec(0).end = c.n
+ c.Root = c.rec(0)
+ c.id.Store(int64(nextTreeID()))
+ c.DocType, c.XMLVersion = t.DocType, "1.0"
+ if b.v11 {
+ c.XMLVersion = "1.1"
+ }
+ if s := t.source; s != nil {
+ c.ownSource().externalSubset = s.externalSubset
+ }
+ c.names = t.names[:len(t.names):len(t.names)]
+ c.text.blocks = make([][]byte, len(t.text.blocks))
+ for i, blk := range t.text.blocks {
+ c.text.blocks[i] = blk[:len(blk):len(blk)]
+ }
+ c.text.size = t.text.size
+ c.frames = t.frames[:len(t.frames):len(t.frames)]
+ c.frameData = t.frameData[:len(t.frameData):len(t.frameData)]
+ for _, i := range bt.els {
+ if err := b.fn(c.rec(i)); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
// deliver hands el to fn in a tree of its own. By now the parse's tree holds
// nothing but the document node, the spine and el (a burst parse does not
// keep the prolog's comments and processing instructions, which every burst
@@ -157,6 +460,7 @@ func (b *burster) deliver(el *Node) error {
b.moved = k + 1
}
c := m(el)
+ b.spineCopies += int(el.self)
c.tree.XMLVersion = "1.0"
if b.v11 {
c.tree.XMLVersion = "1.1"
diff --git a/xdm/bursts_test.go b/xdm/bursts_test.go
index 46bd9a75..0f9d84ff 100644
--- a/xdm/bursts_test.go
+++ b/xdm/bursts_test.go
@@ -5,6 +5,8 @@ import (
"fmt"
"strings"
"testing"
+
+ "github.com/knroy/go-xml/v2/internal/xdmburst"
)
// burstDoc exercises what a burst must carry over from the whole tree: an
@@ -67,7 +69,13 @@ func TestParseBurstsMatchesTree(t *testing.T) {
if g := dump(r) + ancestry(r); g != want[i] {
t.Fatalf("burst %d differs\nwant\n%s\ngot\n%s", i, want[i], g)
}
- if i > 0 && got[i-1].Tree().ident() >= r.Tree().ident() {
+ // Small bursts share a tree (burstBatch); either way they order
+ // as they came.
+ if i == 0 {
+ continue
+ }
+ if p := got[i-1]; p.Tree() == r.Tree() && p.self >= r.self ||
+ p.Tree() != r.Tree() && p.Tree().ident() >= r.Tree().ident() {
t.Fatalf("burst %d does not order after burst %d", i, i-1)
}
}
@@ -121,8 +129,9 @@ func TestParseBurstsSpineStaysSmall(t *testing.T) {
b.WriteString("")
n := 0
opts := ParseOptions{bursts: &burster{
- path: []QName{{Local: "a"}, {Local: "b"}, {Local: "r"}},
- fn: func(*Node) error { n++; return nil },
+ steps: []xdmburst.Step{{Local: "a"}, {Local: "b"}, {Local: "r"}},
+ fn: func(*Node) error { n++; return nil },
+ states: []uint64{1},
}}
tree, err := parse(strings.NewReader(b.String()), "", opts)
if err != nil {
@@ -152,15 +161,18 @@ func TestParseBurstsCopyNoIndexes(t *testing.T) {
}
b.WriteString("")
i := 0
+ seen := map[*Tree]bool{}
err := ParseBursts(strings.NewReader(b.String()), ParseOptions{}, []QName{{Local: "a"}, {Local: "r"}},
func(r *Node) error {
c := r.tree
- if c.nameIx != nil || c.frameOne != nil || c.frameMany != nil {
+ // Checked on a tree's first burst: the intern below builds one.
+ if !seen[c] && (c.nameIx != nil || c.frameOne != nil || c.frameMany != nil) {
t.Fatalf("burst %d: copied indexes: %d names, %d+%d frames", i, len(c.nameIx), len(c.frameOne), len(c.frameMany))
}
if got, want := r.FirstChild().Name().Local, fmt.Sprintf("n%d", i); got != want {
t.Fatalf("burst %d: child %q, want %q", i, got, want)
}
+ seen[c] = true
if got := c.intern(QName{Local: fmt.Sprintf("n%d", i)}); got != r.FirstChild().name {
t.Fatalf("burst %d: intern finds %d, the child has %d", i, got, r.FirstChild().name)
}
@@ -180,9 +192,13 @@ func TestParseBurstsNoProlog(t *testing.T) {
err := ParseBursts(strings.NewReader(src), ParseOptions{}, []QName{{Local: "a"}, {Local: "r"}},
func(r *Node) error {
n++
- // The document node, a, r and x.
- if got := r.tree.n; got != 4 {
- t.Fatalf("burst %d: the tree holds %d records, want 4", n, got)
+ // The document node holds a, and a only bursts: r and x.
+ kids := uint32(0)
+ for range r.Parent().Children() {
+ kids++
+ }
+ if got, want := r.tree.n, 2+2*kids; got != want {
+ t.Fatalf("burst %d: the tree holds %d records, want %d", n, got, want)
}
return nil
})
@@ -210,8 +226,9 @@ func TestParseBurstsNamesStaySmall(t *testing.T) {
b.WriteString("")
var got []*Node
opts := ParseOptions{bursts: &burster{
- path: []QName{{Local: "a"}, {Local: "r"}},
- fn: func(r *Node) error { got = append(got, r); return nil },
+ steps: []xdmburst.Step{{Local: "a"}, {Local: "r"}},
+ fn: func(r *Node) error { got = append(got, r); return nil },
+ states: []uint64{1},
}}
tree, err := parse(strings.NewReader(b.String()), "", opts)
if err != nil {
@@ -281,3 +298,107 @@ func TestParseBurstsLargeNotCopied(t *testing.T) {
i++
}
}
+
+// A pattern with descendant steps and wildcards delivers each outermost
+// match once, in document order, with the matches nested in it inside it
+// and its ancestors' names and attributes as in the whole tree.
+func TestParseBurstsPattern(t *testing.T) {
+ const src = `` +
+ ``
+ for _, tc := range []struct {
+ name string
+ steps []xdmburst.Step
+ want string
+ }{
+ {"//rec", []xdmburst.Step{{Local: "rec", Descendant: true}}, "1(2,3)/docd 4(5)/docd/gg 8/docd"},
+ {"doc//rec", []xdmburst.Step{{Local: "doc"}, {Local: "rec", Descendant: true}}, "1(2,3)/docd 4(5)/docd/gg 8/docd"},
+ {"/*/*/rec", []xdmburst.Step{{AnyURI: true, AnyLocal: true}, {AnyURI: true, AnyLocal: true}, {Local: "rec"}}, "2(3)/docd/rec1 4(5)/docd/gg"},
+ {"/*/*:rec", []xdmburst.Step{{AnyURI: true, AnyLocal: true}, {AnyURI: true, Local: "rec"}}, "1(2,3)/docd 6/docd 8/docd"},
+ {"//q:*", []xdmburst.Step{{URI: "urn:q", AnyLocal: true, Descendant: true}}, "6/docd 7/docd/g/g"},
+ {"//g/rec", []xdmburst.Step{{Local: "g", Descendant: true}, {Local: "rec"}}, "4(5)/docd/gg"},
+ {"//*", []xdmburst.Step{{AnyURI: true, AnyLocal: true, Descendant: true}}, "d(1,2,3,g,4,5,6,,,7,8)/"},
+ } {
+ var got []string
+ err := parseBursts(strings.NewReader(src), ParseOptions{}, tc.steps, func(r *Node) error {
+ key := func(n *Node) string {
+ if a := n.Attr("", "k"); a != nil {
+ return a.Value()
+ }
+ if a := n.Attr("", "id"); a != nil {
+ return a.Value()
+ }
+ return ""
+ }
+ var in []string
+ for d := range r.Descendants() {
+ if d.Kind() == KindElement && d.Name().Local != "a" && d.Name().Local != "b" {
+ in = append(in, key(d))
+ }
+ }
+ s := key(r)
+ if len(in) > 0 {
+ s += "(" + strings.Join(in, ",") + ")"
+ }
+ anc := ""
+ for p := r.Parent(); p != nil && p.Kind() == KindElement; p = p.Parent() {
+ anc = "/" + p.Name().Local + key(p) + anc
+ }
+ if r.Parent().Kind() == KindDocument {
+ anc = "/"
+ }
+ got = append(got, s+anc)
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if g := strings.Join(got, " "); g != tc.want {
+ t.Errorf("%s: bursts %q, want %q", tc.name, g, tc.want)
+ }
+ }
+}
+
+// Under a deep spine, small bursts share copies of their ancestors: the
+// ancestors copied stay in proportion to the elements read, rather than
+// growing with bursts x depth.
+func TestParseBurstsSpineCopiedOnce(t *testing.T) {
+ const depth, n = 300, 3000
+ for _, wrap := range []bool{false, true} {
+ var b strings.Builder
+ b.WriteString(strings.Repeat(``, depth))
+ for i := range n {
+ if wrap {
+ b.WriteString("")
+ }
+ fmt.Fprintf(&b, ``, i)
+ if wrap {
+ b.WriteString("")
+ }
+ }
+ b.WriteString(strings.Repeat("", depth))
+ got := 0
+ bs := &burster{steps: []xdmburst.Step{{Local: "rec", Descendant: true}}, states: []uint64{1}, gens: []uint64{0},
+ fn: func(r *Node) error {
+ // Each burst still has its whole ancestry.
+ d := 0
+ for p := r.Parent(); p != nil && p.Kind() == KindElement; p = p.Parent() {
+ d++
+ }
+ if want := depth + map[bool]int{false: 0, true: 1}[wrap]; d != want || r.Attr("", "k").Value() != fmt.Sprint(got) {
+ t.Fatalf("burst %d: %d ancestors, want %d", got, d, want)
+ }
+ got++
+ return nil
+ }}
+ if _, err := parse(strings.NewReader(b.String()), "", ParseOptions{bursts: bs}); err != nil {
+ t.Fatal(err)
+ }
+ opened := depth + 2*n
+ if wrap {
+ opened += n
+ }
+ if got != n || bs.spineCopies > opened {
+ t.Errorf("wrap %v: %d bursts; %d ancestor records copied for %d elements read", wrap, got, bs.spineCopies, opened)
+ }
+ }
+}
diff --git a/xdm/parse.go b/xdm/parse.go
index 3bf95cef..23a290fe 100644
--- a/xdm/parse.go
+++ b/xdm/parse.go
@@ -370,7 +370,9 @@ func parse(r io.Reader, whole string, opts ParseOptions) (*Tree, error) {
if _, text := tok.(*xml.CharData); !text {
run.flush(&spaces, stripSpaceAt)
if bs != nil {
- bs.trimLeaves(cur, depth, &spaces, &names)
+ if err := bs.trimLeaves(cur, depth, &spaces, &names); err != nil {
+ return nil, err
+ }
}
}
diff --git a/xpath/context.go b/xpath/context.go
index 38813dc5..9f1a277f 100644
--- a/xpath/context.go
+++ b/xpath/context.go
@@ -323,6 +323,44 @@ type Env struct {
// accumulate its own. Nil means unbounded, which is what a hand-built
// Context gets; NewContext installs a budget.
nodes *int64
+ // limits are the bounds nodes and bytes are checked against, and the
+ // running totals a host's caller may bound beside them. A pointer that
+ // travels with the counters, so a nested evaluation that adopts the
+ // budget cannot raise its bounds. Nil means the package defaults.
+ limits *budgetLimits
+}
+
+// budgetLimits bounds an evaluation's result-node and byte counters.
+//
+// The counters themselves (Env.nodes, Env.bytes) are what is checked against
+// nodes and bytes. A host may refund them for what it knows is no longer held
+// -- XSLT does for a streamed loop's items -- so they measure what the
+// evaluation holds rather than everything it built; nodesBuilt and
+// bytesBuilt, when positive, bound everything it built, through totals no
+// refund lowers. Set through xpathleaf.SetLimits.
+type budgetLimits struct {
+ nodes, bytes int64 // resolved: math.MaxInt64 for none
+ nodesBuilt, bytesBuilt int64 // zero for none
+ builtNodes, builtBytes atomic.Int64
+ // nodesHit latches a node refusal, so that NodesRefused still reports a
+ // refusal a builder latched after a refund has brought the counter back
+ // under the bound.
+ nodesHit atomic.Bool
+}
+
+// nodeLimit and byteLimit are the bounds in force.
+func (e *Env) nodeLimit() int64 {
+ if e.limits == nil {
+ return MaxNodes
+ }
+ return e.limits.nodes
+}
+
+func (e *Env) byteLimit() int64 {
+ if e.limits == nil {
+ return MaxBytes
+ }
+ return e.limits.bytes
}
// staticContext is the static part of the context: the properties of the
@@ -444,8 +482,8 @@ func (c *Context) WithEnv(set func(e *Env)) *Context {
own := c.ev()
env := *own
set(&env)
- env.items, env.bytes, env.entities, env.nodes =
- own.items, own.bytes, own.entities, own.nodes
+ env.items, env.bytes, env.entities, env.nodes, env.limits =
+ own.items, own.bytes, own.entities, own.nodes, own.limits
n := *c
n.env = &env
return &n
@@ -856,12 +894,15 @@ func NewContext(item xdm.Item, funcs FunctionLibrary, configure ...func(e *Env))
r := &struct {
c Context
e Env
+ l budgetLimits
}{}
r.e.Ctx = context.Background()
for _, set := range configure {
set(&r.e)
}
r.e.items, r.e.bytes, r.e.nodes = new(int64), new(int64), new(int64)
+ r.l.nodes, r.l.bytes = MaxNodes, MaxBytes
+ r.e.limits = &r.l
// One entity-expansion allowance for the whole evaluation. Unlike items
// and bytes it is NOT reset per expression by Compiled.Eval: the reset is
// what the per-call mint already amounted to, and a budget an expression
@@ -1606,7 +1647,13 @@ func (c *Context) countBytes(n int) error {
if c == nil || c.ev().bytes == nil || n <= 0 {
return nil
}
- if atomic.AddInt64(c.ev().bytes, int64(n)) > MaxBytes {
+ e := c.ev()
+ lim := e.byteLimit()
+ over := atomic.AddInt64(e.bytes, int64(n)) > lim
+ if b := e.limits; !over && b != nil && b.bytesBuilt > 0 && b.builtBytes.Add(int64(n)) > b.bytesBuilt {
+ over, lim = true, b.bytesBuilt
+ }
+ if over {
// XPDY0130 is this engine's own code for "the evaluation asked for
// more than I will allocate", and the wording says bytes rather than
// items so the two refusals are not confused. The suite sanctions it
@@ -1617,7 +1664,7 @@ func (c *Context) countBytes(n int) error {
return fmt.Errorf(
"XPDY0130: evaluation built more than %d bytes of string "+
"content; the expression is building a string too large to "+
- "hold: %w", MaxBytes, xdm.ErrResourceLimit)
+ "hold: %w", lim, xdm.ErrResourceLimit)
}
return nil
}
@@ -1639,6 +1686,9 @@ func (c *Context) refundBytes(n int) {
return
}
atomic.AddInt64(c.ev().bytes, -int64(n))
+ if b := c.ev().limits; b != nil && b.bytesBuilt > 0 {
+ b.builtBytes.Add(-int64(n))
+ }
}
// ChargeBytes charges n bytes of built string content against the evaluation
@@ -1662,14 +1712,23 @@ func (c *Context) countNodes(n int) error {
if c == nil || c.ev().nodes == nil || n <= 0 {
return nil
}
- if atomic.AddInt64(c.ev().nodes, int64(n)) > MaxNodes {
- return nodesRefusal()
+ e := c.ev()
+ lim := e.nodeLimit()
+ over := atomic.AddInt64(e.nodes, int64(n)) > lim
+ if b := e.limits; !over && b != nil && b.nodesBuilt > 0 && b.builtNodes.Add(int64(n)) > b.nodesBuilt {
+ over, lim = true, b.nodesBuilt
+ }
+ if over {
+ if e.limits != nil {
+ e.limits.nodesHit.Store(true)
+ }
+ return nodesRefusal(lim)
}
return nil
}
// nodesRefusal is the XPDY0130 the node budget raises.
-func nodesRefusal() error {
+func nodesRefusal(lim int64) error {
// XPDY0130 again, and the wording says nodes so that the three
// refusals are not confused with one another. XPath 3.1 §2.3.1
// sanctions exactly this: "limitations may exist on the maximum
@@ -1680,7 +1739,7 @@ func nodesRefusal() error {
return fmt.Errorf(
"XPDY0130: evaluation constructed more than %d result-tree "+
"nodes; the transformation is building a tree too large to "+
- "hold: %w", MaxNodes, xdm.ErrResourceLimit)
+ "hold: %w", lim, xdm.ErrResourceLimit)
}
// NodesRefused reports the node-budget refusal if any charge has already
@@ -1689,10 +1748,19 @@ func nodesRefusal() error {
// temporary tree that crossed the bound would otherwise end as truncated
// output with no error.
func (c *Context) NodesRefused() error {
- if c == nil || c.ev().nodes == nil || atomic.LoadInt64(c.ev().nodes) <= MaxNodes {
+ if c == nil || c.ev().nodes == nil {
return nil
}
- return nodesRefusal()
+ e := c.ev()
+ switch b := e.limits; {
+ case atomic.LoadInt64(e.nodes) > e.nodeLimit():
+ return nodesRefusal(e.nodeLimit())
+ case b != nil && b.nodesBuilt > 0 && b.builtNodes.Load() > b.nodesBuilt:
+ return nodesRefusal(b.nodesBuilt)
+ case b != nil && b.nodesHit.Load():
+ return nodesRefusal(e.nodeLimit())
+ }
+ return nil
}
// ChargeNodes charges n constructed result-tree nodes against the evaluation
@@ -1817,6 +1885,10 @@ func (c *Context) AdoptBudget(src *Context) *Context {
if from.nodes != nil {
env.nodes = from.nodes
}
+ // The bounds travel with the counters they are checked against.
+ if from.limits != nil {
+ env.limits = from.limits
+ }
n.env = &env
return &n
}
diff --git a/xpath/leafhook.go b/xpath/leafhook.go
index 456ca95f..4e9556d9 100644
--- a/xpath/leafhook.go
+++ b/xpath/leafhook.go
@@ -1,6 +1,9 @@
package xpath
import (
+ "math"
+ "sync/atomic"
+
"github.com/knroy/go-xml/v2/internal/xpathleaf"
"github.com/knroy/go-xml/v2/xdm"
)
@@ -68,3 +71,50 @@ func hostOnCall(h *xpathleaf.Host, mark bool) *xpathleaf.Host {
n.Absent = n.Absent || mark
return &n
}
+
+func init() {
+ xpathleaf.SetLimits = func(c any, l xpathleaf.Limits) {
+ b := c.(*Context).ev().limits
+ if b == nil {
+ return
+ }
+ b.nodes, b.bytes = limitOr(l.Nodes, MaxNodes), limitOr(l.Bytes, MaxBytes)
+ b.nodesBuilt, b.bytesBuilt = max(l.NodesBuilt, 0), max(l.BytesBuilt, 0)
+ }
+ xpathleaf.Charged = func(c any) (int64, int64) {
+ e := c.(*Context).ev()
+ if e.nodes == nil || e.bytes == nil {
+ return 0, 0
+ }
+ return atomic.LoadInt64(e.nodes), atomic.LoadInt64(e.bytes)
+ }
+ xpathleaf.Refund = func(c any, nodes, bytes int64) {
+ e := c.(*Context).ev()
+ refund(e.nodes, nodes)
+ refund(e.bytes, bytes)
+ }
+}
+
+// limitOr is a caller's limit: def for zero, none for negative.
+func limitOr(v, def int64) int64 {
+ switch {
+ case v == 0:
+ return def
+ case v < 0:
+ return math.MaxInt64
+ }
+ return v
+}
+
+// refund lowers *p by n, never below zero.
+func refund(p *int64, n int64) {
+ if p == nil || n <= 0 {
+ return
+ }
+ for {
+ old := atomic.LoadInt64(p)
+ if atomic.CompareAndSwapInt64(p, old, max(old-n, 0)) {
+ return
+ }
+ }
+}
diff --git a/xslt/compile_instr.go b/xslt/compile_instr.go
index ed1314c4..a66b61da 100644
--- a/xslt/compile_instr.go
+++ b/xslt/compile_instr.go
@@ -977,7 +977,7 @@ func (c *compiler) compileForEach(n *xdm.Node, ns xpath.NamespaceResolver) (Inst
return nil, err
}
instr := &forEachInstr{sel: sel, sorts: sorts, body: body}
- c.noteBurstLoop(n, instr, sel)
+ c.noteBurstLoop(n, instr, sel, ns)
return instr, nil
}
@@ -2201,12 +2201,17 @@ func (i *iterateInstr) Execute(rt *runtime, out *outputBuilder) error {
// After xsl:break the rest of the document is still read, so that a
// malformed tail is the error it is when the tree is built first.
pos := 0
+ live := rt.liveLoop()
err := feed.read(func(it xdm.Item) error {
if broke {
return nil
}
pos++
+ live.begin(rt, out)
if err := step(pos, it, 0); err != errStopIterate {
+ if err == nil {
+ live.end(rt, out, carried)
+ }
return err
}
return nil
@@ -2302,7 +2307,7 @@ func (c *compiler) compileIterate(n *xdm.Node, ns *nsResolver) (Instruction, err
return nil, err
}
instr.body = body
- c.noteBurstLoop(n, instr, sel)
+ c.noteBurstLoop(n, instr, sel, ns)
return instr, nil
}
diff --git a/xslt/copyfuncs.go b/xslt/copyfuncs.go
index 958f684d..d118b856 100644
--- a/xslt/copyfuncs.go
+++ b/xslt/copyfuncs.go
@@ -43,6 +43,9 @@ func registerCopyFuncs(l *xpath.Library) {
if err != nil {
return nil, err
}
+ if err := chargeCopies(ctx, args, false); err != nil {
+ return nil, err
+ }
return mapItemsChecked(ctx, args, checkParentlessQName,
func(it xdm.Item) xdm.Item {
return noteCopy(rt, it, copyItem(it))
@@ -57,6 +60,9 @@ func registerCopyFuncs(l *xpath.Library) {
if err != nil {
return nil, err
}
+ if err := chargeCopies(ctx, args, true); err != nil {
+ return nil, err
+ }
return mapItems(ctx, args, func(it xdm.Item) xdm.Item {
return noteCopy(rt, it, snapshotItem(it))
})
@@ -126,6 +132,31 @@ func mapItems(ctx *xpath.Context, args []xdm.Sequence,
return out, nil
}
+// chargeCopies charges the nodes fn:copy-of, or with spine fn:snapshot, is
+// about to copy, as xsl:copy-of charges its copies: before they are made, so
+// a refusal allocates none of them. A snapshot copies each ancestor too.
+// An absent context item is left for the copy to report.
+func chargeCopies(ctx *xpath.Context, args []xdm.Sequence, spine bool) error {
+ in, err := inputItems(ctx, args)
+ if err != nil {
+ return nil
+ }
+ n := 0
+ for _, it := range in {
+ v, ok := it.(*xdm.Node)
+ if !ok {
+ continue
+ }
+ n += countSubtree(v)
+ if spine && v.Kind() != xdm.KindDocument {
+ for a := v.Parent(); a != nil; a = a.Parent() {
+ n += 1 + a.NumAttrs() + a.NumNamespaceDecls()
+ }
+ }
+ }
+ return ctx.ChargeNodes(n)
+}
+
// copyItem is internal:copy-item: a deep copy of a node, anything else
// unchanged.
func copyItem(it xdm.Item) xdm.Item {
diff --git a/xslt/instructions.go b/xslt/instructions.go
index 6cc10189..fb096615 100644
--- a/xslt/instructions.go
+++ b/xslt/instructions.go
@@ -1666,12 +1666,18 @@ type forEachInstr struct {
func (i *forEachInstr) Execute(rt *runtime, out *outputBuilder) error {
if f := rt.burstsFor(i); f != nil {
pos := 0
+ live := rt.liveLoop()
return f.read(func(it xdm.Item) error {
if err := rt.ctx.Err(); err != nil {
return err
}
pos++
- return execSequence(i.body, rt.withCurrent(it, pos, 0).clearCurrentRule(), out)
+ live.begin(rt, out)
+ if err := execSequence(i.body, rt.withCurrent(it, pos, 0).clearCurrentRule(), out); err != nil {
+ return err
+ }
+ live.end(rt, out, nil)
+ return nil
})
}
seq, err := i.sel.Eval(rt.ctx)
diff --git a/xslt/livebudget.go b/xslt/livebudget.go
new file mode 100644
index 00000000..50a6ccb7
--- /dev/null
+++ b/xslt/livebudget.go
@@ -0,0 +1,194 @@
+package xslt
+
+// What a streamed loop holds, rather than everything it built.
+//
+// The result-node and byte budgets (xpath.MaxNodes, xpath.MaxBytes) are
+// cumulative: nothing tells them a temporary tree or a string has been
+// dropped, so they bound what a transform builds. Over a tree that is the
+// same thing as what it holds, near enough -- the whole document is held
+// anyway. A streamed loop is the one place that changes: it reads a document
+// of any size in bursts, and a correct stream with a two-node temporary
+// variable per record was refused after a million records at 3 MB of heap.
+//
+// So an item of a streamed xsl:for-each or xsl:iterate (a burst feed drives
+// it) hands back, when it finishes, what it built and does not hold. Nothing
+// is tracked per tree; the item is judged as a whole, conservatively. Its
+// charges are the rise of the two counters across it. It holds nothing it
+// built when all of these are so:
+//
+// - it wrote nothing to the loop's output -- no item, child, attribute or
+// namespace, and no text merged into the last text node (outSig). Output
+// is held for good, and anything written may refer to what it built;
+// - it forced no global variable, whose value outlives it;
+// - the runtime records no copies (accumOrigin, made only for a stylesheet
+// with accumulators), since that map keeps every copy alive;
+// - what xsl:next-iteration carries on is nodes and atomic values only,
+// which are measured: each distinct tree in full (treeSize) and each
+// value's string. A map, an array or a function item could hold
+// anything, so it ends the hand-back.
+//
+// The burst admission rules (streamburst.go) are what make that list
+// complete: a burst body calls no template or stylesheet function, uses no
+// key, accumulator, message or result document, and carries only atomic
+// values or a new temporary tree from one item to the next. Any other item
+// keeps everything charged, which is the old cumulative behaviour.
+//
+// The carried state is charged while it is carried: it stays on the counters
+// (liveLoop.nodes and .bytes say how much of them it is) and is handed back
+// by the item that replaces it. A stream whose carried state grows is thus
+// refused once that state crosses the bound, as it should be.
+//
+// Unless the caller bounds the total itself (MaxNodesBuilt, MaxBytesBuilt),
+// an item hands back at most liveNodeAllowance nodes and liveByteAllowance
+// bytes. The total built is then at most the bound plus the allowance per
+// item: linear in the input, and a stream whose work per item keeps growing
+// -- carrying a tree and copying it into the next one -- is still refused in
+// bounded time rather than only once what it holds reaches the bound, which
+// at quadratic work could take hours.
+
+import (
+ "github.com/knroy/go-xml/v2/internal/xpathleaf"
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// The most a streamed item hands back when the caller sets no total. Far
+// above what a record's temporaries take (a few nodes and a few hundred bytes
+// in every streamed shape measured), and small enough that a stream whose
+// per-item work grows is refused within seconds.
+const (
+ liveNodeAllowance = 4096
+ liveByteAllowance = 64 << 10
+)
+
+// liveLoop is one streamed loop's accounting across its items.
+type liveLoop struct {
+ on bool
+ // nodes and bytes are the part of the counters that stands for the
+ // iteration parameters being carried.
+ nodes, bytes int64
+ // The state at the start of the current item.
+ n0, b0 int64
+ forced int
+ sig outSig
+}
+
+// outSig is what an item could change by writing to its output.
+type outSig struct{ items, kids, attrs, ns, text int }
+
+func signature(out *outputBuilder) outSig {
+ s := outSig{items: len(out.Items())}
+ var last *xdm.Node
+ if e := out.Open(); e != nil {
+ s.kids, s.attrs, s.ns = e.NumChildren(), e.NumAttrs(), e.NumNamespaceDecls()
+ last = e.LastChild()
+ } else if s.items > 0 {
+ last, _ = out.Items()[s.items-1].(*xdm.Node)
+ }
+ if last != nil && last.Kind() == xdm.KindText {
+ s.text = len(last.Value())
+ }
+ return s
+}
+
+// liveLoop starts the accounting of a streamed loop writing to out.
+func (rt *runtime) liveLoop() *liveLoop {
+ return &liveLoop{on: rt.accumOrigin == nil}
+}
+
+// begin marks the start of an item.
+func (l *liveLoop) begin(rt *runtime, out *outputBuilder) {
+ if !l.on {
+ return
+ }
+ l.n0, l.b0 = xpathleaf.Charged(rt.ctx)
+ l.forced, l.sig = rt.globalsForced, signature(out)
+}
+
+// end hands back what the item that began at the last begin built and does
+// not hold, carried being the iteration parameters it leaves for the next.
+func (l *liveLoop) end(rt *runtime, out *outputBuilder, carried []xdm.Sequence) {
+ if !l.on {
+ return
+ }
+ n1, b1 := xpathleaf.Charged(rt.ctx)
+ cn, cb, ok := carriedSize(carried)
+ if !ok || n1 < l.n0 || b1 < l.b0 || rt.globalsForced != l.forced || signature(out) != l.sig {
+ // Everything it built stays charged, and so does what the previous
+ // items left carried: none of it is known to be dropped.
+ l.nodes, l.bytes = 0, 0
+ return
+ }
+ an, ab := int64(liveNodeAllowance), int64(liveByteAllowance)
+ if rt.opts.MaxNodesBuilt != 0 {
+ an = -1
+ }
+ if rt.opts.MaxBytesBuilt != 0 {
+ ab = -1
+ }
+ xpathleaf.Refund(rt.ctx, settle(n1-l.n0, &l.nodes, cn, an), settle(b1-l.b0, &l.bytes, cb, ab))
+}
+
+// settle returns how much of an item's charges to hand back: of built, and of
+// held (what the previous items left charged for the carried state), all but
+// need (what the carried state now takes), and at most allow when that is not
+// negative. held becomes what stays charged for the carried state.
+func settle(built int64, held *int64, need, allow int64) int64 {
+ pool := built + *held
+ r := max(pool-need, 0)
+ if allow >= 0 {
+ r = min(r, allow)
+ }
+ *held = min(need, pool-r)
+ return r
+}
+
+// carriedSize measures the iteration parameters an item carries on: nodes and
+// string bytes, each tree counted once in full. ok is false for an item it
+// cannot see into.
+func carriedSize(carried []xdm.Sequence) (nodes, bytes int64, ok bool) {
+ var seen map[*xdm.Node]bool
+ for _, seq := range carried {
+ for _, it := range seq {
+ switch v := it.(type) {
+ case *xdm.Atomic:
+ bytes += int64(len(v.Str()))
+ case *xdm.Node:
+ r := v
+ for p := r.Parent(); p != nil; p = p.Parent() {
+ r = p
+ }
+ if seen[r] {
+ continue
+ }
+ if seen == nil {
+ seen = map[*xdm.Node]bool{}
+ }
+ seen[r] = true
+ n, b := treeSize(r)
+ nodes, bytes = nodes+n, bytes+b
+ default:
+ return 0, 0, false
+ }
+ }
+ }
+ return nodes, bytes, true
+}
+
+// treeSize is the nodes of n's subtree, counted as the builder charges them
+// (countSubtree), and the bytes of the text, attribute, comment and
+// processing-instruction values in it.
+func treeSize(n *xdm.Node) (nodes, bytes int64) {
+ nodes = int64(1 + n.NumAttrs() + n.NumNamespaceDecls())
+ for a := range n.Attrs() {
+ bytes += int64(len(a.Value()))
+ }
+ switch n.Kind() {
+ case xdm.KindText, xdm.KindComment, xdm.KindPI, xdm.KindAttribute:
+ bytes += int64(len(n.Value()))
+ }
+ for ch := range n.Children() {
+ cn, cb := treeSize(ch)
+ nodes, bytes = nodes+cn, bytes+cb
+ }
+ return nodes, bytes
+}
diff --git a/xslt/livebudget_test.go b/xslt/livebudget_test.go
new file mode 100644
index 00000000..251fb1a3
--- /dev/null
+++ b/xslt/livebudget_test.go
@@ -0,0 +1,244 @@
+package xslt
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/knroy/go-xml/v2/xdm"
+)
+
+// liveRun runs body inside a streamable xsl:source-document over n records,
+// in bursts or over the tree, with the budget fields of opts.
+func liveRun(t *testing.T, body string, n int, bursts bool, opts TransformOptions) (string, error) {
+ t.Helper()
+ dir := t.TempDir()
+ var b strings.Builder
+ b.WriteString("")
+ for i := range n {
+ fmt.Fprintf(&b, `t`, i, i%97)
+ }
+ b.WriteString("")
+ if err := os.WriteFile(filepath.Join(dir, "doc.xml"), []byte(b.String()), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ src := `` +
+ `` + body +
+ ``
+ tree, err := xdm.ParseString(src, xdm.ParseOptions{BaseURI: fileURIOf(filepath.Join(dir, "s.xsl"))})
+ if err != nil {
+ t.Fatal(err)
+ }
+ sheet, err := Compile(tree.Root, CompileOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ docs, err := NewFileResolver(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ran := false
+ noBursts, traceBursts = !bursts, func(string) { ran = true }
+ defer func() { noBursts, traceBursts = false, nil }()
+ opts.InitialTemplate, opts.Documents = "main", docs
+ res, err := sheet.Transform(context.Background(), nil, opts)
+ if ran != bursts {
+ t.Fatalf("bursts ran %v, want %v", ran, bursts)
+ }
+ if err != nil {
+ return "", err
+ }
+ return res.String(), nil
+}
+
+func refused(err error, what string) bool {
+ return err != nil && errors.Is(err, xdm.ErrResourceLimit) &&
+ strings.Contains(err.Error(), "XPDY0130") && strings.Contains(err.Error(), what)
+}
+
+// The bound lane's shapes (m10 bound, findings B and C): a stream whose state
+// stays flat built more than the budget and was refused. Small bounds stand in
+// for the defaults, which the same shapes crossed at a million records.
+const (
+ liveOC = ``
+ // V10: a two-node temporary tree per record.
+ liveV10 = `x`
+ // V1: an fn:copy-of per record, now charged.
+ liveV1 = `` + liveOC + ``
+ // L2: value-of into a body variable.
+ liveL2 = `` + liveOC + ``
+ // L6: string-join per record.
+ liveL6 = `` + liveOC + ``
+ // G8: a carried temporary tree copied into the next, growing.
+ liveG8 = ``
+ // G5: a carried sequence of strings, growing while each item builds
+ // one; and the same for a sequence of new elements.
+ liveG5 = ``
+ liveG5n = ``
+ // G4: a carried string, growing.
+ liveG4 = ``
+)
+
+// MaxNodes: a streamed item hands back the temporary trees and copies it
+// dropped, so flat state streams past the bound; over the tree nothing is
+// handed back and the same bound refuses.
+func TestLiveNodesStreamed(t *testing.T) {
+ for _, s := range []struct{ name, body, want string }{
+ {"V10", liveV10, ""},
+ {"V1", liveV1, ``},
+ } {
+ opts := TransformOptions{MaxNodes: 500}
+ got, err := liveRun(t, s.body, 2000, true, opts)
+ if err != nil || !strings.Contains(got, s.want) {
+ t.Errorf("%s streamed: %q, %v", s.name, got, err)
+ }
+ if _, err := liveRun(t, s.body, 2000, false, opts); !refused(err, "more than 500 result-tree nodes") {
+ t.Errorf("%s tree: want the node refusal, got %v", s.name, err)
+ }
+ }
+}
+
+// MaxBytes on the same terms, for string content.
+func TestLiveBytesStreamed(t *testing.T) {
+ for _, s := range []struct{ name, body string }{{"L2", liveL2}, {"L6", liveL6}} {
+ opts := TransformOptions{MaxBytes: 4096}
+ got, err := liveRun(t, s.body, 2000, true, opts)
+ if err != nil || !strings.Contains(got, ``) {
+ t.Errorf("%s streamed: %q, %v", s.name, got, err)
+ }
+ if _, err := liveRun(t, s.body, 2000, false, opts); !refused(err, "more than 4096 bytes") {
+ t.Errorf("%s tree: want the byte refusal, got %v", s.name, err)
+ }
+ }
+}
+
+// What an item carries on stays charged, so growing state is refused by the
+// held bound however much is handed back.
+func TestLiveGrowingRefused(t *testing.T) {
+ if _, err := liveRun(t, liveG8, 2000, true,
+ TransformOptions{MaxNodes: 500, MaxNodesBuilt: -1}); !refused(err, "more than 500 result-tree nodes") {
+ t.Errorf("G8: want the node refusal, got %v", err)
+ }
+ if _, err := liveRun(t, liveG4, 2000, true,
+ TransformOptions{MaxBytes: 4096, MaxBytesBuilt: -1}); !refused(err, "more than 4096 bytes") {
+ t.Errorf("G4: want the byte refusal, got %v", err)
+ }
+ // Here no one item builds much: only keeping the carried state charged
+ // refuses them.
+ if _, err := liveRun(t, liveG5, 2000, true,
+ TransformOptions{MaxBytes: 4096, MaxBytesBuilt: -1}); !refused(err, "more than 4096 bytes") {
+ t.Errorf("G5: want the byte refusal, got %v", err)
+ }
+ if _, err := liveRun(t, liveG5n, 2000, true,
+ TransformOptions{MaxNodes: 500, MaxNodesBuilt: -1}); !refused(err, "more than 500 result-tree nodes") {
+ t.Errorf("G5 elements: want the node refusal, got %v", err)
+ }
+ // Under the bound, they run.
+ for _, b := range []string{liveG5, liveG5n} {
+ if got, err := liveRun(t, b, 400, true, TransformOptions{MaxNodes: 500, MaxBytes: 4096}); err != nil || !strings.Contains(got, ``) {
+ t.Errorf("under the bound: %q, %v", got, err)
+ }
+ }
+}
+
+// MaxNodesBuilt: by default an item hands back at most liveNodeAllowance, so
+// G8, whose work per item grows, is refused by the default bound well before
+// what it holds reaches it; negative lifts that, and a positive value bounds
+// the total instead.
+func TestLiveNodesBuilt(t *testing.T) {
+ const n = 1800 // holds 10,800 nodes at the end, builds about 9.7 million
+ if _, err := liveRun(t, liveG8, n, true, TransformOptions{}); !refused(err, "more than 2000000 result-tree nodes") {
+ t.Errorf("default: want the node refusal, got %v", err)
+ }
+ got, err := liveRun(t, liveG8, n, true, TransformOptions{MaxNodesBuilt: -1})
+ if err != nil || !strings.Contains(got, fmt.Sprintf(``, n)) {
+ t.Errorf("negative: %q, %v", got, err)
+ }
+ if _, err := liveRun(t, liveG8, n, true, TransformOptions{MaxNodesBuilt: 3_000_000}); !refused(err, "more than 3000000 result-tree nodes") {
+ t.Errorf("positive: want the total refusal, got %v", err)
+ }
+ if _, err := liveRun(t, liveV10, 2000, true, TransformOptions{MaxNodes: 500, MaxNodesBuilt: 20_000_000}); err != nil {
+ t.Errorf("positive above the work: %v", err)
+ }
+}
+
+// MaxBytesBuilt bounds every byte built, whatever is handed back.
+func TestLiveBytesBuilt(t *testing.T) {
+ if _, err := liveRun(t, liveL6, 2000, true, TransformOptions{MaxBytes: 4096, MaxBytesBuilt: 50_000}); !refused(err, "more than 50000 bytes") {
+ t.Errorf("want the total refusal, got %v", err)
+ }
+ if _, err := liveRun(t, liveL6, 2000, true, TransformOptions{MaxBytes: 4096, MaxBytesBuilt: -1}); err != nil {
+ t.Errorf("negative: %v", err)
+ }
+}
+
+// fn:copy-of and fn:snapshot are charged as xsl:copy-of is; they were not
+// charged at all.
+func TestCopyFunctionsCharged(t *testing.T) {
+ for _, f := range []string{"copy-of", "snapshot"} {
+ body := ``
+ _, err := liveRun(t, body, 200, false, TransformOptions{MaxNodes: 500})
+ if !refused(err, "more than 500 result-tree nodes") {
+ t.Errorf("%s: want the node refusal, got %v", f, err)
+ }
+ }
+}
+
+// An item that writes to its output, or forces a global variable, keeps
+// charged everything it built: the output and the global's value outlive it.
+func TestLiveKeepsWhatItHolds(t *testing.T) {
+ opts := TransformOptions{MaxNodes: 500}
+ out := ``
+ if _, err := liveRun(t, out, 600, true, opts); !refused(err, "more than 500 result-tree nodes") {
+ t.Errorf("output: want the node refusal, got %v", err)
+ }
+ // $g, 301 nodes, is built inside the first item and copied after the
+ // loop: 602 nodes held.
+ global := `` +
+ `x`
+ _, err := liveRunGlobal(t, global, ``, opts)
+ if !refused(err, "more than 500 result-tree nodes") {
+ t.Errorf("global: want the node refusal, got %v", err)
+ }
+}
+
+// liveRunGlobal runs template main, body, over 10 records, with a top-level
+// declaration.
+func liveRunGlobal(t *testing.T, body, decl string, opts TransformOptions) (string, error) {
+ t.Helper()
+ dir := t.TempDir()
+ doc := "" + strings.Repeat(`t`, 10) + ""
+ if err := os.WriteFile(filepath.Join(dir, "doc.xml"), []byte(doc), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ src := `` + decl +
+ `` + body + ``
+ tree, err := xdm.ParseString(src, xdm.ParseOptions{BaseURI: fileURIOf(filepath.Join(dir, "s.xsl"))})
+ if err != nil {
+ t.Fatal(err)
+ }
+ sheet, err := Compile(tree.Root, CompileOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ docs, err := NewFileResolver(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ran := false
+ traceBursts = func(string) { ran = true }
+ defer func() { traceBursts = nil }()
+ opts.InitialTemplate, opts.Documents = "main", docs
+ res, err := sheet.Transform(context.Background(), nil, opts)
+ if !ran {
+ t.Fatal("bursts did not run")
+ }
+ if err != nil {
+ return "", err
+ }
+ return res.String(), nil
+}
diff --git a/xslt/runtime.go b/xslt/runtime.go
index 07287806..ae0aaf8c 100644
--- a/xslt/runtime.go
+++ b/xslt/runtime.go
@@ -77,6 +77,10 @@ type runtime struct {
type transformState struct {
sheet *Stylesheet
+ // globalsForced counts the global variables evaluated so far, so that a
+ // streamed item that forced one keeps charged what it built (liveItem).
+ globalsForced int
+
// locals holds the local variables and parameters in scope; see
// locals.go. A transform fn:transform starts shares its caller's.
locals *xpathleaf.Locals
@@ -892,6 +896,10 @@ func newRuntime(s *Stylesheet, ctx context.Context, root *xdm.Node, opts Transfo
return rt.deferredError(c, name)
}
})
+ xpathleaf.SetLimits(xctx, xpathleaf.Limits{
+ Nodes: int64(opts.MaxNodes), Bytes: opts.MaxBytes,
+ NodesBuilt: opts.MaxNodesBuilt, BytesBuilt: opts.MaxBytesBuilt,
+ })
// A transform started by fn:transform continues its caller's item and byte
// allowances rather than restarting on fresh ones, the same policy the
// recursion depth above follows. The counters travel with their held flags;
@@ -1062,6 +1070,8 @@ func (rt *runtime) evalGlobals(s *Stylesheet, opts TransformOptions) error {
// eval computes one global's value.
eval := func(g *Variable) (xdm.Sequence, error) {
+ // Its value outlives whatever streamed item forced it; see liveItem.
+ rt.globalsForced++
// The DECLARED local name, which is what a reference inside a match
// pattern is written with and all recoverPatternError can compare.
if rt.globalActive == nil {
diff --git a/xslt/streamburst.go b/xslt/streamburst.go
index 76c6ed46..582618bb 100644
--- a/xslt/streamburst.go
+++ b/xslt/streamburst.go
@@ -4,11 +4,13 @@ package xslt
//
// The narrowest common streaming pattern -- a streamable xsl:source-document
// whose body, under literal result elements at most, is one xsl:for-each or
-// xsl:iterate over a child path of the document, reading only the item it is
-// given -- is executed without building the document. xdm.ParseBursts hands
-// over one selected element at a time, in a small tree of its own holding
-// the element and copies of its ancestors, and the loop's body runs on it
-// before the next one is read.
+// xsl:iterate over a path of the document's elements, reading only the item
+// it is given -- is executed without building the document. The path's
+// steps are child or descendant steps by name or wildcard (//rec, /*/*/rec,
+// *:rec), or outermost() of such a path. xdm.ParseBursts hands over one
+// outermost match at a time, in a small tree of its own holding the element
+// and copies of its ancestors, and the loop's body runs on it, and on any
+// match nested in it, before the next one is read.
//
// That is correct only where the body cannot tell a burst from the whole
// tree, so the plan is made from the stylesheet source and fails closed:
@@ -44,6 +46,7 @@ import (
"os"
"strings"
+ "github.com/knroy/go-xml/v2/internal/xdmburst"
"github.com/knroy/go-xml/v2/internal/xpathleaf"
"github.com/knroy/go-xml/v2/xdm"
"github.com/knroy/go-xml/v2/xpath"
@@ -61,8 +64,9 @@ var (
// burstPlan is a streamable xsl:source-document that can run in bursts.
type burstPlan struct {
- // path is the select expression's child steps, from the document node.
- path []xdm.QName
+ // path is the pattern of the select expression's steps, from the
+ // document node, that the bursts are cut at.
+ path []xdmburst.Step
// target is the xsl:for-each or xsl:iterate element, and instr its
// compiled form; sel is its select expression, evaluated against each
// burst's document node to apply the last step's predicates.
@@ -119,7 +123,7 @@ type burstResolver interface {
// parseBursts is xdm.ParseBursts over the document uri names, read with
// the options and limits ResolveDocument would read it with. An error
// from fn is returned as it is.
- parseBursts(ctx *xpath.Context, uri, base string, path []xdm.QName, fn func(*xdm.Node) error) error
+ parseBursts(ctx *xpath.Context, uri, base string, path []xdmburst.Step, fn func(*xdm.Node) error) error
}
// planBursts decides, from the stylesheet source, whether the streamable
@@ -210,8 +214,8 @@ func selectTargetOK(el *xdm.Node) bool {
// burstSplit is a select expression taken apart around its record path.
type burstSplit struct {
- // path is the element names the bursts are cut at.
- path []xdm.QName
+ // path is the pattern the bursts are cut at.
+ path []xdmburst.Step
// items is the record path, put puts an expression in its place, and
// outer holds the whole expression, whatever put has made of it.
items xpath.Expr
@@ -440,57 +444,103 @@ func (s *burstSplit) slot(e xpath.Expr, put func(xpath.Expr)) bool {
return s.walk(e, put)
}
-// recordPath returns the element names to cut bursts at when e is a record
-// path, nil otherwise. A record path is child steps by name from the
-// document node, the last of them perhaps with predicates that do not
-// depend on position (as burstPath requires), followed by steps that stay
-// within the element -- down, or to its attributes, with any predicates
-// burstSafe allows -- and perhaps one last expression that makes atomic
-// values or copies. Each burst then holds what the path returns from one
-// element, in document order, and the bursts in turn hold all of it.
-func recordPath(e xpath.Expr) []xdm.QName {
+// recordPath returns the pattern to cut bursts at when e is a record
+// path, nil otherwise. A record path is steps by name from the document
+// node (burstPattern), the last of them perhaps with predicates that do not
+// depend on position, followed by steps that stay within the element --
+// down, or to its attributes, with any predicates burstSafe allows -- and
+// perhaps one last expression that makes atomic values or copies; or
+// outermost() or innermost() of a record path, perhaps followed by such
+// steps. Each burst then holds what the path returns from one outermost
+// match of the pattern, in document order, and the bursts in turn hold all
+// of it: a match inside another is in the same burst, so outermost() and
+// innermost() see in each burst every node of the whole sequence that is
+// an ancestor or descendant of one of its own.
+func recordPath(e xpath.Expr) []xdmburst.Step {
var steps []xpath.Expr
switch x := e.(type) {
case *xpath.Step:
steps = []xpath.Expr{x}
+ case *xpath.FuncCall:
+ if outerInner(x) {
+ return recordPath(x.Args[0])
+ }
+ return nil
case *xpath.PathExpr:
steps = x.Steps
+ if f, ok := firstOf(steps).(*xpath.FuncCall); ok && !x.Root && outerInner(f) {
+ if !pathTail(steps[1:]) {
+ return nil
+ }
+ return recordPath(f)
+ }
if _, ok := firstOf(steps).(*xpath.ContextItem); ok && !x.Root {
steps = steps[1:]
}
default:
return nil
}
- var path []xdm.QName
- k := 0
+ path, k, _ := burstPattern(steps)
+ if len(path) == 0 || len(path) > xdmburst.MaxSteps || !pathTail(steps[k:]) {
+ return nil
+ }
+ return path
+}
+
+// outerInner reports whether f is fn:outermost or fn:innermost.
+func outerInner(f *xpath.FuncCall) bool {
+ return f.Name.URI == fnNS && len(f.Args) == 1 &&
+ (f.Name.Local == "outermost" || f.Name.Local == "innermost")
+}
+
+// burstPattern takes from steps the pattern of element name tests bursts
+// can be cut at: child or descendant steps (// included) whose test is a
+// name or a wildcard, and returns it with the number of steps it used and
+// whether they include a descendant step. Only the last may have
+// predicates, which must not depend on position: they are evaluated in the
+// burst, where siblings are missing, and a match nested in another is in
+// the other's burst. Earlier steps match ancestors, which a burst holds only
+// shallow copies of.
+func burstPattern(steps []xpath.Expr) (path []xdmburst.Step, k int, desc bool) {
+ pending := false // a descendant-or-self::node() step was taken
for ; k < len(steps); k++ {
st, ok := steps[k].(*xpath.Step)
- if !ok || st.Axis != xpath.AxisChild {
+ if !ok {
+ break
+ }
+ if kt, ok := st.Test.(*xpath.KindTest); ok && kt.Any && st.Axis == xpath.AxisDescendantOrSelf &&
+ len(st.Predicates) == 0 && !pending && k+1 < len(steps) {
+ pending = true
+ continue
+ }
+ d := pending
+ if st.Axis == xpath.AxisDescendant {
+ d = true
+ } else if st.Axis != xpath.AxisChild {
break
}
nt, ok := st.Test.(*xpath.NameTest)
- if !ok || nt.AnyURI || nt.AnyLocal {
+ if !ok {
break
}
- if len(st.Predicates) > 0 {
- // Positional predicates count within the element above,
- // which is then the burst.
- for _, p := range st.Predicates {
- if !booleanShaped(p) || !burstSafe(p, true) {
- goto tail
- }
+ for _, p := range st.Predicates {
+ if !booleanShaped(p) || !burstSafe(p, true) {
+ goto tail
}
- path = append(path, xdm.QName{URI: nt.Name.URI, Local: nt.Name.Local})
- k++
- break
}
- path = append(path, xdm.QName{URI: nt.Name.URI, Local: nt.Name.Local})
+ pending = false
+ desc = desc || d
+ path = append(path, xdmburst.Step{URI: nt.Name.URI, Local: nt.Name.Local,
+ AnyURI: nt.AnyURI, AnyLocal: nt.AnyLocal, Descendant: d})
+ if len(st.Predicates) > 0 {
+ return path, k + 1, desc
+ }
}
tail:
- if len(path) == 0 || !pathTail(steps[k:]) {
- return nil
+ if pending {
+ k-- // the descendant-or-self::node() step belongs to the tail
}
- return path
+ return path, k, desc
}
// pathTail reports whether steps stay within the nodes they start from:
@@ -591,42 +641,37 @@ func lreAttrOK(a *xdm.Node) bool {
return false
}
-// burstPath returns the element names of the loop's select, a path of child
-// steps from the document node, or nil. Only the last step may have
-// predicates, and they must not depend on position: a burst's element is
-// the first and only one of its name in its tree.
-func burstPath(target *xdm.Node) (path []xdm.QName, filtered bool) {
- e := parseAt(target, target.AttrValue("select"))
+// burstPath returns the pattern of the loop's select, a path of steps
+// from the document node that burstPattern takes whole, or outermost() or
+// innermost() of one; or nil. filtered says each burst's element need not be
+// the burst's only item: the last step has predicates, or a match may hold
+// another.
+func burstPath(target *xdm.Node) (path []xdmburst.Step, filtered bool) {
+ return loopPattern(parseAt(target, target.AttrValue("select")))
+}
+
+func loopPattern(e xpath.Expr) (path []xdmburst.Step, filtered bool) {
var steps []xpath.Expr
switch x := e.(type) {
case *xpath.PathExpr:
steps = x.Steps
case *xpath.Step:
steps = []xpath.Expr{x}
+ case *xpath.FuncCall:
+ if outerInner(x) {
+ path, _ = loopPattern(x.Args[0])
+ return path, true
+ }
+ return nil, false
default:
return nil, false
}
- for k, s := range steps {
- st, ok := s.(*xpath.Step)
- if !ok || st.Axis != xpath.AxisChild {
- return nil, false
- }
- nt, ok := st.Test.(*xpath.NameTest)
- if !ok || nt.AnyURI || nt.AnyLocal {
- return nil, false
- }
- if len(st.Predicates) > 0 && k < len(steps)-1 {
- return nil, false
- }
- for _, p := range st.Predicates {
- if !booleanShaped(p) || !burstSafe(p, true) {
- return nil, false
- }
- }
- filtered = len(st.Predicates) > 0
- path = append(path, xdm.QName{URI: nt.Name.URI, Local: nt.Name.Local})
+ path, k, desc := burstPattern(steps)
+ if k != len(steps) || len(path) == 0 || len(path) > xdmburst.MaxSteps {
+ return nil, false
}
- return path, filtered
+ last := steps[k-1].(*xpath.Step)
+ return path, desc || len(last.Predicates) > 0
}
// parseAt parses src as an expression written on el, or returns nil.
@@ -1019,13 +1064,83 @@ func pathSafe(steps []xpath.Expr, noPos, upEnd bool) bool {
}
// noteBurstLoop records instr as the compiled loop of the burst plan being
-// compiled, when n is its loop element.
-func (c *compiler) noteBurstLoop(n *xdm.Node, instr Instruction, sel *xpath.Compiled) {
+// compiled, when n is its loop element, and sel, written where ns resolves
+// names, as its select; a select that starts with // is evaluated in each
+// burst as descendant:: (descendantFirst).
+func (c *compiler) noteBurstLoop(n *xdm.Node, instr Instruction, sel *xpath.Compiled,
+ ns xpath.NamespaceResolver) {
+
if c.burst != nil && c.burst.target == n {
c.burst.instr, c.burst.sel = instr, sel
+ if e := reparse(sel, ns); e != nil && descendantFirst(e) {
+ c.burst.sel = xpathleaf.WithExpr(sel, e).(*xpath.Compiled)
+ }
}
}
+// reparse parses the source of sel, written where ns resolves names,
+// exactly as compileExpr parsed it, or returns nil.
+func reparse(sel *xpath.Compiled, ns xpath.NamespaceResolver) xpath.Expr {
+ o := exprOptions(ns)
+ var (
+ e xpath.Expr
+ err error
+ )
+ if o.RefFloor != 0 && o.RefFloor != o.Version {
+ e, err = xpath.ParseVersionRefFloor(sel.Source(), ns, o.Version, o.RefFloor)
+ } else {
+ e, err = xpath.ParseVersion(sel.Source(), ns, o.Version)
+ }
+ if err != nil {
+ return nil
+ }
+ return e
+}
+
+// descendantFirst rewrites the record path e when it starts from the
+// document node with //X[p], descendant-or-self::node()/child::X[p], as
+// descendant::X[p], and reports whether it did. The two select the same
+// nodes in the same order, since burstPattern admitted no predicate that
+// depends on position; but package xpath memoises the first form over a
+// parsed document node for the rest of the transform (stepmemo.go), and
+// each burst's document node held there would keep its whole tree: a
+// count(//rec) over 100 MB peaked at 1.8 GB against the tree's 0.8 GB.
+func descendantFirst(e xpath.Expr) bool {
+ switch x := e.(type) {
+ case *xpath.FuncCall:
+ return outerInner(x) && descendantFirst(x.Args[0])
+ case *xpath.PathExpr:
+ if f, ok := firstOf(x.Steps).(*xpath.FuncCall); ok && !x.Root {
+ return descendantFirst(f)
+ }
+ k := 0
+ if _, ok := firstOf(x.Steps).(*xpath.ContextItem); ok && !x.Root {
+ k = 1
+ }
+ if k+1 >= len(x.Steps) {
+ return false
+ }
+ dos, ok := x.Steps[k].(*xpath.Step)
+ if !ok || dos.Axis != xpath.AxisDescendantOrSelf || len(dos.Predicates) > 0 {
+ return false
+ }
+ if kt, ok := dos.Test.(*xpath.KindTest); !ok || !kt.Any {
+ return false
+ }
+ ch, ok := x.Steps[k+1].(*xpath.Step)
+ if !ok || ch.Axis != xpath.AxisChild {
+ return false
+ }
+ if _, ok := ch.Test.(*xpath.NameTest); !ok {
+ return false
+ }
+ d := &xpath.Step{Axis: xpath.AxisDescendant, Test: ch.Test, Predicates: ch.Predicates}
+ x.Steps = append(append(x.Steps[:k:k], d), x.Steps[k+2:]...)
+ return true
+ }
+ return false
+}
+
// noteBurstSelect records instr as the compiled target of the burst plan
// being compiled, when n is its value-of, copy-of or sequence, and compiles
// the parts of its select, sel, written where ns resolves names. The select
@@ -1045,20 +1160,7 @@ func (c *compiler) noteBurstSelect(n *xdm.Node, instr Instruction, sel *xpath.Co
}
}
parse := func() *burstSplit {
- o := exprOptions(ns)
- var (
- e xpath.Expr
- err error
- )
- if o.RefFloor != 0 && o.RefFloor != o.Version {
- e, err = xpath.ParseVersionRefFloor(sel.Source(), ns, o.Version, o.RefFloor)
- } else {
- e, err = xpath.ParseVersion(sel.Source(), ns, o.Version)
- }
- if err != nil {
- return nil
- }
- return splitSelect(e, isXSL(n, "value-of"))
+ return splitSelect(reparse(sel, ns), isXSL(n, "value-of"))
}
part := func(e xpath.Expr) *xpath.Compiled {
return xpathleaf.WithExpr(sel, e).(*xpath.Compiled)
@@ -1068,6 +1170,7 @@ func (c *compiler) noteBurstSelect(n *xdm.Node, instr Instruction, sel *xpath.Co
return
}
s.put(&xpath.VarRef{Name: burstItems})
+ descendantFirst(s.items)
p.items, p.outer, p.atomize = part(s.items), part(*s.outer), s.atomize
if f := parse(); f != nil && f.call != nil {
f.putCall(&xpath.VarRef{Name: burstValue})
@@ -1139,15 +1242,26 @@ func (i *sourceDocumentInstr) executeBursts(rt *runtime, out *outputBuilder, hre
stand := xdm.NewTree()
sub := rt.withCurrent(stand.Root, 1, 1).clearCurrentRule()
plan := i.bursts
- // records calls fn with each burst's document node and element,
- // whitespace stripped as the tree would have been.
- records := func(fn func(root, rec *xdm.Node) error) error {
+ // records calls fn with each burst's document node and element. Small
+ // bursts come several to a tree (xdm's burst batches), and a select
+ // evaluated against the document node reads all of them: with perRoot,
+ // fn is called once for each tree, whitespace stripped as the tree would
+ // have been, and with no element.
+ records := func(perRoot bool, fn func(root, rec *xdm.Node) error) error {
+ var last *xdm.Tree
return br.parseBursts(rt.ctx, href, base, plan.path, func(rec *xdm.Node) error {
root := rec.Tree().Root
+ if !perRoot {
+ return fn(root, rec)
+ }
+ if rec.Tree() == last {
+ return nil
+ }
+ last = rec.Tree()
if strip != nil {
- root, rec = strip.stripWhitespaceForFrom(pkg, root, rec)
+ root = strip.stripWhitespaceFor(pkg, root)
}
- return fn(root, rec)
+ return fn(root, nil)
})
}
if plan.items != nil {
@@ -1175,8 +1289,8 @@ func (i *sourceDocumentInstr) executeBursts(rt *runtime, out *outputBuilder, hre
}
sub.bursts = &burstFeed{instr: plan.instr, read: func(each func(xdm.Item) error) error {
var bodyErr error
- err := records(func(root, rec *xdm.Node) error {
- if !plan.filtered {
+ err := records(plan.filtered || strip != nil, func(root, rec *xdm.Node) error {
+ if rec != nil {
if err := each(rec); err != nil {
bodyErr = err
return err
@@ -1212,7 +1326,7 @@ var errBurstGiveUp = errors.New("bursts: the fold cannot be exact")
// over the bursts records reads. The item and byte budgets are held across
// all of it, as they are across one evaluation of the select over the tree.
func (p *burstPlan) evalBursts(rt *runtime,
- records func(fn func(root, rec *xdm.Node) error) error) (xdm.Sequence, error) {
+ records func(perRoot bool, fn func(root, rec *xdm.Node) error) error) (xdm.Sequence, error) {
ctx := rt.ctx.HoldItemBudget().HoldByteBudget()
fold := p.reducer != "" && !noBurstFold
@@ -1221,7 +1335,7 @@ func (p *burstPlan) evalBursts(rt *runtime,
n int64
typ string
)
- err := records(func(root, _ *xdm.Node) error {
+ err := records(true, func(root, _ *xdm.Node) error {
if err := ctx.Err(); err != nil {
return err
}
@@ -1318,7 +1432,7 @@ func (r *FileResolver) wantsBursts(uri, base string) (string, bool) {
// does -- the same confinement, options, entity allowance and byte limit, and
// the same errors -- but in bursts, and caches nothing: there is no tree.
func (r *FileResolver) parseBursts(ctx *xpath.Context, uri, base string,
- steps []xdm.QName, fn func(*xdm.Node) error) error {
+ steps []xdmburst.Step, fn func(*xdm.Node) error) error {
path, err := r.resolvePath(uri, base)
if err != nil {
@@ -1337,8 +1451,8 @@ func (r *FileResolver) parseBursts(ctx *xpath.Context, uri, base string,
src = io.LimitReader(f, lim)
}
cr := &countedReader{r: src}
- err := xdm.ParseBursts(cr, opts, steps, func(n *xdm.Node) error {
- fnErr = fn(n)
+ err := xdmburst.Parse(cr, opts, steps, func(n any) error {
+ fnErr = fn(n.(*xdm.Node))
return fnErr
})
if err != nil && err == fnErr {
diff --git a/xslt/streamburst_test.go b/xslt/streamburst_test.go
index 4add77b2..0fd5cc66 100644
--- a/xslt/streamburst_test.go
+++ b/xslt/streamburst_test.go
@@ -10,8 +10,11 @@ import (
"sync/atomic"
"testing"
"time"
+ "weak"
+ "github.com/knroy/go-xml/v2/internal/xdmburst"
"github.com/knroy/go-xml/v2/xdm"
+ "github.com/knroy/go-xml/v2/xpath"
)
const burstDoc = `
@@ -368,3 +371,126 @@ func TestBurstSelectAtomizes(t *testing.T) {
t.Fatalf("got %q, %v, bursts %v, heap grew %d MB", out, err, burstRuns, grew)
}
}
+
+// Records nested in records, and in other elements; one in a namespace, and
+// a document in a default namespace.
+const (
+ burstNestDoc = `x` +
+ `t`
+ burstNSDoc = ``
+)
+
+// TestBurstPatternsMatchTree runs record paths with descendant steps,
+// wildcards and outermost() with bursts and without: a match nested in
+// another is delivered inside it, in document order, and outermost() keeps
+// only the outer one.
+func TestBurstPatternsMatchTree(t *testing.T) {
+ for _, tc := range []struct {
+ body, doc, attrs string
+ bursts bool
+ want string
+ }{
+ {``, burstNestDoc, "", true,
+ ``},
+ {``, burstNestDoc, "", true,
+ ``},
+ {``, burstNestDoc, "", true, ""},
+ {``, burstNestDoc, "", true, ``},
+ {``, burstNestDoc, "", true, ``},
+ {``, burstNestDoc, "", true, ``},
+ {``, burstNestDoc, "", true, "22"},
+ {``, burstNestDoc, "", true, "3"},
+ {``, burstNestDoc, "", true, "1,4,7"},
+ {``, burstNestDoc, "", true, "7"},
+ {``, burstNestDoc, "", true, "1"},
+ {``, burstNestDoc, "", true, "4"},
+ {``, burstNestDoc, "", true, "2"},
+ {``, burstNestDoc, "", true, "1 2 3 4 5 7"},
+ {``, burstNSDoc, `xpath-default-namespace="urn:d"`, true, "1 2 3"},
+ {``, burstNSDoc, `xpath-default-namespace="urn:d"`, true, ``},
+ {``, burstNSDoc, "", true, "0"},
+ // refused: no element to cut at, or a position among siblings
+ {``, burstNestDoc, "", false, "2"},
+ {``, burstNestDoc, "", false, "28"},
+ {``, burstNestDoc, "", false, ""},
+ } {
+ src := `` +
+ tc.body + ``
+ got, err := burstRun(t, src, "doc.xml", tc.doc, true)
+ ran := burstRuns
+ want, err2 := burstRun(t, src, "doc.xml", tc.doc, false)
+ if err != nil || err2 != nil || got != want {
+ t.Errorf("%s: bursts differ from the tree\nbursts: %q %v\ntree: %q %v", tc.body, got, err, want, err2)
+ continue
+ }
+ if ran != tc.bursts {
+ t.Errorf("%s: ran in bursts: %v, want %v", tc.body, ran, tc.bursts)
+ }
+ if tc.want != "" && !strings.HasSuffix(got, ">"+tc.want) {
+ t.Errorf("%s: output %q, want %q", tc.body, got, tc.want)
+ }
+ }
+}
+
+// keepCounter is a FileResolver that remembers, weakly, each burst's
+// document node and counts how many are still reachable once the parse is
+// over, while the transform that read them is still running.
+type keepCounter struct {
+ *FileResolver
+ live int
+}
+
+func (k *keepCounter) parseBursts(ctx *xpath.Context, uri, base string, path []xdmburst.Step, fn func(*xdm.Node) error) error {
+ var seen []weak.Pointer[xdm.Node]
+ err := k.FileResolver.parseBursts(ctx, uri, base, path, func(n *xdm.Node) error {
+ seen = append(seen, weak.Make(n.Tree().Root))
+ return fn(n)
+ })
+ goruntime.GC()
+ goruntime.GC()
+ for _, w := range seen {
+ if w.Value() != nil {
+ k.live++
+ }
+ }
+ return err
+}
+
+// Each burst's tree is let go once it has been read: //rec from a burst's
+// document node is not left in package xpath's per-transform step memo,
+// which would keep every burst's tree to the end of the transform.
+func TestBurstsDropTheirTrees(t *testing.T) {
+ dir := t.TempDir()
+ doc := "" + strings.Repeat(`x`, 2000) + ""
+ if err := os.WriteFile(filepath.Join(dir, "doc.xml"), []byte(doc), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ for _, body := range []string{
+ ``,
+ ``,
+ ``,
+ } {
+ src := `` +
+ body + ``
+ tree, err := xdm.ParseString(src, xdm.ParseOptions{BaseURI: fileURIOf(filepath.Join(dir, "s.xsl"))})
+ if err != nil {
+ t.Fatal(err)
+ }
+ sheet, err := Compile(tree.Root, CompileOptions{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ fr, err := NewFileResolver(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ k := &keepCounter{FileResolver: fr}
+ if _, err := sheet.Transform(context.Background(), nil, TransformOptions{InitialTemplate: "main", Documents: k}); err != nil {
+ t.Fatal(err)
+ }
+ if k.live > 10 {
+ t.Errorf("%s: %d of 2000 burst trees still held after the parse", body, k.live)
+ }
+ }
+}
diff --git a/xslt/transform.go b/xslt/transform.go
index 994d45f2..81c1437d 100644
--- a/xslt/transform.go
+++ b/xslt/transform.go
@@ -77,6 +77,45 @@ type TransformOptions struct {
// bound whatever this says.
MaxItems int
+ // MaxNodes bounds the result-tree and temporary-tree nodes the transform
+ // holds: zero is xpath.MaxNodes (2,000,000), negative is no bound, and
+ // past it the transform fails with XPDY0130 wrapping
+ // xdm.ErrResourceLimit.
+ //
+ // What counts as held is every node built, except in a streamed
+ // xsl:for-each or xsl:iterate over an xsl:source-document read in
+ // bursts: there an item that writes nothing to the loop's output hands
+ // back, when it finishes, what it built and dropped -- its variables'
+ // temporary trees, its fn:copy-of copies, the iteration parameters the
+ // next item replaced -- and keeps charged only what it carries on. So a
+ // correct stream whose state stays flat runs over any input, while one
+ // whose carried state grows is refused once that state crosses the bound.
+ // Only the first 4,096 nodes an item drops are handed back unless
+ // MaxNodesBuilt is set; see there.
+ MaxNodes int
+
+ // MaxBytes bounds the string content the transform holds, in bytes, on
+ // MaxNodes's terms: zero is xpath.MaxBytes (1 GiB), negative is no bound,
+ // XPDY0130 past it, and a streamed item hands back up to 65,536 bytes of
+ // what it built and dropped unless MaxBytesBuilt is set.
+ MaxBytes int64
+
+ // MaxNodesBuilt bounds every node the transform builds, whatever it
+ // later drops. Zero leaves the total bounded by the per-item hand-back
+ // MaxNodes describes: each streamed item returns at most 4,096 of the
+ // nodes it dropped, so the transform builds at most MaxNodes plus 4,096
+ // per streamed item, and a stream whose work per item keeps growing is
+ // refused in bounded time even while what it holds is small. A positive
+ // value bounds the total at that instead and lets every item hand back
+ // all it dropped; a negative value removes both, leaving only what is
+ // held bounded -- the choice for trusted input whose items legitimately
+ // do more work than the allowance. XPDY0130 past it.
+ MaxNodesBuilt int64
+
+ // MaxBytesBuilt is MaxNodesBuilt for string content, with a hand-back of
+ // 65,536 bytes per streamed item when it is zero.
+ MaxBytesBuilt int64
+
// DisableAssertions turns off xsl:assert checking for the whole
// transformation. The zero value leaves assertions enabled, which is what
// XSLT 3.0 section 22.2 requires: "By default, assertions are enabled."