Skip to content

avoid cubic ~ matching in xpath_indirect_adjacent_combinator - #227

Open
itecz26 wants to merge 1 commit into
scrapy:masterfrom
itecz26:indirect-adjacent-first-sibling
Open

itecz26 wants to merge 1 commit into
scrapy:masterfrom
itecz26:indirect-adjacent-first-sibling

Conversation

@itecz26

@itecz26 itecz26 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

a ~ b was translated to a/following-sibling::b, so when many siblings match a their following-sibling sets overlap and libxml2 merges each one in with a linear duplicate scan, which makes matching cubic in the list length (lxml 6.1.3 / libxml2 2.14.6: li ~ li takes 1.5 s on a 19 KiB page of 2000 <li>, 13 s at 4000 and 154 s at 8000, same story for dt ~ dd and li ~ li ~ li, so one fetched page is enough to stall a spider); the siblings after the first match cover those after every later one, so for a plain name test on the left xpath_indirect_adjacent_combinator() now only walks from that one via [position() = 1 or not(preceding-sibling::a[1])] (position() = 1 keeps the first match from each context node, i.e. the context node itself or the first match after each node of a ~ chain), which brings li ~ li down to 2 ms at 8000 siblings and 37 ms at 128000 and selects the same elements as before on ~1.1M random selector/document/context-node checks with every prefix except ancestor::, following:: and preceding:: ones; a left side with a condition (li.a ~ li) is left alone because copying it into the predicate would double the XPath at each nested :has() level.

a ~ b was translated to a/following-sibling::b. When many siblings
match a, their following-sibling node-sets overlap, and libxml2 merges
each one into the result with a linear duplicate scan, so matching was
cubic in the number of siblings: li ~ li took 154 s on a list of 8000
items. The siblings after the first match of a in a sibling list
include those after every later one, so only walk from that one, found
with preceding-sibling::a[1], which stops at the nearest match.
position() = 1 keeps the first match from each context node, which
matters when the context node itself matches and has matching siblings
before it, and in ~ chains.

Only a plain name test is handled: a condition would have to be copied
into the predicate, doubling the XPath at each nested :has() level.
@codecov

codecov Bot commented Nov 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1730a48) to head (abebd6a).

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #227   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines         1036      1038    +2     
  Branches       174       175    +1     
=========================================
+ Hits          1036      1038    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant