Skip to content

test only the next sibling in xpath_direct_adjacent_combinator - #226

Open
itecz26 wants to merge 1 commit into
scrapy:masterfrom
itecz26:adjacent-sibling-first-step
Open

itecz26 wants to merge 1 commit into
scrapy:masterfrom
itecz26:adjacent-sibling-first-step

Conversation

@itecz26

@itecz26 itecz26 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

xpath_direct_adjacent_combinator() emitted a/following-sibling::*[(self::b) and (position() = 1)], and since libxml2 only stops early on a bare [n] predicate, that predicate ran on every following sibling of every a, so one fetched page can stall a spider (li + li took 100 s on a 312 KiB page of 32000 <li>, dt + dd 39 s on 16000 pairs); following-sibling::*[1]/self::b, the shape :has(+ b) already uses, takes 0.57 s and 0.19 s there and selects the same elements as before on ~1.3M random selector/document checks.

xpath_direct_adjacent_combinator() translated "a + b" to
a/following-sibling::*[(self::b) and (position() = 1)]. libxml2 only
stops an axis walk early for a bare [n] predicate, so this one was
evaluated on every following sibling of every match and the + combinator
was quadratic in the number of siblings: li + li took about 100 s on a
list of 32000 items. Emit following-sibling::*[1]/self::b instead, the
shape :has(+ b) already uses, which only looks at the next sibling.
@codecov

codecov Bot commented Nov 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1730a48) to head (eb00969).

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #226   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines         1036      1034    -2     
  Branches       174       174           
=========================================
- Hits          1036      1034    -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants