Skip to content

Latest commit

 

History

History
162 lines (127 loc) · 6.62 KB

README.md

File metadata and controls

162 lines (127 loc) · 6.62 KB

htmlparse

An HTML parser for ris: the WHATWG tokenizer and tree construction, ported rule for rule from html5ever 0.35 (as Campfire's Rust port vendors it), and its serializer. It builds the same tree as html5ever for all 1792 of html5lib-tests' tree construction cases and the same tokens for all 7037 tokenizer cases.

import htmlparse::{htmlparse}

html = htmlparse(body) # a String: the HTML, parsed and written back
puts html

htmlparse parses its input as a fragment in body (document: true for a whole document) and answers the HTML that html5ever's serializer writes for it, on the tape, at the same speed as the calls it makes.

A longer example, which parses a fragment with parse_fragment, walks its tree node by node and writes it back, is in examples/tree.ri (ris run examples/tree.ri).

parse_document reads a whole document; Document#root is then the document node. Both yield the Document to a block: the parse's state lives in the call's frame, so the tree is good until the block ends.

Memory

A parse keeps its state in the call's frame and opens no tape of its own. The tree, its names, text and attribute values are tape values: views of the input wherever they are its bytes unchanged, copied only where they differ (a name lowercased, a character reference read, a NUL replaced). The input must outlive the tree. Nodes are one Array, linked by index, so foster parenting and the adoption agency move a node in a few writes. Text added next to text is kept in pieces and joined once, when read.

Options

Options.new has html5ever's defaults: scripting on, a <template>'s content in a fragment of its own (Dom#contents), a byte order mark dropped at the start and after each </script>.

Options.gumbo is what Nokogiri's HTML5 parser does: no scripting, template content among the children, a byte order mark left (Gumbo drops only a leading one; the caller does), and Gumbo's limits of 400 elements deep and 400 attributes a tag, which end the parse with Document#error set.

SerializeOptions writes html5ever's serialization; Nokogiri's is the same with raw_noscript always on. escape_attribute_brackets writes < and > in attribute values as the standard has since 2025.

Limits

An array on the tape stops at 128 MiB, ris's reservation for it, and going past that ends the program; it is not reported in Document#error. That is about two million nodes: some 18 MB of ordinary HTML, far less for input that reconstructs many formatting elements, whose tree grows with the square of its size.

The depth of a tree is not limited by the stack: parsing, serializing and Dom#text_content do not recurse.

Departures from html5ever

None that html5lib-tests' inputs show in the trees, tokens or serialization, save one the standard asks for: <svg> and <math> reconstruct the active formatting elements, which html5ever 0.35 skips. Parse errors are not reported. The tree sink's choices are fixed to the ones the Rust port and the oracle make: no form association, no declarative shadow roots, no MathML annotation-xml integration points.

Speed

python3 bench/run.py times htmlparse against html5ever 0.35 from crates.io (into markup5ever_rcdom, LTO, one codegen unit). Each run parses an input n times and serializes every tree, timing the two apart; the two parsers' outputs must be the same byte for byte. The inputs are two chat messages, parsed as fragments in body, and generated pages of 1, 20, 400 and 2000 articles, parsed as documents: text with entities, attributes, tables, lists, SVG, comments, a script and style, unclosed and misnested tags. One warmup and five rounds in a shuffled order; medians.

Local result, 2026-11-04, AMD Ryzen 7 PRO 6850U, Linux 7.0, rustc 1.99.0, ris 0.1.0:

Parse:

Input htmlparse html5ever html5ever vs htmlparse
message, 118 B 2.5 µs 2.8 µs 14% slower
rich message, 1 KiB 28.2 µs 35.8 µs 27% slower
page, 3 KiB 61.8 µs 90.5 µs 46% slower
page, 45 KiB 834.4 µs 1,285.1 µs 54% slower
page, 904 KiB 17,577.2 µs 24,667.5 µs 40% slower
page, 4.5 MiB 97,559.4 µs 127,613.0 µs 31% slower

Serialize:

Input htmlparse html5ever html5ever vs htmlparse
message, 118 B 0.5 µs 0.9 µs 72% slower
rich message, 1 KiB 6.3 µs 10.7 µs 68% slower
page, 3 KiB 11.1 µs 16.3 µs 48% slower
page, 45 KiB 153.4 µs 262.3 µs 71% slower
page, 904 KiB 3,038.4 µs 6,351.0 µs 2.1× slower
page, 4.5 MiB 17,441.5 µs 37,987.0 µs 2.2× slower

Parsing, html5ever is 14 to 54% slower than htmlparse; serializing, 48% slower to 2.2 times as slow.

The memory one parse takes, without serializing: the peak RSS of a process that parses the input once, less that of one that only reads it. That is whole pages, so it is given for the inputs of 44 KiB and up.

Memory per parse:

Input htmlparse html5ever html5ever vs htmlparse
page, 45 KiB 0.6 MiB 0.6 MiB 16% more
page, 904 KiB 6.6 MiB 11.8 MiB 78% more
page, 4.5 MiB 33.5 MiB 58.2 MiB 74% more

htmlparse's tree takes about seven and a half bytes for each byte of input on the larger pages, and html5ever's about three quarters as much again. The tape keeps all of it until the parse's block ends, and gives its pages back when it does: a loop of parses faults them in again each time, where html5ever's allocator reuses its own.

On threads, bench/threads.ri and bench/rust/src/bin/threads.rs: 1 thread and then 8 (--threads) at once, each parsing and serializing the input as many times as a single run does, htmlparse(input) on its own tape; parses per second over all of them, from the first start to the last join. The CPU has 8 cores, 16 hardware threads.

Parses and serializations per second on 8 threads:

Input htmlparse html5ever html5ever vs htmlparse
message, 118 B 1,807,818 1,569,328 15% slower
rich message, 1 KiB 159,173 120,656 32% slower
page, 45 KiB 5,728 3,623 58% slower
page, 904 KiB 271 165 65% slower

From 1 thread to 8, how many times as many per second:

Input htmlparse html5ever
message, 118 B 6.3× 6.4×
rich message, 1 KiB 6.2× 6.5×
page, 45 KiB 6.3× 6.4×
page, 904 KiB 6.2× 5.8×

Each thread's tape is its own, so the parses share nothing and need no lock: htmlparse goes from 1 thread to 8 about as html5ever does, six times, and keeps its lead. These are one machine's numbers, not a promise; bench/results.json has the raw samples.