Repository navigation
HTMLParser differences from the HTML5 specification #135661
Description
Bug report
Originally, the definition of the HTML format was not formally strict. It was similar to SGML and XML, but with a lot of looseness. HTMLParser tried its best to parse anything that looked like HTML. But after creation of HTML5, its specification defines the parsing rules for HTML documents, whether they are syntactically correct or not. It is important to follow these rules, for security reasons.
The current HTMLParser mainly follows the HTML5 specification, but there are a number of differences:
- ✅
--!>should end the comment. gh-102555: Fix comment parsing in HTMLParser #135664 - ✅
-- >should not end the comment. gh-102555: Fix comment parsing in HTMLParser #135664 - ✅
<-->and<--->should be abnormally ended empty comments. gh-102555: Fix comment parsing in HTMLParser #135664 - ✅
] ]>and]] >should not end the CDATA section. gh-135661: Fix CDATA section parsing in HTMLParser #135665 - 🤕 CDATA handling should depend on the current node. This is important, because the ending condition are different for the CDATA section and the bogus comment (
]]>and>). - ✅ Whitespaces should not be acceptable between
</and the tag name. E.g.</ script>should not end the script section. gh-135661: Fix parsing start and end tags in HTMLParser #135930 - ✅ Vertical tabulation (
\v) and non-ASCII whitespaces should not be recognized as whitespaces. The only whitespaces are\t\n\r\f. gh-135661: Fix parsing start and end tags in HTMLParser #135930 - ✅ Null character (U+0000) should not end the tag name. gh-135661: Fix parsing start and end tags in HTMLParser #135930
- Null character (U+0000), surrogate characters and many other special characters should be replaced by
\xfffd. I think we can leave this, because it is easy to do in pre-processing or post-processing, and they usually do not cause issues in Python. - ✅ End tag can have attributes and slashes after tag name. It can not end after the first
>. E.g.</script/foo=">"/>. gh-135661: Fix parsing start and end tags in HTMLParser #135930 - ✅ Case-insensitive matching should only transform ASCII letters. E.g.
</script>does not match</ſcript>, andLINKdoes not matchLINK(the last letter is U+212A). gh-135661: Fix parsing start and end tags in HTMLParser #135930 - ✅ There may be multiple slashes and whitespaces between the last attribute and closing
>in both start and end tags. E.g.<a foo=bar/ //>. gh-135661: Fix parsing start and end tags in HTMLParser #135930 - ✅ There should only be one
=separator between attribute name and value. E.g.<a foo==bar>should have attribute "foo" with value "=bar". gh-135661: Fix parsing start and end tags in HTMLParser #135930 No whitespace should be acceptable between the=separator and attribute name and value. E.g.<a foo =bar>should have two attributes "foo" and "=bar", both with value None;<a foo= bar>should have two attributes: "foo" with value "" and "bar" with value None.
This can cause security issues for some programs. If the program uses HTMLParser to check the HTML input for dangerous code, it can miss some code. For example, "<!----!><script>...</script><!---->" is parsed by browsers as a script block surrounded by two comments, but the current HTMLParser parses it as a single comment.
Linked PRs
- gh-102555: Fix comment parsing in HTMLParser #135664
- gh-135661: Fix CDATA section parsing in HTMLParser #135665
- gh-135661: Fix parsing start and end tags in HTMLParser #135930
- [3.14] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) #136255
- [3.13] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) #136256
- [3.12] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) #136268
- [3.11] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) (GH-136268) #136291
- [3.10] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) (GH-136268) #136292
- [3.9] gh-135661: Fix parsing start and end tags in HTMLParser according to the HTML5 standard (GH-135930) (GH-136268) #136293
- gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser #136908
- [3.13] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136918
- [3.12] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136919
- [3.11] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136920
- [3.10] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136921
- [3.9] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136922
- [3.14] gh-135661: Fix parsing attributes with whitespaces around the "=" separator in HTMLParser (GH-136908) #136927
- [3.14] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) #137772
- [3.13] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) #137773
- [3.12] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) #137774
- gh-135661: Fix parsing unterminated bogus comments in HTMLParser #137873
- [3.13] gh-135661: Fix parsing unterminated bogus comments in HTMLParser (GH-137873) #137875
- [3.11] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (GH-137774) #139659
- [3.10] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (GH-137774) #139660
- [3.9] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (GH-137774) #139661
- gh-135661: Fix abrupt closing of empty comment in HTMLParser #153007
- [3.15] gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007) #153024
- [3.14] gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007) #153025
- [3.13] gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007) #153026
Activity
pythongh-135661: Fix parsing start and end tags in HTMLParser accordi…
[3.14] gh-135661: Fix parsing start and end tags in HTMLParser accord…
[3.13] gh-135661: Fix parsing start and end tags in HTMLParser accord…
[3.12] pythongh-135661: Fix parsing start and end tags in HTMLParser …
[3.12] pythongh-135661: Fix parsing start and end tags in HTMLParser …
[3.12] gh-135661: Fix parsing start and end tags in HTMLParser accord…
[3.12] pythongh-135661: Fix parsing start and end tags in HTMLParser …
[3.12] pythongh-135661: Fix parsing start and end tags in HTMLParser …
[3.12] pythongh-135661: Fix parsing start and end tags in HTMLParser …
pythongh-135661: Fix parsing start and end tags in HTMLParser accordi…
[3.11] gh-135661: Fix parsing start and end tags in HTMLParser accord…
[3.10] gh-135661: Fix parsing start and end tags in HTMLParser accord…
[3.9] gh-135661: Fix parsing start and end tags in HTMLParser accordi…
pythongh-135661: Fix parsing start and end tags in HTMLParser accordi…
pythongh-135661: Fix parsing start and end tags in HTMLParser accordi…
cjmayo commentedon Jul 19, 2025
"Unquoted attribute value syntax", "Single-quoted attribute value syntax" and "Single-quoted attribute value syntax" [1] all currently (19 July 2025) say:
The attribute name, followed by zero or more ASCII whitespace, followed by a single U+003D EQUALS SIGN character, followed by zero or more ASCII whitespace
[1] https://html.spec.whatwg.org/multipage/syntax.html#attributes-2
pythongh-135661: Fix parsing attributes with whitespaces around the "…
serhiy-storchaka commentedon Jul 21, 2025
Thank you for noticing this @cjmayo. You are right! And the parser specification agrees -- whitespaces are ignored in "after attribute name state" and "before attribute value state".
https://html.spec.whatwg.org/multipage/parsing.html#attribute-name-state
https://html.spec.whatwg.org/multipage/parsing.html#after-attribute-name-state
https://html.spec.whatwg.org/multipage/parsing.html#before-attribute-value-state
Fortunately there were no releases yet, so we can fix this in #136908.
gh-135661: Fix parsing attributes with whitespaces around the "=" sep…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
[3.14] pythongh-135661: Fix parsing attributes with whitespaces aroun…
[3.14] gh-135661: Fix parsing attributes with whitespaces around the …
fix(parser): Adapt to html.parser changes in Python 3.14
hugovk commentedon Jul 22, 2025
3.14 RC1 note: this has been fixed and backported to 3.14 (thank you!), let's keep the
release-blockerfor the older branches.serhiy-storchaka commentedon Jul 22, 2025
There are also #135310 and #135665 which all are parts of the same issue.
[3.13] gh-135661: Fix parsing attributes with whitespaces around the …
[3.12] gh-135661: Fix parsing attributes with whitespaces around the …
[3.11] gh-135661: Fix parsing attributes with whitespaces around the …
[3.10] gh-135661: Fix parsing attributes with whitespaces around the …
[3.9] gh-135661: Fix parsing attributes with whitespaces around the "…
hugovk commentedon Jul 22, 2025
One of these has been merged, the other needs more work and will also have security backports, so doesn't need to hold up the RC.
[3.9] pythongh-135661: Fix parsing start and end tags in HTMLParser a…
[3.9] pythongh-135661: Fix parsing attributes with whitespaces around…
pythongh-135661: Fix parsing start and end tags in HTMLParser accordi…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
Yhg1s commentedon Aug 14, 2025
Does this need more work in 3.13? (Releasing another version today...)
serhiy-storchaka commentedon Aug 14, 2025
It would be nice to have all changes in one version.
Yhg1s commentedon Aug 14, 2025
Do I need to wait with 3.13 then? When can the remaining changes land, do you think?
serhiy-storchaka commentedon Aug 14, 2025
When the PR is approved. This is not a complete solution to the problem, but it is the only one we can have realistically. The discussion was about whether to make a new method public or private.
Yhg1s commentedon Aug 14, 2025
Have the other PRs all been backported, at least the ones that should be? (The long list makes it hard to tell.)
serhiy-storchaka commentedon Aug 14, 2025
Yes, all other PRs were successfully backported.
gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665)
pythongh-135661: Fix CDATA section parsing in HTMLParser (pythonGH-13…
pythongh-135661: Fix CDATA section parsing in HTMLParser (pythonGH-13…
[3.12] pythongh-135661: Fix CDATA section parsing in HTMLParser (pyth…
[3.13] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (…
[3.12] pythongh-135661: Fix CDATA section parsing in HTMLParser (pyth…
pythongh-135661: Fix parsing unterminated bogus in HTMLParser
gh-135661: Fix parsing unterminated bogus comments in HTMLParser (GH-…
pythongh-135661: Fix parsing unterminated bogus comments in HTMLParser (
pythongh-135661: Fix CDATA section parsing in HTMLParser (pythonGH-13…
[3.13] gh-135661: Fix parsing unterminated bogus comments in HTMLPars…
pythongh-135661: Fix parsing start and end tags in HTMLParser accordi…
pythongh-135661: Fix parsing attributes with whitespaces around the "…
pythongh-135661: Fix CDATA section parsing in HTMLParser (pythonGH-13…
pythongh-135661: Fix parsing unterminated bogus comments in HTMLParser (
[3.14] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (…
[3.14] pythongh-135661: Fix parsing attributes with whitespaces aroun…
[3.14] pythongh-135661: Fix CDATA section parsing in HTMLParser (pyth…
[3.12] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (…
[3.12] pythongh-135661: Fix CDATA section parsing in HTMLParser (pyth…
[3.12] pythongh-135661: Fix CDATA section parsing in HTMLParser (pyth…
[3.12] pythongh-135661: Fix CDATA section parsing in HTMLParser (pyth…
[3.11] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (…
[3.10] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (…
[3.9] gh-135661: Fix CDATA section parsing in HTMLParser (GH-135665) (G…
encukou commentedon Oct 14, 2025
All that remains is exposing the context-dependent CDATA parsing in some better way, but that's not a release blocker.
gh-135661: Fix abrupt closing of empty comment in HTMLParser (GH-153007)
[3.13] gh-135661: Fix abrupt closing of empty comment in HTMLParser (G…
[3.14] gh-135661: Fix abrupt closing of empty comment in HTMLParser (G…
[3.15] gh-135661: Fix abrupt closing of empty comment in HTMLParser (G…
serhiy-storchaka commentedon Jul 4, 2026
#153028 provide complete fix of the CDATA issue on the stdlib side. I wonder if it can be backported.