Repository navigation
wit-parser: distinguish doc comments from non-doc comments - #1
Closed
cestercian wants to merge 3 commits into
Closed
cestercian wants to merge 3 commits into
cestercian wants to merge 3 commits into
Conversation
Split `Token::Comment` into `Token::Comment` (non-doc) and a new `Token::DocComment`. The tokenizer now classifies comments the same way rustdoc does: * `//...` is a non-doc line comment (`Token::Comment`) * `///...` is a doc line comment (`Token::DocComment`) * `////...` is treated as a non-doc line comment * `/* ... */` is a non-doc block comment * `/** ... */` is a doc block comment * `/**/` and `/*** ... */` are non-doc block comments `parse_docs` now only aggregates `Token::DocComment`s into the `docs` field of parsed items. Non-doc comments are transparent for doc-attachment purposes, matching Rust's behavior where a stray `//` line does not detach a preceding `///` block from the next item. This lets downstream tools such as componentize-py and wit-bindgen drop their preprocessors that strip `//` before feeding source to wit-parser (see bytecodealliance/componentize-py#251). The old `comments.wit` fixture (which exercised both comment kinds without distinguishing them) is expanded into a proper regression test that asserts `//`, `/* */`, `////`, `/**/`, and `/*** */` do not land in `docs` while `///` and `/** */` do. Tokenizer unit tests for the new classification are also added in `lex.rs`. Public API impact is minimal: the `ast` module (which is where `Token` lives) is private to `wit-parser`, so no publicly exposed types change. The observable behavior change is that `Docs::contents` will no longer contain the text of `//` or `/* */` comments; downstream fixtures need to be re-blessed if they relied on that (as done in wit-component and the top-level `tests/cli` fixtures in follow-up commits). Co-authored-by: Cestercian <yashafaid@gmail.com>
The `maps.wit` fixture uses `// ...` (non-doc) comments; after wit-parser stopped treating those as docs the printed .wit and its encoded custom section no longer include them. The `doc-comments/foo.wit` fixture intentionally has a stray `// non-doc in the middle` line between two `///` lines to verify the new behavior — it is now correctly dropped from the resulting docs contents. Co-authored-by: Cestercian <yashafaid@gmail.com>
The top-level CLI test WIT files use a leading `// RUN: ...` directive line which wit-parser previously absorbed into the following item's docs. With the new non-doc/doc split those `//` lines are no longer part of the docs, so the associated .stdout snapshots drop the corresponding `/// RUN: ...` doc lines the printer used to emit. Co-authored-by: Cestercian <yashafaid@gmail.com>
Owner
Author
|
superseded by upstream bytecodealliance#2670 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Today
wit-parser's tokenizer emits a singleToken::Commentfor every//,///,/* */, and/** */comment, andparse_docsaggregatesall of them into the
docsfield of the following item. That means anunrelated
// TODO(...)line above aninterfaceorfuncsilentlybecomes part of that item's public documentation, which is almost never
what the author intended.
Several downstream tools that consume
wit-parser(componentize-py,wit-bindgen, etc.) work around this today with preprocessors that strip
//lines before feeding source towit-parser. Fixing this upstreamlets every consumer drop that preprocessor.
Change
Token::Commentincrates/wit-parser/src/ast/lex.rsintotwo variants:
Token::Comment— non-doc comments (//...,/* ... */)Token::DocComment— doc comments (///...,/** ... */)////-prefixed lines and/**//
/*** ... */blocks are treated as non-doc, so we don'tover-eagerly capture separator-style banners.
Tokenizer::nextskips bothCommentandDocComment, so therest of the parser is unaffected.
parse_docsincrates/wit-parser/src/ast.rsnow only pushesToken::DocCommentspans intoDocs. Non-doc comments aretransparent — they don't contribute to
docs, and they alsodon't detach a preceding
///block from the following item(again matching Rust).
commentsfield was added for non-doc comments. That wascalled out as optional in the tracking discussion, would be new
surface area on every AST node, and no consumer has asked for
it; the churn didn't seem justified. Happy to add it in a
follow-up if that changes.
Tests
crates/wit-parser/tests/ui/comments.wit(plus its.wit.jsonexpectation) into a regression test that asserts//,/* */,////,/**/, and/*** */do not land indocs, while///and/** */do.test_comment_kindsincrates/wit-parser/src/ast/lex.rscovering the raw token classification directly.
removal of stray
//lines fromdocs(wit-component'smaps.wit,doc-comments/foo.wit, and top-leveltests/cli/*.stdoutsnapshots for.witfiles that begin with a// RUN: ...directive).Test commands used:
All of the above pass on this branch.
API breakage
wit_parser::ast(whereTokenlives) is a private module —pub use ast::lex::Span;is the only re-export from there — soadding
Token::DocCommentand narrowingToken::Commentis nota source-visible change for external crates.
Docs::contentsno longer containsthe text of
//or/* */comments. Any tool that relied onthat behavior needs to be updated. All in-tree consumers were
updated in this branch.
Upstream
Compare link for opening this against upstream:
bytecodealliance/wasm-tools@main...Cestercian:wasm-tools:cursor/wit-parser-doc-comment-token-8236