Skip to content

wit-parser: distinguish doc comments from non-doc comments - #1

Closed
cestercian wants to merge 3 commits into
mainfrom
cursor/wit-parser-doc-comment-token-8236
Closed

cestercian wants to merge 3 commits into
mainfrom
cursor/wit-parser-doc-comment-token-8236

Conversation

@cestercian

@cestercian cestercian commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Fork-only staging PR for a change intended to be sent upstream to
bytecodealliance/wasm-tools. See the compare link at the bottom
of this description.

Motivation

Today wit-parser's tokenizer emits a single Token::Comment for every
//, ///, /* */, and /** */ comment, and parse_docs aggregates
all of them into the docs field of the following item. That means an
unrelated // TODO(...) line above an interface or func silently
becomes part of that item's public documentation, which is almost never
what the author intended.

Several downstream tools that consume wit-parser (componentize-py,
wit-bindgen, etc.) work around this today with preprocessors that strip
// lines before feeding source to wit-parser. Fixing this upstream
lets every consumer drop that preprocessor.

Change

  • Split Token::Comment in crates/wit-parser/src/ast/lex.rs into
    two variants:
    • Token::Comment — non-doc comments (//..., /* ... */)
    • Token::DocComment — doc comments (///..., /** ... */)
  • Classification matches rustdoc: ////-prefixed lines and /**/
    / /*** ... */ blocks are treated as non-doc, so we don't
    over-eagerly capture separator-style banners.
  • Tokenizer::next skips both Comment and DocComment, so the
    rest of the parser is unaffected.
  • parse_docs in crates/wit-parser/src/ast.rs now only pushes
    Token::DocComment spans into Docs. Non-doc comments are
    transparent — they don't contribute to docs, and they also
    don't detach a preceding /// block from the following item
    (again matching Rust).
  • No comments field was added for non-doc comments. That was
    called out as optional in the tracking discussion, would be new
    surface area on every AST node, and no consumer has asked for
    it; the churn didn't seem justified. Happy to add it in a
    follow-up if that changes.

Tests

  • Expanded crates/wit-parser/tests/ui/comments.wit (plus its
    .wit.json expectation) into a regression test that asserts
    //, /* */, ////, /**/, and /*** */ do not land in
    docs, while /// and /** */ do.
  • Added test_comment_kinds in crates/wit-parser/src/ast/lex.rs
    covering the raw token classification directly.
  • Re-blessed downstream fixtures whose only diff was the correct
    removal of stray // lines from docs (wit-component's
    maps.wit, doc-comments/foo.wit, and top-level tests/cli/*
    .stdout snapshots for .wit files that begin with a
    // RUN: ... directive).

Test commands used:

cargo test -p wit-parser
cargo test -p wit-component@0.259.0 -p wit-encoder
BLESS=1 cargo test --test cli   # after `git submodule update --init`

All of the above pass on this branch.

API breakage

  • wit_parser::ast (where Token lives) is a private module —
    pub use ast::lex::Span; is the only re-export from there — so
    adding Token::DocComment and narrowing Token::Comment is not
    a source-visible change for external crates.
  • Observable behavior change: Docs::contents no longer contains
    the text of // or /* */ comments. Any tool that relied on
    that behavior needs to be updated. All in-tree consumers were
    updated in this branch.

Upstream

Compare link for opening this against upstream:
bytecodealliance/wasm-tools@main...Cestercian:wasm-tools:cursor/wit-parser-doc-comment-token-8236

Open in Web Open in Cursor 

cestercian and others added 3 commits September 22, 2026 04:04
Split `Token::Comment` into `Token::Comment` (non-doc) and a new
`Token::DocComment`. The tokenizer now classifies comments the same
way rustdoc does:

  * `//...` is a non-doc line comment (`Token::Comment`)
  * `///...` is a doc line comment (`Token::DocComment`)
  * `////...` is treated as a non-doc line comment
  * `/* ... */` is a non-doc block comment
  * `/** ... */` is a doc block comment
  * `/**/` and `/*** ... */` are non-doc block comments

`parse_docs` now only aggregates `Token::DocComment`s into the
`docs` field of parsed items. Non-doc comments are transparent for
doc-attachment purposes, matching Rust's behavior where a stray
`//` line does not detach a preceding `///` block from the next
item.

This lets downstream tools such as componentize-py and wit-bindgen
drop their preprocessors that strip `//` before feeding source to
wit-parser (see bytecodealliance/componentize-py#251).

The old `comments.wit` fixture (which exercised both comment
kinds without distinguishing them) is expanded into a proper
regression test that asserts `//`, `/* */`, `////`, `/**/`,
and `/*** */` do not land in `docs` while `///` and `/** */`
do. Tokenizer unit tests for the new classification are also added
in `lex.rs`.

Public API impact is minimal: the `ast` module (which is where
`Token` lives) is private to `wit-parser`, so no publicly
exposed types change. The observable behavior change is that
`Docs::contents` will no longer contain the text of `//` or
`/* */` comments; downstream fixtures need to be re-blessed if
they relied on that (as done in wit-component and the top-level
`tests/cli` fixtures in follow-up commits).

Co-authored-by: Cestercian <yashafaid@gmail.com>
The `maps.wit` fixture uses `// ...` (non-doc) comments; after
wit-parser stopped treating those as docs the printed .wit and
its encoded custom section no longer include them. The
`doc-comments/foo.wit` fixture intentionally has a stray
`// non-doc in the middle` line between two `///` lines to
verify the new behavior — it is now correctly dropped from the
resulting docs contents.

Co-authored-by: Cestercian <yashafaid@gmail.com>
The top-level CLI test WIT files use a leading
`// RUN: ...` directive line which wit-parser previously
absorbed into the following item's docs. With the new
non-doc/doc split those `//` lines are no longer part of
the docs, so the associated .stdout snapshots drop the
corresponding `/// RUN: ...` doc lines the printer used
to emit.

Co-authored-by: Cestercian <yashafaid@gmail.com>
@cestercian

Copy link
Copy Markdown
Owner Author

superseded by upstream bytecodealliance#2670

@cestercian cestercian closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant