feat(rust): the core nodes' DOM hooks and @lexical/html's HTML export and import

rust/DESIGN.md

@@ -165,6 +165,17 @@ model of jsdom's"), since the vectors record what jsdom does:
   Unlike jsdom it keeps most values it can't parse, collapsing their
   whitespace, as Swift's does; it rejects a shadow of nothing but
   identifiers, as jsdom does.
+- **The nodes' DOM hooks are `LexicalNode` methods** behind the feature:
+  `create_dom(this, config, editor)`, `export_dom(this, editor)` and
+  `import_dom()` (the class's static `importDOM`, read from a default
+  value when the class registers). Like the other hooks they get the
+  node's value and its handle, so an export can run on a value that isn't
+  in the editor state (the text a selection cuts short). The base classes'
+  behavior is the hooks' defaults (`TextNode` renders its text,
+  `ElementNode` adds the indent and direction), as for
+  `get_text_content`. The theme is the JSON of `EditorThemeClasses`
+  (`CreateEditorArgs.theme`), and a `DOMConversionMap` is a list of
+  name and importer pairs in the order JS's object keys would be.
 - **`DOMParser` is a WHATWG tokenizer and tree builder**, checked against
   jsdom's by the `html/parse` and `html/entities` vectors. As in Swift it
   leaves out templates, framesets, the `<select>` and `<ruby>` insertion
@@ -215,7 +226,7 @@ Each milestone ends with green CI and a report to the Space.
 | P | Desktop playgrounds (`apps/`): `lexical-playground-rune` on Rune, then `lexical-playground-gpui` on GPUI, sharing `lexical-playground-core`; they grow with each milestone below | the vectors, replayed in the app; a session exported from the app replays in JS (`replay.test.ts`); in CI, a GPUI session typed in a real window replays in JS | both apps run on macOS with typing, Return, deletion, formatting, input methods and the Parity panel; they grow with each milestone |
 | M2 | `rich-text` (headings, quotes, and the command handlers a text surface dispatches: typing, Enter, Backspace, formatting) and `history` (undo), which the playgrounds need first; then the node packages `list`, `link`, `code`, `overflow`, `hashtag`, with `selection` and `utils` | `serialization/*`, `property/documents` (JSON and text), `entities/hashtag-*`, `scripts/history` | done (every vector it names passes): `history` is done (`scripts/history` passes); `rich-text`'s nodes and `registerRichText`'s model command handlers are done (`scripts/rich-text` and `scripts/rich-text-commands` pass, with the heading and quote cases of `property/documents`); `plain-text`'s model command handlers are done (`scripts/plain-text` passes); `list`'s nodes, `registerList` and the model part of `registerCheckList` are done (`scripts/list` and `serialization/list` pass, with the list cases of `property/documents`); `link`'s nodes, `$toggleLink` and `registerLink`'s model handlers are done (`scripts/link` and `serialization/link` pass, with the link cases of `property/documents` that need no other package), `registerAutoLink` and its matchers are done behind `autolink` (`scripts/autolink` and `entities/autolink-match` pass); `selection`'s `$setBlocksType` is done (`scripts/set-blocks-type` passes); `code`'s nodes, `FlatStructureUtils.ts`, `CodeExtension`'s transform and `registerCodeIndentation`'s model handlers are done (`scripts/code` and `serialization/code` pass, with the code cases of `property/documents` that need no other package); `overflow`'s `OverflowNode` is done (`scripts/overflow` and `serialization/overflow` pass, with the overflow cases of `property/documents` that need no other package); `hashtag`'s `HashtagNode`, its matcher (generated character tables, no `regex`) and `registerLexicalHashtag`, with `text`'s `registerLexicalTextEntity`, are done (`entities/hashtag-match`, `entities/hashtag-documents` and `entities/hashtag-typing` pass, with the last waiting cases of `property/documents`), so M2's node packages are done |
 | M3 | The commands `property/sequences` dispatches | `property/sequences` | |
-| M4 | `html`: a DOM abstraction over an HTML parser, CSS declarations, export and import | `html/*`, `css/*`, `property/documents` (HTML) | in progress: the DOM (`dom`: nodes, `DOMParser`, the serializer, the inline style declaration and its tables) and `getStyleObjectFromCSS` are done (`html/parse`, `html/entities`, `css/properties` and `css/values` pass); next, the core nodes' DOM hooks and `@lexical/html`'s export and import |
+| M4 | `html`: a DOM abstraction over an HTML parser, CSS declarations, export and import | `html/*`, `css/*`, `property/documents` (HTML) | in progress: the DOM (`dom`: nodes, `DOMParser`, the serializer, the inline style declaration and its tables), `getStyleObjectFromCSS`, the core nodes' DOM hooks and `@lexical/html`'s export and import are done (every `html/*` and `css/*` file passes); next, the node packages' DOM parts (list, link, code, overflow, hashtag, rich text, the decorators) and `property/documents`' HTML |
 | M5 | `markdown` | `markdown/import-export`, `property/documents` (Markdown) | |
 | M6 | `yjs`: a port of `@lexical/yjs` on `yrs` (behind the feature), with new JS vectors for the binding (sevki/lexical-rs's `lexical-yjs` and jpcamara/yrby's `lexical-yjs-html` only as cross-checks) | the new collab vectors | |
 | M7 | Releases: the mirror, changesets, a name tmm settles | | |

rust/PARITY.md

@@ -47,6 +47,8 @@ updated with each milestone.
 | `registerLexicalTextEntity` (the `text` feature) | `packages/lexical-text/src/registerLexicalTextEntity.ts`: `EntityMatch`, `registerLexicalTextEntity` (the `TextNode` transform and `$reverseNodeTransform`, `$replaceWithSimpleText`, `$wasTurnedIntoText`, `getFirstMatchAfter`, `isNextTextEntity`, `isTextAfterOutsideSource`, `isEntityAfter`, `$checkNextSibling`, `$moveSelectionIntoEntity`, `$wouldMerge`, `MAX_MATCHES_PASSED`, `MAX_TURNED_INTO_TEXT`, the per-update `markedInUpdate` and `turnedIntoTextInUpdate`); from the core `INTERNAL_getActiveEditorState` | `text`, `updates` | `entities/hashtag-documents`, `entities/hashtag-typing` |
 | `$setBlocksType` (the `selection` feature) | `packages/lexical-selection/src/range-selection.ts` (`$setBlocksType`, with its default `$afterCreateElement`, `$copyBlockFormatIndent`, and `$isPointAtBlockEdge`), through `LexicalNode.replace` and `ListItemNode.replace` | `selection` | `scripts/set-blocks-type`, the block-decorator case of `scripts/decorator` |
 | The DOM (the `html` feature) | what a browser or jsdom gives JS: `Node` (`nodeType`, `nodeName`, `parentNode`, `childNodes`, siblings, `textContent`, `insertBefore`, `appendChild`, `append`, `prepend`, `removeChild`, `replaceChild`, `remove`, `replaceWith`, `replaceChildren`, `contains`, `cloneNode`), `Element` (attributes in order, `id`, `dir`, `className`, `classList`, `style`, `innerHTML`, `outerHTML`, `getElementsByTagName`), `Text`, `Comment`, `DocumentFragment`, `Document` (`createElement` and the other factories, `head`, `body`), `DOMTokenList`, `CSSStyleDeclaration` (`setProperty`, `getPropertyValue`, `getPropertyPriority`, `removeProperty`, `cssText`, `item`) with jsdom's normalization, `DOMParser.parseFromString`, the HTML fragment serialization algorithm; ported from the Swift port's `DOM/` | `dom` | `html/parse`, `html/entities`, `css/properties`, `css/values` |
+| Core DOM hooks (the `html` feature) | `LexicalNode.createDOM`, `exportDOM` and the static `importDOM` as node hooks (`create_dom`, `export_dom`, `import_dom`), `ElementNode.exportDOM` (indent as `padding-inline-start` and `data-lexical-indent`, `dir`); `DOMExportOutput`, `DOMConversionOutput`, `DOMChildConversion`, `DOMConversion`, `DOMConversionFn`, `DOMConversionMap`; `createEditor`'s `theme` and `html` (`export` overrides, `import` conversions), `editor._config`, `initializeConversionCache`; `DEFAULT_EDITOR_DOM_CONFIG` (`$createDOM`, `$exportDOM`, `$extractWithChild`, `$shouldExclude`, `$shouldInclude`); `LexicalNode.isSelected`; `TextNode` (`createDOM` with `getElementOuterTag`, `getElementInnerTag`, `$createTextInnerDOM` and `setTextThemeClassNames`; `exportDOM`; `importDOM` with `$convertTextDOMNode`, `findParentPreDOMNode`, `findTextInLine`, `convertSpanElement`, `convertBringAttentionToElement`, `convertTextFormatElement`, `applyTextFormatFromStyle`), `TabNode.createDOM`, `LineBreakNode` (`createDOM`, `importDOM`), `ParagraphNode` (`createDOM`, `exportDOM`, `importDOM` with `$convertParagraphElement`, `extractWithChild`), `ArtificialNode__DO_NOT_USE`; from `LexicalUtils.ts` `isInlineDomNode`, `isBlockDomNode`, `getCachedClassNameArray`, `removeEmptyDOMAttribute`, `setNodeIndentFromDOM`, `$setDirectionFromDOM`, `$setFormatFromDOM`, `isOnlyChildInBlockNode`, `isLastChildInBlockNode`; `utils/classNames.ts` | `lexical_dom`, `node`, `nodes::*`, `editor` | `html/export`, `html/import` |
+| `@lexical/html` (the `html` feature) | `packages/lexical-html/src/index.ts`: `$generateHtmlFromNodes`, `$generateDOMFromNodes`, `$generateDOMFromRoot`, `$appendNodeToHTML` (`$appendNodesToHTML` with `$appendTerminatingLineBreak`, and `@lexical/selection`'s `$sliceSelectedTextNodeContent` for a selection), `$generateNodesFromDOM` (`IGNORE_TAGS`, `getConversionFunction`, `$createNodesFromDOM`, `wrapContinuousInlines`, `$unwrapArtificialNodes`, `isDomNodeBetweenTwoInlineNodes`) | `html` | `html/export`, `html/import` |
 | Inline CSS text | `utils/setDOMStyle.ts` (`getStyleObjectFromCSS`, with JS's property order; with the `html` feature `setDOMStyleObject` and `setDOMStyleFromCSS`) | `set_dom_style` | through `css/values`; unit tests against JS's output |
 | UTF-16 offsets | `String#length`, `String#slice` | `utf16` | through the text methods; unit tests against Node |
 | Upstream version | `meta.json` | `UPSTREAM_VERSION` | `meta` |
@@ -133,9 +135,9 @@ touches:
   DOM. What `NodeStructureExtension` does for links where it enforces node
   structure (lifting blocks, merging equal adjacent links, which
   `$linkNodeTransform` then leaves to it) isn't ported, so the transform
-  always does both. `extractWithChild` is a node hook that nothing calls
-  yet: its caller is `@lexical/html`'s copy. `HeadingNode`'s override is
-  ported; `ParagraphNode`'s, which reads the selection, comes with `html`. In a headless editor,
+  always does both. `extractWithChild` is a node hook that `@lexical/html`'s
+  export calls (the `html` feature). `HeadingNode`'s override is
+  ported, and so is `ParagraphNode`'s, which reads the selection. In a headless editor,
   `LinkNode.canInsertTextBefore` and `canInsertTextAfter` change nothing a
   vector can see: `insertText` at the edge of an inline parent's text puts
   the text outside the parent either way, and
@@ -234,6 +236,16 @@ touches:
   its blocks) is ported but no vector reaches it: headless JS needs the
   DOM's `Selection.modify` to get there and throws, which `scripts/rich-text`
   records.
+- Of `@lexical/html` and the core's DOM side (the `html` feature): the
+  `DOMImportExtension` and `DOMRenderExtension` pipelines (import rules,
+  `CoreImportRules`, selectors, the render context and render config
+  overrides), `BlockIdHTMLExtension`, and `$inlineStylesFromStyleSheets`,
+  which needs a CSS parser and a selector engine (Swift's leans on its
+  import-rule selectors); `$generateNodesFromDOM` leaves `<style>` rules
+  alone, where JS would copy their declarations onto the elements they
+  match. No vector has a `<style>`. Also `updateDOM`, `getDOMSlot` and the
+  rest of what only the reconciler calls, and `$setTextContent`'s
+  composition suffix (nothing composes headless).
 - Composition, the DOM reconciler, mutation listeners, selection-change
   notifications and `$onUpdate` callbacks (none of which a headless editor
   runs, except `$onUpdate`), and `RootNode.__cachedText`. So `history` passes

rust/conformance/src/scripts.rs

@@ -1082,7 +1082,10 @@ impl Session {
         if with_extension {
             nodes.extend(lexical::extension::extension_nodes());
         }
-        let editor = create_editor(CreateEditorArgs { nodes });
+        let editor = create_editor(CreateEditorArgs {
+            nodes,
+            ..Default::default()
+        });
         let ctx = Rc::new(Ctx::default());
         if rich_text {
             let _unregister = lexical::rich_text::register_rich_text(&editor);

rust/conformance/src/suites/entities.rs

@@ -61,6 +61,7 @@ pub fn check_hashtag_match(case: &Value, _file: &Value) -> Result<Outcome, Strin
 fn build() -> Editor {
     let editor = create_editor(CreateEditorArgs {
         nodes: hashtag_nodes(),
+        ..Default::default()
     });
     let _unregister = register_lexical_hashtag(&editor, HashtagConfig::default());
     editor

rust/conformance/src/suites/html.rs

@@ -1,6 +1,10 @@
 use crate::case::{Case, Outcome, same};
 use lexical::dom::{DomNode, is_legacy_entity_name, named_entity, parse_from_string};
+use lexical::html::{generate_html_from_nodes, generate_nodes_from_dom};
 use lexical::json::Value;
+use lexical::{CreateEditorArgs, NodeRef, UpdateOptions, create_editor};
+use std::cell::RefCell;
+use std::rc::Rc;
 
 /// `html/parse`: `DOMParser` builds the same document as jsdom's, compared
 /// by the `innerHTML` of its head and body.
@@ -46,3 +50,66 @@ pub fn check_entity(case: &Value, file: &Value) -> Result<Outcome, String> {
     same("textContent", body.text_content(), expected)?;
     Ok(Outcome::Pass)
 }
+
+/// `html/export`: `$generateHtmlFromNodes` writes the HTML jsdom gave for
+/// the document, with the case's theme.
+pub fn check_export(case: &Value, _file: &Value) -> Result<Outcome, String> {
+    let theme = match case.get("theme") {
+        Some(Value::Object(theme)) => theme.clone(),
+        _ => lexical::json::Object::new(),
+    };
+    let editor = create_editor(CreateEditorArgs {
+        theme,
+        ..Default::default()
+    });
+    let state = editor
+        .parse_editor_state(case.str("editorState")?)
+        .map_err(|e| format!("editorState: {e}"))?;
+    editor
+        .set_editor_state(&state, None)
+        .map_err(|e| e.to_string())?;
+    let html = editor
+        .read(|| generate_html_from_nodes(&editor, None))
+        .and_then(|html| html)
+        .map_err(|e| e.to_string())?;
+    same("html", html.as_str(), case.str("html")?)?;
+    Ok(Outcome::Pass)
+}
+
+/// A node and its subtree as JSON, as `exportJSON` writes each node
+/// (`$nodeToJSON`).
+fn node_to_json(node: &NodeRef) -> Value {
+    let mut json = node.export_json(false);
+    if node.is_element() {
+        json.insert(
+            "children",
+            Value::Array(node.get_children().iter().map(node_to_json).collect()),
+        );
+    }
+    Value::Object(json)
+}
+
+/// `html/import`: `$generateNodesFromDOM` makes the nodes jsdom's import
+/// made of the parsed HTML.
+pub fn check_import(case: &Value, _file: &Value) -> Result<Outcome, String> {
+    let editor = create_editor(CreateEditorArgs::default());
+    let dom = parse_from_string(case.str("html")?);
+    let nodes = Rc::new(RefCell::new(String::new()));
+    let (out, update_editor) = (nodes.clone(), editor.clone());
+    editor
+        .update(
+            move || {
+                let nodes = generate_nodes_from_dom(&update_editor, &dom)?;
+                *out.borrow_mut() =
+                    Value::Array(nodes.iter().map(node_to_json).collect()).to_string();
+                Ok(())
+            },
+            UpdateOptions {
+                discrete: true,
+                ..UpdateOptions::default()
+            },
+        )
+        .map_err(|e| e.to_string())?;
+    same("nodes", nodes.borrow().as_str(), case.str("nodes")?)?;
+    Ok(Outcome::Pass)
+}

rust/conformance/src/suites/mod.rs

@@ -32,6 +32,16 @@ pub fn all() -> Vec<Suite> {
             variant: None,
             check: html::check_entity,
         },
+        Suite {
+            file: "html/export",
+            variant: None,
+            check: html::check_export,
+        },
+        Suite {
+            file: "html/import",
+            variant: None,
+            check: html::check_import,
+        },
         Suite {
             file: "html/parse",
             variant: None,

rust/conformance/src/suites/property.rs

@@ -38,6 +38,7 @@ pub fn check_document(case: &Value, _file: &Value) -> Result<Outcome, String> {
             lexical::hashtag::hashtag_nodes(),
         ]
         .concat(),
+        ..Default::default()
     });
     let parsed = editor.parse_editor_state(input);
     if case.get("error").is_some() {

rust/conformance/src/suites/serialization.rs

@@ -17,6 +17,7 @@ pub fn check(case: &Value, _file: &Value) -> Result<Outcome, String> {
             lexical::hashtag::hashtag_nodes(),
         ]
         .concat(),
+        ..Default::default()
     });
     for (form, input) in [("legacy", legacy), ("compact", compact)] {
         let state = editor

rust/lexical-editor/src/code/flat_structure_utils.rs

@@ -302,6 +302,7 @@ mod tests {
         reset_random_key();
         let editor = create_editor(CreateEditorArgs {
             nodes: code_nodes(),
+            ..Default::default()
         });
         editor
             .update(