Use OFT tag importer

README.md

@@ -48,10 +48,11 @@ tokens.
 
 ### OFT coverage-tag highlighting
 
-Valid long-form coverage tags, such as `[impl->dsn~editor-presentation~3]`,
+Valid long-form coverage tags, such as `[impl->dsn~editor-presentation~5]`,
 are semantically highlighted in source, configuration, and markup files
-supported by OpenFastTrace. This does not validate that the target exists; it
-only recognizes the tag syntax while preserving the host language mode. See
+supported by OpenFastTrace. The bundled OpenFastTrace tag importer determines
+both the supported files and valid tag syntax. This does not validate that the
+target exists; it only recognizes the tag syntax while preserving the host language mode. See
 [Known Issues](#known-issues) for language-server compatibility limitations.
 
 ## Requirements

doc/changesets/06-authoritative-oft-coverage-tag-highlighting.md

@@ -0,0 +1,48 @@
+# 06 Authoritative OFT Coverage-Tag Highlighting
+
+## Goal
+
+Use OpenFastTrace's tag-importer APIs to decide which source-file coverage tags
+the language server highlights.
+
+## Scope
+
+In scope:
+
+* Replace the copied coverage-tag regular expression and file-extension list.
+* Upgrade the bundled OpenFastTrace dependency and use its tag importer.
+
+Out of scope:
+
+* Configured short tags, validation diagnostics, navigation, and decorations.
+
+## Design References
+
+* [System Requirements](../system_requirements.md)
+* [LSP-first Architecture](../design/architecture.md)
+* [Quality Requirements](../design/quality_requirements.md)
+
+## Task List
+
+### Requirements And Design
+
+- [x] Confirm `req~highlight-coverage-tags~1` and
+      `scn~recognize-and-highlight-oft-content~1` cover this behavior.
+- [x] Revise the editor-presentation and semantic-token implementation items
+      to delegate coverage-tag recognition to OpenFastTrace.
+
+### Implementation
+
+- [x] Upgrade to OpenFastTrace 4.7.0 and add the tag-importer dependency.
+- [x] Use the tag importer for supported-file selection and coverage-tag
+      recognition before emitting semantic tokens.
+
+### Verification
+
+- [x] Run server tests, extension compile/lint/unit/integration tests, the full
+      Gradle build, and the OpenFastTrace requirements trace.
+
+### Update User Documentation
+
+- [x] Document that coverage-tag recognition follows the bundled OpenFastTrace
+      tag importer.

doc/design/architecture.md

@@ -27,15 +27,6 @@ Covers:
 
 Needs: impl
 
-#### Open Issue: Authoritative OFT Parsing
-
-The current coverage-tag recognition uses LSP-local parsing logic, including a
-regular expression and a copied list of supported file extensions. Replace it
-with the OpenFastTrace library's parser and tag-importer APIs. The language
-server must obtain validation and ranges from that authoritative implementation
-so its highlighting, navigation, and trace behavior cannot drift from
-OpenFastTrace.
-
 ### VS Code Language Client
 `dsn~vscode-language-client~2`
 
@@ -97,7 +88,7 @@ Covers:
 - `impl~workspace-symbol-protocol-slice~1`
 
 ### Editor Presentation
-`dsn~editor-presentation~4`
+`dsn~editor-presentation~5`
 
 The VS Code adapter contributes a TextMate injection grammar for immediate OFT
 declaration coloring in Markdown and reStructuredText. It preserves the host
@@ -109,7 +100,9 @@ set of supported source, configuration, and markup languages would require
 per-host grammar injection rules and lexical matching cannot reliably separate
 valid coverage tags from lookalike text.
 
-The language server owns OFT tag parsing. Its semantic-token response is not a
+The language server delegates coverage-tag recognition and supported-file
+selection to the OpenFastTrace tag-importer APIs. It does not duplicate OFT's
+syntax or supported-file list. Its semantic-token response is not a
 reliable cross-language presentation mechanism: VS Code selects one
 full-document semantic-token provider rather than merging their results. A
 language-specific provider with a more specific document selector, such as the
@@ -139,18 +132,20 @@ editor decorations, preserving the host language's syntax and semantic
 highlighting.
 
 ### Coverage-Tag Semantic Tokens
-`impl~coverage-tag-semantic-tokens~1`
+`impl~coverage-tag-semantic-tokens~2`
 
-The language server tracks the current contents of opened files supported by
-the OpenFastTrace tag importer and serves `textDocument/semanticTokens/full`.
-It emits the standard `type` token only for syntactically valid
-long-form coverage tags, letting VS Code apply the active theme's standard
-semantic-token styling without replacing each document's host language grammar.
+The language server tracks the current contents of opened files and serves
+`textDocument/semanticTokens/full`. It asks the OpenFastTrace tag importer
+whether each file is supported and uses its coverage-tag parser to decide which
+long-form tag ranges receive the standard `type` token. This keeps token output
+aligned with OpenFastTrace while letting VS Code apply the active theme's
+standard semantic-token styling without replacing each document's host
+language grammar.
 
 Tags: mvp
 
 Covers:
-- `dsn~editor-presentation~4`
+- `dsn~editor-presentation~5`
 
 Needs: itest
 

extension/src/test/integration/extension.test.ts

@@ -126,7 +126,7 @@ suite('OpenFastTrace extension', () => {
     assert.equal(captures['4'].name, 'constant.numeric.openfasttrace.revision');
   });
 
-  // [itest->impl~coverage-tag-semantic-tokens~1]
+  // [itest->impl~coverage-tag-semantic-tokens~2]
   test('returns coverage-tag semantic tokens from the live language server', async () => {
     const extension = vscode.extensions.getExtension<OpenFastTraceExports>('itsallcode.openfasttrace');
     assert.ok(extension, 'OpenFastTrace extension must be available to the test host');
@@ -137,7 +137,7 @@ suite('OpenFastTrace extension', () => {
     await client.sendNotification('textDocument/didOpen', {
       textDocument: {
         uri: 'file:///workspace/coverage-tag.java', languageId: 'java', version: 1,
-        text: `// ${coverageTag('dsn~editor-presentation~3')}\n// ${coverageTag('dsn~1invalid~3')}`,
+        text: `// ${coverageTag('dsn~editor-presentation~5')}\n// ${coverageTag('dsn~1invalid~3')}`,
       },
     });
 

gradle/libs.versions.toml

@@ -1,10 +1,11 @@
 [versions]
 lsp4j = "1.0.0"
-openfasttrace = "4.4.0"
+openfasttrace = "4.7.0"
 junit = "5.13.4"
 hamcrest = "3.0"
 
 [libraries]
 lsp4j = { group = "org.eclipse.lsp4j", name = "org.eclipse.lsp4j", version.ref = "lsp4j" }
 openfasttrace = { group = "org.itsallcode.openfasttrace", name = "openfasttrace", version.ref = "openfasttrace" }
+openfasttrace-tag-importer = { group = "org.itsallcode.openfasttrace", name = "openfasttrace-importer-tag", version.ref = "openfasttrace" }
 hamcrest = { group = "org.hamcrest", name = "hamcrest", version.ref = "hamcrest" }

server/build.gradle

@@ -28,6 +28,7 @@ sonar {
 dependencies {
     implementation libs.lsp4j
     implementation libs.openfasttrace
+    implementation libs.openfasttrace.tag.importer
 }
 
 testing {

server/src/main/java/org/itsallcode/openfasttrace/lsp/OpenFastTraceTextDocumentService.java

@@ -1,14 +1,14 @@
 package org.itsallcode.openfasttrace.lsp;
 
+import java.io.BufferedReader;
+import java.io.StringReader;
 import java.net.URI;
+import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
-import java.util.Set;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ConcurrentHashMap;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
 
 import org.eclipse.lsp4j.DidChangeTextDocumentParams;
 import org.eclipse.lsp4j.DidCloseTextDocumentParams;
@@ -17,25 +17,20 @@
 import org.eclipse.lsp4j.SemanticTokens;
 import org.eclipse.lsp4j.SemanticTokensParams;
 import org.eclipse.lsp4j.services.TextDocumentService;
+import org.itsallcode.openfasttrace.api.importer.ImportSettings;
+import org.itsallcode.openfasttrace.api.importer.ImporterContext;
+import org.itsallcode.openfasttrace.api.importer.SpecificationListBuilder;
+import org.itsallcode.openfasttrace.api.importer.input.InputFile;
+import org.itsallcode.openfasttrace.importer.tag.TagImporterFactory;
+import org.itsallcode.openfasttrace.importer.tag.common.CoverageTagParser;
 
 /**
  * Tracks open documents and provides semantic highlighting for coverage tags.
  */
-// [impl->dsn~editor-presentation~4]
+// [impl->dsn~editor-presentation~5]
 final class OpenFastTraceTextDocumentService implements TextDocumentService {
-    private static final Set<String> SUPPORTED_FILE_EXTENSIONS = Set.of(
-            "ads", "adb", "bat", "c", "C", "cc", "cpp", "c++", "h", "H", "h++", "hh", "hpp", "c#", "cs", "cfg",
-            "conf", "ini", "feature", "go", "groovy", "json", "htm", "html", "xhtml", "yaml", "yml", "java",
-            "clj", "kt", "kts", "scala", "js", "mjs", "cjs", "ejs", "ts", "lua", "m", "mm", "php", "proto",
-            "pl", "pm", "py", "robot", "pu", "puml", "plantuml", "r", "rs", "sh", "bash", "zsh", "sv", "v",
-            "inc", "swift", "toml", "tf", "tfvars", "sql", "pls");
-    private static final Pattern COVERAGE_TAG_PATTERN = Pattern.compile(
-            "\\[\\s*\\p{Alpha}+(?:~\\p{Alpha}[\\p{L}\\p{N}_-]*(?:\\.[\\p{L}\\p{N}_-]+)*+~\\d+)?\\s*->\\s*"
-                    + "\\p{Alpha}+~\\p{Alpha}[\\p{L}\\p{N}_-]*(?:\\.[\\p{L}\\p{N}_-]+)*+~\\d+"
-                    + "(?:\\s*>>\\s*\\p{Alpha}+(?:\\s*,\\s*\\p{Alpha}+)*)?\\s*\\]",
-            Pattern.UNICODE_CHARACTER_CLASS);
-
     private final Map<String, String> documents = new ConcurrentHashMap<>();
+    private final TagImporterFactory tagImporterFactory = tagImporterFactory();
 
     @Override
     public void didOpen(final DidOpenTextDocumentParams parameters) {
@@ -67,41 +62,87 @@ public void didSave(final DidSaveTextDocumentParams parameters) {
     @Override
     public CompletableFuture<SemanticTokens> semanticTokensFull(final SemanticTokensParams parameters) {
         final String uri = parameters.getTextDocument().getUri();
-        if (!isSupportedFile(uri)) {
+        final String text = this.documents.get(uri);
+        final InputFile file = new DocumentInput(uri, text);
+        if (!this.tagImporterFactory.supportsFile(file)) {
             return CompletableFuture.completedFuture(new SemanticTokens(List.of()));
         }
-        return CompletableFuture.completedFuture(new SemanticTokens(tokensFor(this.documents.get(uri))));
+        return CompletableFuture.completedFuture(new SemanticTokens(tokensFor(file, text)));
     }
 
-    private static boolean isSupportedFile(final String documentUri) {
-        final String path = URI.create(documentUri).getPath();
-        if (path == null) {
-            return false;
-        }
-        final int extensionStart = path.lastIndexOf('.') + 1;
-        return extensionStart > 0 && SUPPORTED_FILE_EXTENSIONS.contains(path.substring(extensionStart));
+    private static TagImporterFactory tagImporterFactory() {
+        final TagImporterFactory factory = new TagImporterFactory();
+        factory.init(new ImporterContext(ImportSettings.createDefault()));
+        return factory;
     }
 
-    private static List<Integer> tokensFor(final String text) {
+    private static List<Integer> tokensFor(final InputFile file, final String text) {
         if (text == null) {
             return List.of();
         }
         final var tokens = new ArrayList<Integer>();
         int previousLine = 0;
         int previousStart = 0;
-        final String[] lines = text.split("\\R", -1);
-        for (int lineNumber = 0; lineNumber < lines.length; lineNumber++) {
-            final Matcher matcher = COVERAGE_TAG_PATTERN.matcher(lines[lineNumber]);
-            while (matcher.find()) {
+        final List<String> lines = text.lines().toList();
+        for (int lineNumber = 0; lineNumber < lines.size(); lineNumber++) {
+            for (final TagRange range : tagRanges(file, lineNumber + 1, lines.get(lineNumber))) {
                 tokens.add(lineNumber - previousLine);
-                tokens.add(lineNumber == previousLine ? (matcher.start() - previousStart) : matcher.start());
-                tokens.add(matcher.end() - matcher.start());
+                tokens.add(lineNumber == previousLine ? (range.start() - previousStart) : range.start());
+                tokens.add(range.end() - range.start());
                 tokens.add(0);
                 tokens.add(0);
                 previousLine = lineNumber;
-                previousStart = matcher.start();
+                previousStart = range.start();
             }
         }
         return tokens;
     }
+
+    private static List<TagRange> tagRanges(final InputFile file, final int lineNumber, final String line) {
+        final var ranges = new ArrayList<TagRange>();
+        int start = line.indexOf('[');
+        while (start >= 0) {
+            final int end = line.indexOf(']', start + 1);
+            if (end < 0) {
+                return ranges;
+            }
+            if (isCoverageTag(file, lineNumber, line.substring(start, end + 1))) {
+                ranges.add(new TagRange(start, end + 1));
+            }
+            start = line.indexOf('[', end + 1);
+        }
+        return ranges;
+    }
+
+    private static boolean isCoverageTag(final InputFile file, final int lineNumber, final String candidate) {
+        final SpecificationListBuilder listener = SpecificationListBuilder.create();
+        CoverageTagParser.create(null, file, listener).readLine(lineNumber, candidate);
+        return !listener.build().isEmpty();
+    }
+
+    private record TagRange(int start, int end) {
+    }
+
+    private record DocumentInput(String uri, String text) implements InputFile {
+        @Override
+        public BufferedReader createReader() {
+            return new BufferedReader(new StringReader(this.text == null ? "" : this.text));
+        }
+
+        @Override
+        public String getPath() {
+            final String path = URI.create(this.uri).getPath();
+            return path == null ? this.uri : path;
+        }
+
+        @Override
+        public boolean isRealFile() {
+            return false;
+        }
+
+        @Override
+        public Path toPath() {
+            throw new UnsupportedOperationException("An open editor document has no filesystem path.");
+        }
+    }
 }

server/src/test/java/org/itsallcode/openfasttrace/lsp/OpenFastTraceLanguageServerTest.java

@@ -79,7 +79,7 @@ void returnsTokensForValidCoverageTagsInSupportedFilesAndUpdatesThemOnChange() {
         final OpenFastTraceTextDocumentService documents = new OpenFastTraceTextDocumentService();
         final String uri = "file:///workspace/source.ts";
         documents.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(uri, "typescript", 1,
-                "// " + coverageTag("dsn~editor-presentation~3"))));
+                "// " + coverageTag("dsn~editor-presentation~5"))));
 
         assertThat(tokensFor(documents, uri), is(List.of(0, 3, 33, 0, 0)));
 
@@ -94,14 +94,14 @@ void recognizesSourceConfigurationAndMarkupFilesButIgnoresMalformedAndUnsupporte
         final OpenFastTraceTextDocumentService documents = new OpenFastTraceTextDocumentService();
         documents.didOpen(
                 new DidOpenTextDocumentParams(new TextDocumentItem("file:///workspace/source.ts", "typescript", 1,
-                        "// " + coverageTag("dsn~editor-presentation~3"))));
+                        "// " + coverageTag("dsn~editor-presentation~5"))));
         documents.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem("file:///workspace/source.yaml", "yaml", 1,
-                "# " + coverageTag("dsn~editor-presentation~3") + "\n# " + coverageTag("dsn~1invalid~3"))));
+                "# " + coverageTag("dsn~editor-presentation~5") + "\n# " + coverageTag("dsn~1invalid~3"))));
         documents.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem("file:///workspace/page.html", "html", 1,
-                "<!-- " + coverageTag("dsn~editor-presentation~3") + " -->")));
+                "<!-- " + coverageTag("dsn~editor-presentation~5") + " -->")));
         documents.didOpen(
                 new DidOpenTextDocumentParams(new TextDocumentItem("file:///workspace/requirements.md", "markdown", 1,
-                        coverageTag("dsn~editor-presentation~3"))));
+                        coverageTag("dsn~editor-presentation~5"))));
 
         assertAll(
                 () -> assertThat(tokensFor(documents, "file:///workspace/source.ts"), is(List.of(0, 3, 33, 0, 0))),
@@ -114,6 +114,27 @@ void recognizesSourceConfigurationAndMarkupFilesButIgnoresMalformedAndUnsupporte
         assertThat(tokensFor(documents, "file:///workspace/source.ts"), is(List.of()));
     }
 
+    @Test
+    void usesTheTagImporterForSupportedFilesAndCoverageTagSyntax() {
+        final OpenFastTraceTextDocumentService documents = new OpenFastTraceTextDocumentService();
+        final String uri = "file:///workspace/page.xml";
+        documents.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(uri, "xml", 1,
+                "<!-- " + coverageTag("dsn~editor-presentation~5") + " -->")));
+
+        assertThat(tokensFor(documents, uri), is(List.of(0, 5, 33, 0, 0)));
+    }
+
+    @Test
+    void returnsRangesForEachOftRecognizedTagOnALine() {
+        final OpenFastTraceTextDocumentService documents = new OpenFastTraceTextDocumentService();
+        final String uri = "file:///workspace/source.ts";
+        documents.didOpen(new DidOpenTextDocumentParams(new TextDocumentItem(uri, "typescript", 1,
+                "// " + coverageTag("dsn~editor-presentation~5") + " "
+                        + coverageTag("req~highlight-coverage-tags~1"))));
+
+        assertThat(tokensFor(documents, uri), is(List.of(0, 3, 33, 0, 0, 0, 34, 37, 0, 0)));
+    }
+
     private static List<Integer> tokensFor(final OpenFastTraceTextDocumentService documents, final String uri) {
         return documents.semanticTokensFull(new SemanticTokensParams(new TextDocumentIdentifier(uri))).join().getData();
     }