Populate ScriptTokenStream and propagate precise token bounds during AST construction by Copilot · Pull Request #35 · fupacat/tsql-scriptdom-java

Copilot · 2026-10-12T19:38:44Z

TSqlParser.parse(sql) was producing AST fragments without usable token-range metadata: TSqlScript.ScriptTokenStream remained empty and many child fragments kept default token bounds (firstTokenIndex/lastTokenIndex), which prevented downstream analyzers from computing exact source spans.
This change wires token-stream retention and bound propagation through parser construction paths so fragment ranges map to real token positions.

Token stream capture in parser input filter
- TSqlWhitespaceTokenFilter now records emitted tokens in parse order and tracks token-object → stream-index mapping.
- Exposes helper accessors for generated parser internals to retrieve the full script stream and resolve token indices.
Generated parser helper: real token-bound propagation
- Updated parser generation (scripts/generate_parser.py) so generated helper methods (UpdateTokenInfo, AddAndUpdateTokenInfo) now:
  - attach script token stream onto fragments,
  - resolve token indices from parser tokens,
  - update parent fragment bounds when adding child fragments/collections.
AST base behavior: fragment-to-fragment bound merge
- Updated AST generation (AstCodeGenerator) so TSqlFragment.UpdateTokenInfo(Object) merges bounds when passed another TSqlFragment instead of no-op behavior.
Regression coverage for range metadata
- Added parser facade test assertions that script token stream is populated and fragment token bounds are non-default and propagated.

ParseResult<TSqlScript> result = TSqlParser.parse("SELECT 1 AS a; SELECT 2 AS b;");

assertThat(result.fragment().ScriptTokenStream).isNotEmpty();
assertThat(result.fragment().lastTokenIndex)
    .isEqualTo(result.fragment().ScriptTokenStream.size() - 1);

TSqlBatch batch = result.fragment().Batches.get(0);
assertThat(batch.lastTokenIndex).isGreaterThan(batch.firstTokenIndex);

Fixes Populate ScriptTokenStream and precise token bounds during AST construction #34

Co-authored-by: fupacat <595487+fupacat@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The bound-propagation changes contain correctness bugs (a firstTokenIndex == 0 sentinel that corrupts leading fragments, and per-fragment stream copies that are partial and O(fragments × tokens)) in generated-parser code whose runtime behavior cannot be fully verified here.

2 open findings

What changed in this PR

This PR makes TSqlParser.parse(sql) emit usable source-span metadata so downstream analyzers (e.g. SonarQube highlighting, per issue #34) can map AST fragments back to exact token ranges. It captures the full token stream in the parser's input filter, exposes helpers to resolve a token's stream index, and rewrites the generated parser's UpdateTokenInfo/AddAndUpdateTokenInfo helpers plus the TSqlFragment base merge logic to attach the stream and propagate firstTokenIndex/lastTokenIndex bounds through construction.

Changes:

TSqlWhitespaceTokenFilter now records every emitted token in parse order and maintains a token→index map, with accessors for the generated parser.
The generated helper methods attach the script token stream onto fragments, resolve token indices, and propagate parent bounds when adding children; TSqlFragment.UpdateTokenInfo(Object) now merges bounds from another fragment.
A new facade test asserts the root stream is populated and fragment bounds are non-default.

File	Description
tsql-scriptdom-core/src/main/java/io/github/fupacat/TSqlWhitespaceTokenFilter.java	Records emitted tokens into `scriptTokenStream`, maps token→index, and exposes `_getScriptTokenStream`/`_getTokenIndex`.
scripts/generate_parser.py	Rewrites generated helpers to attach the stream and propagate bounds; removes the no-op `UpdateTokenInfo`.
tsql-scriptdom-astgen/src/main/java/io/github/fupacat/AstCodeGenerator.java	Adds fragment-to-fragment bound merge in `TSqlFragment.UpdateTokenInfo(Object)`.
tsql-scriptdom-core/src/test/java/io/github/fupacat/TSqlParserFacadeTest.java	Adds a regression test for populated stream and propagated token bounds.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

-        '    public void AddAndUpdateTokenInfo(Object... args) { if (args.length >= 3 && args[1] instanceof java.util.List list && args[2] != null) { if (args[2] instanceof java.util.Collection<?> c) { list.addAll(c); } else { list.add(args[2]); } if (args[0] instanceof SchemaObjectName son) { int count = son.Identifiers.size(); if (count > 0) son.BaseIdentifier = son.Identifiers.get(count - 1); if (count > 1) son.SchemaIdentifier = son.Identifiers.get(count - 2); if (count > 2) son.DatabaseIdentifier = son.Identifiers.get(count - 3); if (count > 3) son.ServerIdentifier = son.Identifiers.get(count - 4); } } }',
+        '    public void AddAndUpdateTokenInfo(Object... args) { if (args.length >= 3 && args[1] instanceof java.util.List list && args[2] != null) { if (args[2] instanceof java.util.Collection<?> c) { list.addAll(c); for (Object item : c) { UpdateTokenInfo(args[0], item); } } else { list.add(args[2]); UpdateTokenInfo(args[0], args[2]); } if (args[0] instanceof SchemaObjectName son) { int count = son.Identifiers.size(); if (count > 0) son.BaseIdentifier = son.Identifiers.get(count - 1); if (count > 1) son.SchemaIdentifier = son.Identifiers.get(count - 2); if (count > 2) son.DatabaseIdentifier = son.Identifiers.get(count - 3); if (count > 3) son.ServerIdentifier = son.Identifiers.get(count - 4); } } }',
+        '    public int ResolveTokenIndex(Object tokenLike) { if (_tokenSource == null || tokenLike == null) return -1; if (tokenLike instanceof Token tok) { return _tokenSource._getTokenIndex(tok); } return -1; }',
+        '    public void AttachScriptTokenStream(Object fragment) { if (_tokenSource == null || !(fragment instanceof TSqlFragment f) || f.ScriptTokenStream == null) return; if (f.ScriptTokenStream.isEmpty()) { f.ScriptTokenStream.addAll(_tokenSource._getScriptTokenStream()); } }',


                        .addModifiers(javax.lang.model.element.Modifier.PUBLIC)
                        .addParameter(Object.class, "other")
+                        .beginControlFlow("if (other instanceof TSqlFragment fragment)")
+                        .beginControlFlow("if (this.firstTokenIndex == 0 || (fragment.firstTokenIndex > 0 && fragment.firstTokenIndex < this.firstTokenIndex))")


Copilot

🔵 Needs a closer look

Per-fragment partial token-stream snapshots leave child/root ScriptTokenStream incomplete while token indices point beyond them, breaking the intended downstream span lookups for multi-statement/multi-batch scripts.

2 open findings

🧠 Review effort: Balanced

…ript-token-stream # Conflicts: # tsql-scriptdom-core/src/test/java/io/github/fupacat/TSqlParserFacadeTest.java

Copilot

🟡 Changes recommended

The 0-as-sentinel bound logic and the per-fragment stream snapshot produce incorrect firstTokenIndex/ScriptTokenStream values for fragments anchored at token 0 and for multi-batch scripts, undermining the PR's core goal.

4 open findings

🧠 Review effort: Balanced

+        '    public void AddAndUpdateTokenInfo(Object... args) { if (args.length >= 3 && args[1] instanceof java.util.List list && args[2] != null) { if (args[2] instanceof java.util.Collection<?> c) { list.addAll(c); for (Object item : c) { UpdateTokenInfo(args[0], item); } } else { list.add(args[2]); UpdateTokenInfo(args[0], args[2]); } if (args[0] instanceof SchemaObjectName son) { int count = son.Identifiers.size(); if (count > 0) son.BaseIdentifier = son.Identifiers.get(count - 1); if (count > 1) son.SchemaIdentifier = son.Identifiers.get(count - 2); if (count > 2) son.DatabaseIdentifier = son.Identifiers.get(count - 3); if (count > 3) son.ServerIdentifier = son.Identifiers.get(count - 4); } } }',
+        '    public int ResolveTokenIndex(Object tokenLike) { if (_tokenSource == null || tokenLike == null) return -1; if (tokenLike instanceof Token tok) { return _tokenSource._getTokenIndex(tok); } return -1; }',
+        '    public void AttachScriptTokenStream(Object fragment) { if (_tokenSource == null || !(fragment instanceof TSqlFragment f) || f.ScriptTokenStream == null) return; if (f.ScriptTokenStream.isEmpty()) { f.ScriptTokenStream.addAll(_tokenSource._getScriptTokenStream()); } }',
+        '    public void UpdateTokenInfo(Object... args) { if (args == null || args.length == 0 || !(args[0] instanceof TSqlFragment fragment)) return; AttachScriptTokenStream(fragment); if (args.length >= 3 && args[1] instanceof Number first && args[2] instanceof Number last) { fragment.UpdateTokenInfo(first.intValue(), last.intValue()); return; } if (args.length >= 2 && args[1] instanceof TSqlFragment child) { fragment.UpdateTokenInfo(child); return; } if (args.length >= 2) { int idx = ResolveTokenIndex(args[1]); if (idx >= 0) { if (fragment.firstTokenIndex == 0 || idx < fragment.firstTokenIndex) { fragment.firstTokenIndex = idx; } if (idx > fragment.lastTokenIndex) { fragment.lastTokenIndex = idx; } } } }',


        assertThat(result.isSuccess()).isTrue();
    }

+    @Test


Initial plan

2ca6d43

Copilot AI assigned Copilot and fupacat Oct 12, 2026

Copilot started work on behalf of fupacat October 12, 2026 19:39

View session

Copilot AI linked an issue Oct 12, 2026 that may be closed by this pull request

Populate ScriptTokenStream and precise token bounds during AST construction #34

Closed

Populate token stream and token index bounds during parse

06db39e

Co-authored-by: fupacat <595487+fupacat@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Populate ScriptTokenStream and precise token bounds during AST construction~~ Populate ScriptTokenStream and propagate precise token bounds during AST construction Oct 12, 2026

Copilot finished work on behalf of fupacat October 12, 2026 19:48

Copilot AI requested a review from fupacat October 12, 2026 19:48

fupacat marked this pull request as ready for review October 12, 2026 20:08

Copilot AI balanced review requested due to automatic review settings October 12, 2026 20:08

Copilot started reviewing on behalf of fupacat October 12, 2026 20:09

View session

Merge branch 'master' into copilot/populate-script-token-stream

11a229d

fupacat enabled auto-merge (squash) October 12, 2026 20:09

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 12, 2026 20:18

Copilot started reviewing on behalf of fupacat October 12, 2026 20:19

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into copilot/populate-sc…

202782f

…ript-token-stream # Conflicts: # tsql-scriptdom-core/src/test/java/io/github/fupacat/TSqlParserFacadeTest.java

Copilot AI balanced review requested due to automatic review settings October 12, 2026 20:28

Copilot started reviewing on behalf of fupacat October 12, 2026 20:29

View session

fupacat merged commit d420a28 into master Oct 12, 2026
6 checks passed

fupacat deleted the copilot/populate-script-token-stream branch October 12, 2026 20:31

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Navigation Menu

Repository navigation

Populate ScriptTokenStream and propagate precise token bounds during AST construction - #35

Populate ScriptTokenStream and propagate precise token bounds during AST construction#35
fupacat merged 4 commits into
masterfrom
copilot/populate-script-token-stream

Copilot AI commented Oct 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Oct 12, 2026 •

edited

Loading