`, and no sentence collides with a citation's closing parenthesis. + +### Before / After + +Rendered HTML for the reported message, before: + +```html +
Admins can configure the extraction approach for images and PDFs under Admin Settings > Search & Extract. The available modes are:
+``` + +### User Experience Improvements + +- Paragraphs, bullets, and numbered lists after a citation render in the structure the model actually produced. +- A citation followed by more text on the same line keeps its separating space. +- Back-to-back citations no longer collide. +- Copied and exported message markdown is built from the same parsed output, so it keeps its line breaks too. + +## Cross-References + +- Functional test: `functional_tests/test_chat_citation_whitespace_preservation.py` +- UI test: `ui_tests/test_chat_citation_paragraph_spacing.py` +- Related: [Citation Improvements](../features/v0.238.024/CITATION_IMPROVEMENTS.md) diff --git a/docs/explanation/fixes/index.md b/docs/explanation/fixes/index.md index 42ccf6661..e80b770e4 100644 --- a/docs/explanation/fixes/index.md +++ b/docs/explanation/fixes/index.md @@ -31,3 +31,4 @@ category: Version History - [Cosmos Container Throughput Deployer Fix](COSMOS_CONTAINER_THROUGHPUT_DEPLOYER_FIX.md) - [Workflow Task Document Picker Fix](WORKFLOW_TASK_DOCUMENT_PICKER_FIX.md) - [Workflow File Sync Prompt Context Fix](WORKFLOW_FILE_SYNC_PROMPT_CONTEXT_FIX.md) +- [Chat Citation Whitespace Collapse Fix](CHAT_CITATION_WHITESPACE_COLLAPSE_FIX.md) diff --git a/docs/explanation/release_notes.md b/docs/explanation/release_notes.md index c24f19320..5008edca0 100644 --- a/docs/explanation/release_notes.md +++ b/docs/explanation/release_notes.md @@ -2,6 +2,17 @@ For feature-focused and fix-focused drill-downs by version, see [Features by Version](/explanation/features/) and [Fixes by Version](/explanation/fixes/). +### **(v0.250.229)** + +#### Bug Fixes + +* **Citations No Longer Eat the Line Break After Them** + * Text that came after an inline document citation was jammed onto the end of the closing parenthesis instead of starting a new paragraph โ you would see `(Source: uploading_documents.md, Page: 1)Thank you, Paul.` with no break at all. + * The citation parser matched the `[#citation-id]` marker along with the whitespace that followed it, then rebuilt the citation without putting that whitespace back. Because this runs on the raw markdown before it is rendered, a deleted blank line did not just remove a space โ it changed how the rest of the block was read, so a paragraph after a cited list item got absorbed into the list item itself. + * Spacing is now restored exactly as the model wrote it. Paragraphs, bullets, and numbered lists after a citation render in their intended structure, a citation followed by more text on the same line keeps its space, and back-to-back citations stop colliding. Copied and exported message text keeps its line breaks for the same reason. + * The cleanup pass for leftover citation markers had the same flaw in reverse and could swallow the blank line *before* a stray marker. It now only removes horizontal spacing, or the marker's whole line when it sits on one. + * (Ref: #1289, `chat-citations.js`, `parseCitations()`, chat message rendering) + ### **(v0.250.228)** #### Bug Fixes diff --git a/functional_tests/test_chat_citation_whitespace_preservation.py b/functional_tests/test_chat_citation_whitespace_preservation.py new file mode 100644 index 000000000..787736db7 --- /dev/null +++ b/functional_tests/test_chat_citation_whitespace_preservation.py @@ -0,0 +1,410 @@ +#!/usr/bin/env python3 +# test_chat_citation_whitespace_preservation.py +""" +Functional test for inline citation whitespace preservation. +Version: 0.250.229 +Implemented in: 0.250.229 + +This test ensures that parseCitations() in chat-citations.js no longer deletes the +whitespace that follows an inline document citation. The citation regex captures the +whitespace after the trailing [#citation-id] marker, and before this fix that +whitespace was dropped, collapsing the following paragraph onto the end of the +citation and, inside lists, absorbing it into the preceding list item. + +parseCitations() runs on raw markdown before marked.parse(), so the harness below +executes the real production function and then renders the result with the vendored +marked bundle to assert the user-visible block structure. + +Refs: #1289 +""" + +import json +import os +import shutil +import subprocess +import sys +import tempfile + +sys.path.append(os.path.dirname(os.path.abspath(__file__))) + +from test_support.versioning import assert_app_version_at_least + + +ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +CHAT_JS_DIR = os.path.join(ROOT_DIR, 'application', 'single_app', 'static', 'js', 'chat') +CITATIONS_FILE = os.path.join(CHAT_JS_DIR, 'chat-citations.js') +MARKED_FILE = os.path.join(CHAT_JS_DIR, 'marked.min.js') + +CITATION_ONE = '181b54f7-fcf9-479b-a58b-81a5da3ba251_1' +CITATION_TWO = '159255fa-79d6-4619-9131-819813ae7997_1' +CITATION_THREE = '5e583e44-ea82-4569-bda8-e545bf12dca4_1' +CITATION_GENERIC = 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' + +REPORTED_MESSAGE = ( + 'Simple Chat processes uploaded images as part of its document-ingestion pipeline:\n' + '\n' + '5. **Grounded chat:** Once processing completes, the image-derived content can be ' + 'found through hybrid keyword and vector search and used to support cited answers in ' + f'chat. (Source: application_workflows.md, Page: 1) [#{CITATION_ONE}]\n' + '\n' + 'Admins can configure the extraction approach for images and PDFs under ' + '**Admin Settings > Search & Extract**. The available modes are:\n' + '\n' + '- **Auto:** Samples the content and chooses the richer path when the document ' + f'structure warrants it (Source: document-intelligence.md, Page: 1) [#{CITATION_TWO}]\n' + '\n' + 'For best results, upload clear, readable images. ' + f'(Source: uploading_documents.md, Page: 1) [#{CITATION_THREE}]\n' + '\n' + 'Thank you, Paul.' +) + +TEST_CASES = { + 'reported': REPORTED_MESSAGE, + 'inline_space': ( + f'See the policy (Source: Policy.pdf, Page: 12) [#{CITATION_GENERIC}_12] and then act.' + ), + 'trailing_punctuation': ( + f'See the policy (Source: Policy.pdf, Page: 12) [#{CITATION_GENERIC}_12].' + ), + 'back_to_back': ( + f'Both agree (Source: A.pdf, Page: 1) [#{CITATION_GENERIC}_1] ' + f'(Source: B.pdf, Page: 2) [#{CITATION_GENERIC}_2] end.' + ), + 'citation_id_on_next_line': ( + 'Findings are consistent. (Source: Report.pdf, Page: 4)\n' + f'[#{CITATION_GENERIC}_4]\n' + '\n' + 'The next section explains why.' + ), + 'stray_bracket_own_line': ( + 'Model used passim style.\n' + '\n' + f'[#{CITATION_GENERIC}_3]\n' + '\n' + 'Next paragraph.' + ), + 'stray_bracket_starts_paragraph': ( + 'Model used passim style.\n' + '\n' + f'[#{CITATION_GENERIC}_3] Next paragraph.' + ), + 'stray_bracket_inline': ( + f'Model used passim style [#{CITATION_GENERIC}_3] mid sentence.' + ), + 'stray_bracket_consecutive': ( + f'Model used passim style [#{CITATION_GENERIC}_3] [#{CITATION_GENERIC}_4] mid sentence.' + ), +} + +NODE_HARNESS = r""" +const fs = require('fs'); +const vm = require('vm'); + +const ESCAPE_MAP = { '&': '&', '<': '<', '>': '>', '"': '"' }; + +function escapeText(value) { + return String(value).replace(/[&<>]/g, character => ESCAPE_MAP[character]); +} + +function escapeAttribute(value) { + return String(value).replace(/[&<>"]/g, character => ESCAPE_MAP[character]); +} + +class FakeElement { + constructor(tagName) { + this.tagName = tagName; + this.childNodes = []; + this.dataset = {}; + this.attributes = {}; + this.textValue = ''; + } + set textContent(value) { + this.textValue = value === null || value === undefined ? '' : String(value); + this.childNodes = []; + } + get textContent() { return this.textValue; } + set href(value) { this.attributes.href = value; } + set className(value) { this.attributes.class = value; } + set target(value) { this.attributes.target = value; } + set rel(value) { this.attributes.rel = value; } + appendChild(child) { this.childNodes.push(child); return child; } + get innerHTML() { + if (this.childNodes.length) { + return this.childNodes.map(child => child.outerHTML).join(''); + } + return escapeText(this.textValue); + } + get outerHTML() { + const parts = []; + Object.keys(this.attributes).forEach(name => { + parts.push(`${name}="${escapeAttribute(this.attributes[name])}"`); + }); + Object.keys(this.dataset).forEach(name => { + const attributeName = 'data-' + name.replace(/[A-Z]/g, match => '-' + match.toLowerCase()); + parts.push(`${attributeName}="${escapeAttribute(this.dataset[name])}"`); + }); + const attributeText = parts.length ? ' ' + parts.join(' ') : ''; + return `<${this.tagName}${attributeText}>${this.innerHTML}`; + } +} + +const documentStub = { + getElementById: () => null, + createElement: tagName => new FakeElement(tagName), + addEventListener: () => {}, +}; + +const citationSource = fs.readFileSync(process.argv[2], 'utf8') + .replace(/^\s*import\s[^;]*;\s*$/gm, '') + .replace(/^export /gm, ''); + +const citationSandbox = { + module: { exports: {} }, + console, + document: documentStub, + window: { addEventListener: () => {} }, +}; +vm.runInNewContext(citationSource + '\nmodule.exports = { parseCitations };', citationSandbox); +const { parseCitations } = citationSandbox.module.exports; + +const markedSandbox = { module: { exports: {} }, console }; +markedSandbox.exports = markedSandbox.module.exports; +vm.runInNewContext(fs.readFileSync(process.argv[4], 'utf8'), markedSandbox); +const markedParse = markedSandbox.module.exports.parse; + +const cases = JSON.parse(fs.readFileSync(process.argv[3], 'utf8')); +const results = {}; +Object.keys(cases).forEach(name => { + const citationMarkdown = parseCitations(cases[name]); + results[name] = { + markdown: citationMarkdown, + html: markedParse(citationMarkdown), + }; +}); +process.stdout.write(JSON.stringify(results)); +""" + + +def read_file_text(file_path): + with open(file_path, 'r', encoding='utf-8') as file_handle: + return file_handle.read() + + +def run_citation_harness(): + """Execute the production parseCitations() and return markdown/HTML per case.""" + node_executable = shutil.which('node') + if not node_executable: + return None + + with tempfile.TemporaryDirectory() as temp_dir: + harness_path = os.path.join(temp_dir, 'citation_harness.cjs') + cases_path = os.path.join(temp_dir, 'cases.json') + + with open(harness_path, 'w', encoding='utf-8') as harness_file: + harness_file.write(NODE_HARNESS) + with open(cases_path, 'w', encoding='utf-8') as cases_file: + json.dump(TEST_CASES, cases_file) + + completed = subprocess.run( + [node_executable, harness_path, CITATIONS_FILE, cases_path, MARKED_FILE], + capture_output=True, + text=True, + timeout=60, + ) + + assert completed.returncode == 0, f'Citation harness failed: {completed.stderr}' + return json.loads(completed.stdout) + + +def test_reported_message_keeps_its_paragraph_breaks(): + """The reported message must keep every block separate after citation parsing.""" + print('๐ Testing reported citation message keeps paragraph breaks...') + + try: + results = run_citation_harness() + if results is None: + print('โ ๏ธ Node.js not available, skipping executable citation parsing check') + return True + + markdown = results['reported']['markdown'] + html = results['reported']['html'] + + collapsed_forms = [ + ')Admins can configure', + ')For best results', + ')Thank you, Paul.', + ] + for collapsed in collapsed_forms: + assert collapsed not in markdown, \ + f'Citation swallowed the whitespace before "{collapsed.lstrip(")")}"' + + assert ')\n\nAdmins can configure' in markdown, \ + 'Blank line between the citation and the following paragraph was not preserved' + assert ')\n\nFor best results' in markdown, \ + 'Blank line before "For best results" was not preserved' + assert ')\n\nThank you, Paul.' in markdown, \ + 'Blank line before the closing sentence was not preserved' + + assert 'Admins can configure the extraction approach' in html, \ + 'Follow-on paragraph must render as its own block, not inside the numbered list item' + assert '
For best results, upload clear, readable images.' in html, \ + 'Follow-on paragraph must render as its own block, not inside the bullet list item' + assert '
Thank you, Paul.
' in html, \ + 'Closing sentence must render as its own paragraph' + + assert 'Model used passim style.
' in own_line_html, \ + 'A bracket alone on its own line must not merge the paragraphs around it' + assert 'Next paragraph.
' in own_line_html, \ + 'The paragraph after a whole-line bracket must stay separate' + + starts_paragraph = results['stray_bracket_starts_paragraph'] + assert 'Model used passim style. Next paragraph.' not in starts_paragraph['markdown'], \ + 'Cleanup must not pull the following paragraph onto the previous line' + assert 'Next paragraph.
' in starts_paragraph['html'], \ + 'A bracket that starts a paragraph must not destroy the paragraph break' + + inline_markdown = results['stray_bracket_inline']['markdown'] + assert inline_markdown == 'Model used passim style mid sentence.', \ + f'Inline bracket cleanup should leave a single space: {inline_markdown!r}' + + consecutive_markdown = results['stray_bracket_consecutive']['markdown'] + assert consecutive_markdown == 'Model used passim style mid sentence.', \ + f'Consecutive inline brackets should collapse cleanly: {consecutive_markdown!r}' + + print('โ Stray citation bracket cleanup passed') + return True + + except Exception as exc: + print(f'โ Test failed: {exc}') + import traceback + traceback.print_exc() + return False + + +def test_citation_source_restores_trailing_whitespace(): + """Guard the mechanism in source so the fix cannot be silently reverted.""" + print('๐ Testing chat-citations.js source guards...') + + try: + assert_app_version_at_least( + '0.250.229', + reason='Inline citation whitespace preservation shipped in 0.250.229.', + ) + + source = read_file_text(CITATIONS_FILE) + + assert 'const trailingWhitespaceMatch = /\\s*$/.exec(bracketSection);' in source, \ + 'parseCitations must capture the whitespace consumed by the citation bracket group' + assert '${linkedPagesText})${trailingWhitespace}`' in source, \ + 'parseCitations must re-emit the captured trailing whitespace' + + assert '/\\s*\\[#?[0-9a-f]{8}-' not in source, \ + 'The stray bracket cleanup must not consume leading newlines' + assert 'guidBracketRunPattern' in source, \ + 'Stray bracket runs should be cleaned up as a unit' + assert "`^[ \\\\t]*${guidBracketRunPattern}\\\\r?\\\\n`" in source, \ + 'A whole-line stray bracket run should be removed as a complete line' + assert "`(?:[ \\\\t]*${guidBracketPattern})+`" in source, \ + 'The inline stray bracket cleanup should only consume spaces and tabs' + + print('โ chat-citations.js source guards passed') + return True + + except Exception as exc: + print(f'โ Test failed: {exc}') + import traceback + traceback.print_exc() + return False + + +if __name__ == '__main__': + tests = [ + test_reported_message_keeps_its_paragraph_breaks, + test_inline_citation_spacing_is_preserved, + test_stray_citation_bracket_cleanup_keeps_line_structure, + test_citation_source_restores_trailing_whitespace, + ] + results = [] + + for test in tests: + print(f'\n๐งช Running {test.__name__}...') + results.append(test()) + + success = all(results) + print(f'\n๐ Results: {sum(results)}/{len(results)} tests passed') + sys.exit(0 if success else 1) diff --git a/ui_tests/test_chat_citation_paragraph_spacing.py b/ui_tests/test_chat_citation_paragraph_spacing.py new file mode 100644 index 000000000..030851528 --- /dev/null +++ b/ui_tests/test_chat_citation_paragraph_spacing.py @@ -0,0 +1,192 @@ +# test_chat_citation_paragraph_spacing.py +""" +UI test for paragraph spacing around inline document citations. + +Version: 0.250.229 +Implemented in: 0.250.229 + +This test ensures that text following an inline citation renders as its own block +instead of being glued onto the closing parenthesis of the citation. Before the fix, +parseCitations() consumed the newlines after the trailing [#citation-id] marker, so +the next paragraph was absorbed into the preceding list item and sentences ran +together with no space after ")". + +Refs: #1289 +""" + +import json +import os +from pathlib import Path + +import pytest +from playwright.sync_api import expect + + +BASE_URL = os.getenv("SIMPLECHAT_UI_BASE_URL", "").rstrip("/") +STORAGE_STATE = os.getenv("SIMPLECHAT_UI_STORAGE_STATE", "") + +CITATION_ONE = "181b54f7-fcf9-479b-a58b-81a5da3ba251_1" +CITATION_TWO = "159255fa-79d6-4619-9131-819813ae7997_1" +CITATION_THREE = "5e583e44-ea82-4569-bda8-e545bf12dca4_1" + +MESSAGE_CONTENT = ( + "Simple Chat processes uploaded images as part of its document-ingestion pipeline:\n" + "\n" + "5. **Grounded chat:** Once processing completes, the image-derived content can be " + "found through hybrid keyword and vector search and used to support cited answers in " + f"chat. (Source: application_workflows.md, Page: 1) [#{CITATION_ONE}]\n" + "\n" + "Admins can configure the extraction approach for images and PDFs under " + "**Admin Settings > Search & Extract**. The available modes are:\n" + "\n" + "- **Auto:** Samples the content and chooses the richer path when the document " + f"structure warrants it (Source: document-intelligence.md, Page: 1) [#{CITATION_TWO}]\n" + "\n" + "For best results, upload clear, readable images. " + f"(Source: uploading_documents.md, Page: 1) [#{CITATION_THREE}]\n" + "\n" + "Thank you, Paul." +) + + +def _require_ui_env(): + if not BASE_URL: + pytest.skip("Set SIMPLECHAT_UI_BASE_URL to run this UI test.") + if not STORAGE_STATE or not Path(STORAGE_STATE).exists(): + pytest.skip("Set SIMPLECHAT_UI_STORAGE_STATE to a valid authenticated Playwright storage state file.") + + +def _fulfill_json(route, payload, status=200): + route.fulfill( + status=status, + content_type="application/json", + body=json.dumps(payload), + ) + + +def _build_message_payload(): + hybrid_citations = [ + { + "file_name": "application_workflows.md", + "document_id": "181b54f7-fcf9-479b-a58b-81a5da3ba251", + "citation_id": CITATION_ONE, + "page_number": 1, + "chunk_id": CITATION_ONE, + }, + { + "file_name": "document-intelligence.md", + "document_id": "159255fa-79d6-4619-9131-819813ae7997", + "citation_id": CITATION_TWO, + "page_number": 1, + "chunk_id": CITATION_TWO, + }, + { + "file_name": "uploading_documents.md", + "document_id": "5e583e44-ea82-4569-bda8-e545bf12dca4", + "citation_id": CITATION_THREE, + "page_number": 1, + "chunk_id": CITATION_THREE, + }, + ] + + return { + "id": "assistant-citation-spacing-1", + "conversation_id": "test-citation-spacing", + "role": "assistant", + "content": MESSAGE_CONTENT, + "timestamp": "2026-08-18T15:32:45.480651", + "augmented": True, + "hybrid_citations": hybrid_citations, + "web_search_citations": [], + "agent_citations": [], + "metadata": {}, + } + + +@pytest.mark.ui +def test_text_after_a_citation_renders_as_its_own_block(playwright): + """Validate that citations do not absorb the paragraph that follows them.""" + _require_ui_env() + + browser = playwright.chromium.launch() + context = browser.new_context( + storage_state=STORAGE_STATE, + viewport={"width": 1440, "height": 900}, + ) + page = context.new_page() + message_payload = _build_message_payload() + + page.route( + "**/api/user/settings", + lambda route: _fulfill_json(route, {"selected_agent": None, "settings": {"enable_agents": False}}), + ) + page.route("**/api/get_conversations", lambda route: _fulfill_json(route, {"conversations": []})) + + try: + page.goto(f"{BASE_URL}/chats", wait_until="domcontentloaded") + page.wait_for_selector("#chatbox") + + page.evaluate( + """ + async (payload) => { + currentConversationId = payload.conversation_id; + window.currentConversationId = payload.conversation_id; + + const messagesModule = await import('/static/js/chat/chat-messages.js'); + messagesModule.appendMessage( + 'AI', + payload.content, + null, + payload.id, + payload.augmented, + payload.hybrid_citations, + payload.web_search_citations, + payload.agent_citations, + null, + null, + payload, + true + ); + } + """, + message_payload, + ) + + message = page.locator('.message[data-message-id="assistant-citation-spacing-1"]') + message_text = message.locator(".message-text") + expect(message_text).to_be_visible() + + # The three inline citations still render as clickable links. + expect(message_text.locator("a.citation-link")).to_have_count(3) + + # The numbered list item ends at its citation and does not swallow the next paragraph. + numbered_item = message_text.locator("ol > li") + expect(numbered_item).to_have_count(1) + expect(numbered_item).to_contain_text("Grounded chat:") + assert "Admins can configure" not in numbered_item.inner_text(), \ + "The paragraph after the citation was absorbed into the numbered list item" + + # The bullet ends at its citation and does not swallow the next paragraph either. + bullet_item = message_text.locator("ul > li") + expect(bullet_item).to_have_count(1) + expect(bullet_item).to_contain_text("Auto:") + assert "For best results" not in bullet_item.inner_text(), \ + "The paragraph after the citation was absorbed into the bullet list item" + + # Each following block renders as its own paragraph. + paragraph_texts = message_text.locator("p").all_inner_texts() + assert any(text.startswith("Admins can configure the extraction approach") for text in paragraph_texts), \ + f"Expected a standalone paragraph starting with 'Admins can configure': {paragraph_texts}" + assert any(text.startswith("For best results, upload clear, readable images.") for text in paragraph_texts), \ + f"Expected a standalone paragraph starting with 'For best results': {paragraph_texts}" + assert any(text.strip() == "Thank you, Paul." for text in paragraph_texts), \ + f"Expected 'Thank you, Paul.' to render as its own paragraph: {paragraph_texts}" + + # No sentence may collide with the closing parenthesis of a citation. + rendered_text = message_text.inner_text() + for collapsed in (")Admins can configure", ")For best results", ")Thank you, Paul."): + assert collapsed not in rendered_text, \ + f'Citation swallowed the whitespace before "{collapsed.lstrip(")")}"' + finally: + context.close() + browser.close()