fix(server): hide qwen transport markers from semantic responses

[HaileyStorm]

Summary

• suppress Qwen transport and internal multimodal markers at the protocol-neutral semantic generation boundary
• run suppression only after reasoning and tool parsing, keeping parser tags, grounding markers, FIM markers, and tool arguments intact
• preserve matching quoted/code literals while making marker matching chunk-safe and event-order-safe across streaming tool calls
• leave raw /generate, legacy completions, non-Qwen semantic output, and offline token IDs unchanged

Closes #265.

Why this layer

The Qwen detokenizer must keep protocol tags visible long enough for reasoning and tool parsers to consume them. Filtering in the tokenizer would also alter raw-generation and offline compatibility surfaces. generation.py is the shared semantic waist used by OpenAI chat, Anthropic Messages, and Responses, so the cleanup happens once after parsing and before those adapters format their wires.

The filter is request-local. It holds only an ambiguous marker prefix or the suffix of a quote after a complete hidden marker. Structured tool events encountered during that ambiguity are retained in the same sequence, preventing both marker leaks and content/tool reordering. An unmatched quote is sanitized at end-of-stream; a matching quote proves the marker spelling was deliberate literal content.

Validation

Original symptom: observed with a local Qwen3.8 Flash-Next derivative on an NVIDIA RTX 5090, then minimized to deterministic CPU-only server tests. This PR makes no performance claim and does not require model weights or GPU execution.

Ubuntu Linux, GPU visibility explicitly disabled:

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=python CUDA_VISIBLE_DEVICES='' HIP_VISIBLE_DEVICES='' \
/home/hailey/AI/llm/src/freetoken-flash-next-9ef36513/.venv/bin/python -m pytest -q -p no:cacheprovider \
tests/server/test_qwen_semantic_special_tokens.py \
tests/server/test_streaming_model_matrix.py \
tests/server/test_generation_accounting.py \
tests/server/test_openai_api.py

Result: 198 passed, 1 warning in 2.89s. The warning is the existing FastAPI/Starlette TestClient deprecation warning.

Also passed:

• py_compile for both changed Python files
• git diff --check origin/main...HEAD
• independent closure review, including 317,449 bounded text/chunk/event probes

Compatibility tests

The regression suite explicitly covers:

• all hidden markers split at every character boundary
• stream/buffer parity for reasoning, content, and structured tool calls
• overlapping and disproved marker prefixes without event reordering
• closed, escaped, typographic, and unterminated quotes
• preserved grounding and FIM markers
• non-Qwen semantic output
• raw /generate, legacy completion output, and offline token IDs