Skip to content

experiment(index): embed only a chunk's path and file name, not the repeated metadata - #57

Closed
TrinaxCode wants to merge 1 commit into
mainfrom
ccr-fe2460a0-c0akjx
Closed

TrinaxCode wants to merge 1 commit into
mainfrom
ccr-fe2460a0-c0akjx

Conversation

@TrinaxCode

Copy link
Copy Markdown
Owner

Description

Experiment, draft: do not merge until the measurement below is filled in.

Every chunk is embedded together with nine lines of identical metadata (ids, absolute root, collection, pipeline version). On short chunks that boilerplate can blur the vector and make retrieval pick the wrong chunk. This change embeds only rel_path and file_name; the rest of the metadata stays on the node for filtering and citations. extractor_version goes to 3 so existing indexes rebuild.

Measurement (code Q&A, 4 TrinaxAI source files + 1 unanswerable question, real qwen3-embedding:0.6b and qwen3.5:4b)

recall@5 MRR nDCG citation precision answer terms no-answer correct
baseline (main) 1.0 0.7 0.926 0.6 1.0 1.0
this change pending pending pending pending pending pending

I will fill this in, and close the PR if it does not improve on the baseline.

Type of change

  • Bug fix
  • Other: retrieval-quality experiment

Checklist

  • Index tests pass locally; new test covers the embedded text.
  • Measured improvement (pending).

Generated by Claude Code

…epeated metadata

Every chunk was embedded with nine lines of identical metadata (ids, absolute root, collection,
pipeline version). That boilerplate can blur short chunks. Only rel_path and file_name are embedded
now; the rest stays on the node. extractor_version 3 makes existing indexes rebuild.

Pending a measured comparison on the code Q&A evaluation (baseline: recall@5 1.0, MRR 0.7,
citation precision 0.6).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZ4EYrCPbq3hhNsUVRT46K
Signed-off-by: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

Measured and rejected. Same corpus (4 TrinaxAI source files, 37 chunks, 5 questions), real qwen3-embedding:0.6b and qwen3.5:4b, same questions:

recall@5 MRR nDCG citation precision answer terms no-answer correct
baseline (main) 1.0 0.7 0.926 0.6 1.0 1.0
embed only path + file name 1.0 0.7 0.926 0.6 0.8 1.0

Retrieval is identical, so the repeated metadata is not what limits ranking on this corpus; the one-case drop in answer terms is within the noise of a single generation by a 4B model. The 0.6 citation precision comes from how many files get cited, not from the vectors. Closing without merging; no change to main. (Limitation: 37 chunks is a small corpus, so this does not rule out an effect on a large one.)


Generated by Claude Code

@TrinaxCode TrinaxCode closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants