Python context layer: fold source prose into retrieval - #285
Open
jat255 wants to merge 3 commits into
Open
Conversation
jat255
added
py
jat255
force-pushed
the
jat255/context-layer-augment
branch
from
September 7, 2026 00:43
2d37841 to
e012e80
Compare
|
Preview deployed to Connect ( Deployed from commit c76ef72. |
|
Preview deployed to Connect ( Deployed from commit c76ef72. |
jat255
force-pushed
the
jat255/context-layer-augment
branch
from
September 7, 2026 01:00
e012e80 to
b076ab2
Compare
An agent's sources describe themselves, and that prose is worth retrieving. augment_context_layer() appends each source's dictionary chunks and hands back a new layer, so a dictionary alone is searchable context even when the caller passed no context files at all. The new layer is always a new object. Source enrichment belongs to the agent that owns the sources; mutating the caller's layer would leak one agent's sources into the next agent built from the same layer. With nothing to add the argument comes back unchanged, None included, so an agent with neither context nor a dictionary has no layer rather than an empty one. The milestone's acceptance check is a test rather than a one-off script, and asserts equality rather than a match: the glossary term comes back whole, not as a fragment. pkg-r also folds in a warehouse's own semantic models here. This package has no surface for those yet, so that half is a comment at the point where it will go.
Both packages fold a source's dictionary prose into an agent's context layer, and each was checking that on its own. Add an augment_context_layer section to the shared context_layer fixture and a runner in each suite. Each case lists one entry per source, so the cases cover where the sources' chunks land relative to the caller's own documents, that every source contributes in the order given, and that a source carrying no dictionary is skipped rather than counted. An absent layer, an empty layer, and a dictionary whose prose is all empty are separate cases, because they are separate outcomes. Whether the returned layer is a new object with a fresh index stays out of the fixture and out of R. It is what keeps one agent's sources from leaking into the next, but it is not observable text, so it is asserted in the Python suite only. Two R tests in test-data-dictionary.R covered the same ground more loosely, one asserting only that the document count grew. The fixture runner replaces both.
jat255
force-pushed
the
jat255/context-layer-augment
branch
from
September 7, 2026 01:07
b076ab2 to
7c9a3f8
Compare
jat255
added
r
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last of four PRs for M4, the Python context layer (kata
c8a0). Stacked on #284, which adds the chunks this folds in.augment_context_layer(layer, sources)appends each source's dictionary chunks and returns a new layer, so a dictionary alone is searchable context even when the caller passed no context files. It is internal, as it is inpkg-r; the agent constructor in M5 is what calls it.The returned layer is always a new object with a fresh index. Source enrichment belongs to the agent that owns the sources, so mutating the caller's layer would leak one agent's sources into the next agent built from the same layer. With nothing to add the argument comes back unchanged,
Noneincluded, so an agent with neither context files nor dictionary prose gets no layer rather than an empty one.One deliberate gap.
pkg-ralso folds in each source's warehouse semantic models here, the Snowflake semantic views and Databricks metric views. The python side package has no surface for those yet (tracked locally as katagcgj, and notes that porting the reader alone would produce records nothing consumes), so that half is a comment at the point where it will go rather than code.R changes
@simonpcouch
Tests-only.
The behaviour this PR adds in Python already existed in R, and each package was testing it separately.
tests/shared/context_layer.jsongains anaugment_context_layersection andtest-context-layer.Rgains a runner for it. The cases cover where source chunks land relative to the caller's documents, source ordering, skipping a dictionary-less source, and the distinct outcomes for an absent layer, an empty layer, and all-empty prose.The runner replaces two hand-written tests in
test-data-dictionary.R, with a comment at the deletion site pointing at the fixture. "dictionary prose is searchable via the context layer" stays, because it exercises BM25 ranking, which is engine-specific. The fresh-index assertion is Python-only, as the fixture notes: R holds documents in an R6 object, where the equivalent would be object identity rather than behaviour.The runner is guarded against being vacuous (
expect_gt(length(cases), 0)), and I verified the three-source case really builds three sources with the dictionary-less middle one skipped.test-context-layer.Rpasses at 33;test-data-dictionary.R,test-definitions.R, andtest-citations.Ralso pass. Not the full R suite, for the #268 hang.pkg-r/tests/testthat/fixtures/shared/context_layer.jsonis generated byscripts/sync-shared-fixtures.shand needs no review.