Add the section API: conversations, learnings, documents, and recall - #114
Conversation
Add a specification for the memory section API to clarify its behavior and usage. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Document the proposed memory section API and its intended behavior to guide future implementation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add the section type definitions used to represent structured memory data. This provides the foundation for organizing sections within tinymemory. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a view implementation for accessing memory sections through the tinymemory API. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add the recall section to support retrieving stored memories from tinymemory. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Section views now own their section values, allowing callers to build custom sections inline without borrowing them. Recall operations clone sections when constructing views so they can continue using the original values. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce the sections module to organize tinymemory section handling and expose its functionality to the crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add typed surfaces for conversations, learnings, and documents with section-aware recall. Compose only mandatory families so the API works across every driver. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add tests for section behavior to improve coverage and guard against regressions. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add in-memory and null-provider test doubles covering section writes, reads, isolation, namespace validation, listing, recall ranking, truncation, and error handling. Verify the complete section API succeeds even when the provider retains no entries. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add tests covering section behavior in the tinymemory crate to verify its functionality. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the test memory implementation to satisfy the static name contract and allow panic-related lints in integration tests, where assertion failures are expected test behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Apply consistent Rust formatting to section view logic and related tests without changing behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Explain the typed section helpers, namespace conventions, cross-section recall behavior, and the distinction between text recall and document intake. This gives hosts guidance for using the memory surface across drivers. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Extend the end-to-end example to store learnings across multiple scopes and recall them through the section surface. Assert that both namespaces are searched and that the stored entries are found. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Keep the surrounding comment non-documenting so the module’s own `//!` documentation remains separate and its intra-doc links resolve in the correct scope. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Document the sections module and its typed surfaces for conversations, learnings, documents, and section-aware recall. Clarify that it composes mandatory families and works across all drivers. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the plan checklist to show all work is complete and change the specification status from draft to implemented. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Explain that invalid namespace errors may carry the namespace validator's own message, while namespace filter conflicts retain their specific error code. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Canonicalize section names when creating views so custom and built-in names refer to the same section. Validate sections before enumeration to report invalid namespaces as errors instead of incorrectly returning empty results. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Treat NaN and infinite scores as absent so invalid driver scores cannot outrank valid hits. Keep section implementation modules private while documenting source-scope behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add regression tests for section normalization, invalid namespaces, non-finite scores, recall limits, error propagation, malformed scopes, and list filtering. These cases protect the expected behavior across section and recall APIs. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Keep the supplied memory section unchanged instead of rebuilding it from its prefix, preserving the caller's section data in `SectionView`. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Ensure section views store the normalized memory section prefix so lookups use a consistent namespace. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Document deterministic score handling, size-based namespace traversal, section name normalization, and validation errors. Explain why recency ordering is deferred until drivers populate `last_updated`. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Make section accessors and view construction accept references, avoiding unnecessary clones while preserving section normalization and lookup behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update section tests to pass references to MemorySection values, matching the section API and avoiding unnecessary clones. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important Approval pendingCodeRabbit has no unresolved comments, but it has not reviewed the latest commit. Use the checkbox below to review the latest commit. CodeRabbit will approve the changes if it finds no blocking issues.
Comment |
Update the recall section implementation as needed. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the section test coverage to reflect the current memory behavior and guard against regressions. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the section test coverage to reflect the current behavior of the tinymemory implementation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Clarify the documentation for the tinymemory sections to make their usage and organization easier to understand. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformat the multiline `in_scope` invocation for consistent Rust style without changing its behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Restructure the section type definitions without changing their behaviour. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03a09568c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
How this change flows1 changed behaviour across 13 relationships. 6 surrounding behaviours are shown (60 graph nodes walked). 42 further behaviours left out to keep the diagram readable. flowchart LR
n0["main<br/>changed"]:::changed
n1["provider"]:::impacted
n2["as_ref"]:::impacted
n3["across_section"]:::impacted
n4["...on_asks_each_namespace_for_the_full_limit"]:::impacted
n5["...ion_merges_every_scope_and_ranks_by_score"]:::impacted
n6["recall"]:::impacted
n0 -->|calls| n2
n4 -->|calls| n1
n4 -->|tests| n1
n4 -->|calls| n3
n4 -->|tests| n3
n4 -->|calls| n6
n4 -->|tests| n6
n5 -->|calls| n1
n5 -->|tests| n1
n5 -->|calls| n3
n5 -->|tests| n3
n5 -->|calls| n6
n5 -->|tests| n6
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge. |
Clarify that cross-session recall is only allowed for the conversation section and is refused elsewhere to prevent content from being misrepresented under another namespace. Add the section-isolation behavior and acceptance criteria for this restriction. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
tinysweeper found nothing blocking. Approving.
$0.0218 · 95,135 in / 10,636 out · 62,807 cached (66%) · openrouter/openai/text-embedding-3-small, deepseek/deepseek-v4-flash, z-ai/glm-5.2 · 712 embedded
critique: $0.0012 · 14,300 in / 61 out · 0 cached (0%) · deepseek/deepseek-v4-flash
tests: $0.0127 · 43,520 in / 7,401 out · 34,264 cached (79%) · z-ai/glm-5.2
description: $0.0079 · 37,315 in / 3,174 out · 28,543 cached (76%) · z-ai/glm-5.2
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 860c67fa7a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Clarify that session-scoped and cross-session recall is only valid for conversation sections. Add a dedicated conflict message for across-section fan-out, which would duplicate augmented rows across scopes. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Expose the cross-session fan-out conflict constant through the sections module so consumers can access it from the public API. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Treat `session_id` like `cross_session` and validate against the normalized section name. Reject both options during cross-section fan-out to prevent duplicated or mislabeled conversational results. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reject session-scoped and cross-session options when section fan-out would repeat episodic augmentation across namespaces. Allow both options only for scoped recalls outside fan-out, including normalized conversation sections. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add coverage for normalized conversation aliases and session-scoped options in `in_scope`. Verify `across_section` rejects cross-session and session ID options for every section. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformat imports and a multiline method call to follow the project's Rust style without changing test behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the acceptance criteria to specify that cross-session and session-ID recall are limited to in-scope conversation sections and always refused across sections. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the section documentation link to use the fully qualified conversation variant while preserving the existing behavior explanation. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Clarify that cross-session and session-scoped recall are limited to normalized conversation sections for in-scope queries. Document that across-section queries reject both options to prevent duplicated episodic results and recommend using in-scope recall instead. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
What changed and why
Namespaces are the contract's only partitioning primitive, and
MemorySectionalready gave them a shape —conversation:thread-8f21,learning:rust-async,document:handbook— so the three content tiers mean the same thing to every host and every engine. What was missing was any reason to use it: the convention was documented and then left to discipline, so callers concatenated"conversation:" + idby hand, where a typo produces a valid, silently wrong namespace rather than an error.This adds
crates/tinymemory/src/sections/: typed surfaces for conversations, learnings and documents, plus a section-aware recall.conversations(),learnings()anddocuments()are named accessors over oneSectionViewparameterised by section;section()reaches the other four andCustom, so nothing in the vocabulary is second-class.Two findings that shaped the design
namespace: Noneon recall means two different things on two bundled drivers. It is documented as falling back toGLOBAL_NAMESPACE(tinymemory-bus/src/recall.rs:88) and the embedded engine implements exactly that, but the reference driver treats it as all namespaces (tinymemory-conformance/src/reference/mod.rs:176). The suite only ever asserts theSomecase, so nothing catches it.So there is no cross-namespace recall in the contract, and the obvious implementation of
across_section— recall unfiltered, then keep the hits whose namespace is in the section — would have passed its tests on the reference driver and returned nothing in production. It is instead a fan-out: enumeratenamespaces(), filter to the section, recall each by name, merge. That is also what the contract's ownlist_everythingdoes forlist(None, ..), for the same reason.The façade belongs in
crates/tinymemory, notcrates/tinymemory-api. The contract crate carries no[lints]table and is deliberately held byte-identical to itstinycortex-apiorigin, so new code required to document# Errorsbelongs where the lints actually run. The facade also already holds original non-contract code (registry/), its callers are hosts, and its existing dev-dependency ontinymemory-conformancesupplies the test double.Public API changes
Additive only. No trait, signature, driver, capability or error-variant change.
tinymemory::sections—Sections,SectionView,SectionRecall,SectionScope,SectionHits,MAX_SECTION_NAMESPACES,NAMESPACE_FILTER_CONFLICT.tinymemory::namespace. The facade re-exports the contract module by module and had omitted it, sotinymemory::namespace::Namespacedid not resolve — and this module is built entirely on it. (graph,versionandwireare omitted the same way; not load-bearing here, left alone.)Every call composes the mandatory families only, so the whole surface works on every driver: nothing to negotiate, no accessor that can return
None, no unsupported path. On a driver that retains nothing, every call succeeds and returns empty — asserted, not assumed.across_sectionstates its cost (1 + Nprovider calls, capped), its visit order (size descending, ties by namespace, so the cap is deterministic), its merge order (score descending, absent and non-finite scores last, ties by(namespace, key)), and thattruncatedmeans namespaces were skipped and never that hits hit the limit. It refusesopts.namespacewithNAMESPACE_FILTER_CONFLICTrather than silently overriding a caller's filter.Review fixes
A review pass caught a real bug, now fixed with a regression test that was confirmed to fail without it:
SectionViewstored itsMemorySectionun-normalised, whileNamespace::newnormalises throughfrom_prefix— with a comment saying it does so precisely to keepPartialEqin agreement. Sosection(Custom("conversation"))wrote toconversation:butscopes(),list_section()andacross_section()all reported the section empty. Fixed by normalising at construction. Relatedly, an invalid custom prefix made the enumerating calls returnOk(empty)where the addressed ones errored;scopes()now validates its section, so an unusable section is never mistaken for an empty one.Also from review: non-finite scores now rank with absent ones (
total_cmpputs+NaNabove+inf, which would have let oneNaNoutrank every real hit); submodules are private with one public path per item, matchingregistry; and section arguments are taken by reference throughout.Validation
All run from the repository root:
cargo fmt --all -- --check— cleancargo clippy --all-targets --all-features -- -D warnings— 0 warningscargo build --all-targets --all-features— cleancargo test --all-features— 0 failedcargo test --doc -p tinymemory— passes, including a new runnable doctestRUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features— cleancargo run -p tinymemory --features tinycortex --example tinycortex— the fan-out searched 2 namespaces and found 2 hits against the real embedded engineTests: 29 unit tests in
src/sections/test.rs(two doubles — one that can seed scores, since no public API can, andNullMemoryProvider) and 6 integration tests intests/sections.rsagainst the public API only. Both suites' doubles are shaped so a lost namespace pin fails rather than passes.Deliberately untested: behaviour against the remote adapters (Supermemory / Mem0 / Cognee), which need live endpoints — the façade composes only mandatory calls the conformance suite already covers.
SectionScope::last_updatedis carried but unasserted, since no bundled driver populates it.Raised, not fixed
The
namespace: Nonedivergence above is a genuine interchangeability hole — the exact claimtinymemory-conformanceexists to defend — and the fix is a suite assertion plus a contract sentence. That deserves its own spec rather than being smuggled in here; this design is built to not depend on it. Also unstated in the contract: whether a recall hit must populateMemoryEntry.namespace, and whetherscoreis comparable across calls. Both are noted in the spec's open questions.Docs
docs/specs/memory-section-api.md(behaviour, invariants, acceptance criteria, open questions),docs/plans/memory-section-api.md, a## The section surfacesection in the rootREADME.md, rustdoc on every public item, and thetinycortexexample extended to exercise the surface end to end.