Skip to content

Add the section API: conversations, learnings, documents, and recall - #114

Merged
senamakel merged 51 commits into
mainfrom
section-memory-api
Aug 29, 2026
Merged

Add the section API: conversations, learnings, documents, and recall#114
senamakel merged 51 commits into
mainfrom
section-memory-api

Conversation

@senamakel

Copy link
Copy Markdown
Member

What changed and why

Namespaces are the contract's only partitioning primitive, and MemorySection already gave them a shape — conversation:thread-8f21, learning:rust-async, document:handbook — so the three content tiers mean the same thing to every host and every engine. What was missing was any reason to use it: the convention was documented and then left to discipline, so callers concatenated "conversation:" + id by hand, where a typo produces a valid, silently wrong namespace rather than an error.

This adds crates/tinymemory/src/sections/: typed surfaces for conversations, learnings and documents, plus a section-aware recall.

let sections = Sections::new(provider.as_ref());

sections.conversations().put("thread-8f21", "turn-1", text, category, None, taint).await?;
let topics = sections.learnings().scopes().await?;
let hits = sections.recall().across_section(&MemorySection::Learning, "async", 10, &opts, None).await?;

conversations(), learnings() and documents() are named accessors over one SectionView parameterised by section; section() reaches the other four and Custom, so nothing in the vocabulary is second-class.

Two findings that shaped the design

namespace: None on recall means two different things on two bundled drivers. It is documented as falling back to GLOBAL_NAMESPACE (tinymemory-bus/src/recall.rs:88) and the embedded engine implements exactly that, but the reference driver treats it as all namespaces (tinymemory-conformance/src/reference/mod.rs:176). The suite only ever asserts the Some case, so nothing catches it.

So there is no cross-namespace recall in the contract, and the obvious implementation of across_section — recall unfiltered, then keep the hits whose namespace is in the section — would have passed its tests on the reference driver and returned nothing in production. It is instead a fan-out: enumerate namespaces(), filter to the section, recall each by name, merge. That is also what the contract's own list_everything does for list(None, ..), for the same reason.

The façade belongs in crates/tinymemory, not crates/tinymemory-api. The contract crate carries no [lints] table and is deliberately held byte-identical to its tinycortex-api origin, so new code required to document # Errors belongs where the lints actually run. The facade also already holds original non-contract code (registry/), its callers are hosts, and its existing dev-dependency on tinymemory-conformance supplies the test double.

Public API changes

Additive only. No trait, signature, driver, capability or error-variant change.

  • New: tinymemory::sectionsSections, SectionView, SectionRecall, SectionScope, SectionHits, MAX_SECTION_NAMESPACES, NAMESPACE_FILTER_CONFLICT.
  • New re-export: tinymemory::namespace. The facade re-exports the contract module by module and had omitted it, so tinymemory::namespace::Namespace did not resolve — and this module is built entirely on it. (graph, version and wire are omitted the same way; not load-bearing here, left alone.)

Every call composes the mandatory families only, so the whole surface works on every driver: nothing to negotiate, no accessor that can return None, no unsupported path. On a driver that retains nothing, every call succeeds and returns empty — asserted, not assumed.

across_section states its cost (1 + N provider calls, capped), its visit order (size descending, ties by namespace, so the cap is deterministic), its merge order (score descending, absent and non-finite scores last, ties by (namespace, key)), and that truncated means namespaces were skipped and never that hits hit the limit. It refuses opts.namespace with NAMESPACE_FILTER_CONFLICT rather than silently overriding a caller's filter.

Review fixes

A review pass caught a real bug, now fixed with a regression test that was confirmed to fail without it:

SectionView stored its MemorySection un-normalised, while Namespace::new normalises through from_prefix — with a comment saying it does so precisely to keep PartialEq in agreement. So section(Custom("conversation")) wrote to conversation: but scopes(), list_section() and across_section() all reported the section empty. Fixed by normalising at construction. Relatedly, an invalid custom prefix made the enumerating calls return Ok(empty) where the addressed ones errored; scopes() now validates its section, so an unusable section is never mistaken for an empty one.

Also from review: non-finite scores now rank with absent ones (total_cmp puts +NaN above +inf, which would have let one NaN outrank every real hit); submodules are private with one public path per item, matching registry; and section arguments are taken by reference throughout.

Validation

All run from the repository root:

  • cargo fmt --all -- --check — clean
  • cargo clippy --all-targets --all-features -- -D warnings — 0 warnings
  • cargo build --all-targets --all-features — clean
  • cargo test --all-features — 0 failed
  • cargo test --doc -p tinymemory — passes, including a new runnable doctest
  • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features — clean
  • cargo run -p tinymemory --features tinycortex --example tinycortex — the fan-out searched 2 namespaces and found 2 hits against the real embedded engine

Tests: 29 unit tests in src/sections/test.rs (two doubles — one that can seed scores, since no public API can, and NullMemoryProvider) and 6 integration tests in tests/sections.rs against the public API only. Both suites' doubles are shaped so a lost namespace pin fails rather than passes.

Deliberately untested: behaviour against the remote adapters (Supermemory / Mem0 / Cognee), which need live endpoints — the façade composes only mandatory calls the conformance suite already covers. SectionScope::last_updated is carried but unasserted, since no bundled driver populates it.

Raised, not fixed

The namespace: None divergence above is a genuine interchangeability hole — the exact claim tinymemory-conformance exists to defend — and the fix is a suite assertion plus a contract sentence. That deserves its own spec rather than being smuggled in here; this design is built to not depend on it. Also unstated in the contract: whether a recall hit must populate MemoryEntry.namespace, and whether score is comparable across calls. Both are noted in the spec's open questions.

Docs

docs/specs/memory-section-api.md (behaviour, invariants, acceptance criteria, open questions), docs/plans/memory-section-api.md, a ## The section surface section in the root README.md, rustdoc on every public item, and the tinycortex example extended to exercise the surface end to end.

senamakel and others added 28 commits August 29, 2026 19:36
Add a specification for the memory section API to clarify its behavior and usage.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Document the proposed memory section API and its intended behavior to guide future implementation.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add the section type definitions used to represent structured memory data. This provides the foundation for organizing sections within tinymemory.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a view implementation for accessing memory sections through the tinymemory API.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add the recall section to support retrieving stored memories from tinymemory.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Section views now own their section values, allowing callers to build custom sections inline without borrowing them. Recall operations clone sections when constructing views so they can continue using the original values.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce the sections module to organize tinymemory section handling and expose its functionality to the crate.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add typed surfaces for conversations, learnings, and documents with section-aware recall. Compose only mandatory families so the API works across every driver.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add tests for section behavior to improve coverage and guard against regressions.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add in-memory and null-provider test doubles covering section writes, reads, isolation, namespace validation, listing, recall ranking, truncation, and error handling. Verify the complete section API succeeds even when the provider retains no entries.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add tests covering section behavior in the tinymemory crate to verify its functionality.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the test memory implementation to satisfy the static name contract and allow panic-related lints in integration tests, where assertion failures are expected test behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Apply consistent Rust formatting to section view logic and related tests without changing behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Explain the typed section helpers, namespace conventions, cross-section recall behavior, and the distinction between text recall and document intake. This gives hosts guidance for using the memory surface across drivers.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Extend the end-to-end example to store learnings across multiple scopes and recall them through the section surface. Assert that both namespaces are searched and that the stored entries are found.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Keep the surrounding comment non-documenting so the module’s own `//!` documentation remains separate and its intra-doc links resolve in the correct scope.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Document the sections module and its typed surfaces for conversations, learnings, documents, and section-aware recall. Clarify that it composes mandatory families and works across all drivers.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the plan checklist to show all work is complete and change the specification status from draft to implemented.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Explain that invalid namespace errors may carry the namespace validator's own message, while namespace filter conflicts retain their specific error code.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Canonicalize section names when creating views so custom and built-in names refer to the same section. Validate sections before enumeration to report invalid namespaces as errors instead of incorrectly returning empty results.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Treat NaN and infinite scores as absent so invalid driver scores cannot outrank valid hits. Keep section implementation modules private while documenting source-scope behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add regression tests for section normalization, invalid namespaces, non-finite scores, recall limits, error propagation, malformed scopes, and list filtering. These cases protect the expected behavior across section and recall APIs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Keep the supplied memory section unchanged instead of rebuilding it from its prefix, preserving the caller's section data in `SectionView`.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Ensure section views store the normalized memory section prefix so lookups use a consistent namespace.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Document deterministic score handling, size-based namespace traversal, section name normalization, and validation errors. Explain why recency ordering is deferred until drivers populate `last_updated`.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Make section accessors and view construction accept references, avoiding unnecessary clones while preserving section normalization and lookup behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update section tests to pass references to MemorySection values, matching the section API and avoiding unnecessary clones.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 29, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-29T18:27:10.131183Z 3e2d8fa New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown

Important

Approval pending

CodeRabbit has no unresolved comments, but it has not reviewed the latest commit.

Use the checkbox below to review the latest commit. CodeRabbit will approve the changes if it finds no blocking issues.

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

senamakel and others added 7 commits August 29, 2026 20:49
Update the recall section implementation as needed.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the section test coverage to reflect the current memory behavior and guard against regressions.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the section test coverage to reflect the current behavior of the tinymemory implementation.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Clarify the documentation for the tinymemory sections to make their usage and organization easier to understand.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformat the multiline `in_scope` invocation for consistent Rust style without changing its behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Restructure the section type definitions without changing their behaviour.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03a09568c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/tinymemory/src/sections/recall.rs Outdated
Comment thread crates/tinymemory/src/sections/recall.rs Outdated
@tinysweeper

tinysweeper Bot commented Aug 29, 2026

Copy link
Copy Markdown

How this change flows

1 changed behaviour across 13 relationships. 6 surrounding behaviours are shown (60 graph nodes walked). 42 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["main<br/>changed"]:::changed
  n1["provider"]:::impacted
  n2["as_ref"]:::impacted
  n3["across_section"]:::impacted
  n4["...on_asks_each_namespace_for_the_full_limit"]:::impacted
  n5["...ion_merges_every_scope_and_ranks_by_score"]:::impacted
  n6["recall"]:::impacted
  n0 -->|calls| n2
  n4 -->|calls| n1
  n4 -->|tests| n1
  n4 -->|calls| n3
  n4 -->|tests| n3
  n4 -->|calls| n6
  n4 -->|tests| n6
  n5 -->|calls| n1
  n5 -->|tests| n1
  n5 -->|calls| n3
  n5 -->|tests| n3
  n5 -->|calls| n6
  n5 -->|tests| n6
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading

Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.

tinysweeper 0.1.0

@tinysweeper tinysweeper Bot added priority: p2 Soon. Real but survivable — a rough edge, a gap, a thing that will bite later. and removed priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. labels Aug 29, 2026
Clarify that cross-session recall is only allowed for the conversation section and is refused elsewhere to prevent content from being misrepresented under another namespace. Add the section-isolation behavior and acceptance criteria for this restriction.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

             $0.0218 · 95,135 in / 10,636 out · 62,807 cached (66%) · openrouter/openai/text-embedding-3-small, deepseek/deepseek-v4-flash, z-ai/glm-5.2 · 712 embedded
critique:    $0.0012 · 14,300 in / 61 out     · 0 cached (0%)       · deepseek/deepseek-v4-flash
tests:       $0.0127 · 43,520 in / 7,401 out  · 34,264 cached (79%) · z-ai/glm-5.2
description: $0.0079 · 37,315 in / 3,174 out  · 28,543 cached (76%) · z-ai/glm-5.2

Comment thread crates/tinymemory/src/sections/recall.rs
Comment thread crates/tinymemory/src/sections/recall.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 860c67fa7a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/tinymemory/src/sections/view.rs
Comment thread crates/tinymemory/src/sections/recall.rs Outdated
senamakel and others added 11 commits August 29, 2026 21:18
Clarify that session-scoped and cross-session recall is only valid for conversation sections. Add a dedicated conflict message for across-section fan-out, which would duplicate augmented rows across scopes.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Expose the cross-session fan-out conflict constant through the sections module so consumers can access it from the public API.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Treat `session_id` like `cross_session` and validate against the normalized section name. Reject both options during cross-section fan-out to prevent duplicated or mislabeled conversational results.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reject session-scoped and cross-session options when section fan-out would repeat episodic augmentation across namespaces. Allow both options only for scoped recalls outside fan-out, including normalized conversation sections.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add coverage for normalized conversation aliases and session-scoped options in `in_scope`. Verify `across_section` rejects cross-session and session ID options for every section.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformat imports and a multiline method call to follow the project's Rust style without changing test behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the acceptance criteria to specify that cross-session and session-ID recall are limited to in-scope conversation sections and always refused across sections.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Update the section documentation link to use the fully qualified conversation variant while preserving the existing behavior explanation.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Clarify that cross-session and session-scoped recall are limited to normalized conversation sections for in-scope queries. Document that across-section queries reject both options to prevent duplicated episodic results and recommend using in-scope recall instead.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@tinysweeper tinysweeper Bot added priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. and removed priority: p2 Soon. Real but survivable — a rough edge, a gap, a thing that will bite later. labels Aug 29, 2026
@senamakel
senamakel merged commit 68b5a27 into main Aug 29, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant