mem (swarmcode.ai)

Local‑first, semantic version control. Tier‑0 gives you a reproducible, content‑addressed store. Tier‑1 layers typed, language‑aware objects (blocks/trees/patches). Tier‑2 adds Git‑like porcelain (staging, commits, branches, merges) that also attaches semantic context. Tier‑3 brings index/analytics, MCP, and IDE UX.

---

Project Status

• Current milestone: Tier‑2 polish — wiring proposals, stale-check, evaluation, and event flows into porcelain (in progress)
• What’s usable today: Tier‑0 ODB/pack/refs, Tier‑1 semantic extraction/diff/merge/weave, Tier‑2 porcelain (status/add/rm/mv/commit/reset/restore/show/update-index, branching, semantic merge, effects)
• What’s next: ship the proposal → evaluate → weave → commit flow and tighten event/effects integration
• What’s later: Tier‑3 analytics stack (SQLite index/search, health/hotspots, stale-context scoring, MCP daemon, VS Code extension)

---

Feature Matrix

Area	Capability	Status
Audit Trail & History	Shadow git repository (.mem-git) with conversation context tracking	✅ Done
	mem log with full conversation context, --show-diff flag	✅ Done
	mem history <file> showing WHY changes were made	✅ Done
	mem blame <file> with conversation attribution	✅ Done
Tier‑0: ODB / Blob Core	Content‑addressed store (BLAKE3‑256), zstd packs/idx, atomic writes	✅ Done
	CLI plumbing: init, hash-object, blob put/cat, refs, repack, gc, fsck, ls-objects	✅ Done
	On‑disk layout: .mem/objects, .mem/packs, .mem/refs, .mem/lock	✅ Done
Tier‑1: Semantic Layer	Canonical JSON envelopes (mem/block@1, …) with stable semantic IDs	✅ Done
	Block extraction (Tree‑sitter) — Python, JavaScript	✅ Done
	mem scan --lang {python|javascript} --path … [--write]	✅ Done
	Snapshot: mem/tree@1 (workspace → block list + path map)	✅ Done (P3)
	Semantic diff/patch (mem/patch@1)	✅ Done (P4)
	3‑way merge (AST first, text fallback)	✅ Done
	Weave to index (apply semantic patch to staging; optional materialize)	✅ Done
	Proposals + context fingerprint; evaluation/artifact hooks	⏳ In progress
Tier‑2: Git‑like Porcelain	Staging index (.mem/stage/index), status/add/rm/mv/ls-files, update-index plumbing	✅ Done (P9)
	File tree object (mem/ftree@1), write-tree / read-tree	✅ Done (P1)
	mem commit (porcelain): unified file tree + semantic tree + patch + effects	✅ Done
	Branch/switch/tag/reset/restore/show; 3-way merge (file-level + semantic assist)	✅ Done
	Effects propagation (mem/effects@1): refresh capsules, mark stale	✅ Done
	Proposals → weave → commit (--include-proposal, stale/eval gating)	⏳ In progress
Tier‑3: Index / Analytics / MCP / IDE	SQLite (WAL) index, FTS5 search, optional vectors	🔜 Planned
	Event spine → EWMA health/hotspots/heatmap; stale‑context checks	🔜 Planned
	MCP server (JSON‑RPC), VS Code extension (search/context/heatmap)	🔜 Planned
Sync	Local fs push/fetch of .mem (no network protocol yet)	🔜 Optional prototype

Legend: ✅ done · ⏳ in progress · 🔜 planned

---

Why “mem”?

mem aims to be Git for semantics: commits not only version bytes, they attach and propagate structured meaning (symbols, blocks, rationale, context fingerprints). Changes in one place can automatically refresh capsules and flag stale context in dependents.

---

Architecture at a Glance

Audit Trail (shadow git)
  ├─ .mem-git/ (shadow repository tracking all changes)
  ├─ checkpoint.json (conversation context per commit)
  ├─ mem log (chronological history with context)
  ├─ mem history <file> (per-file WHY + diffs)
  └─ mem blame <file> (line-level attribution)
Tier‑0  ODB/Blob Core
  ├─ Loose objects (.mem/objects/aa/…)
  ├─ Packs + idx (.mem/packs/pack-*.pack/.idx)
  ├─ Refs (.mem/refs/{blobs,anchors})
  └─ CLI plumbing (init, hash-object, repack, gc, fsck)
Tier‑1  Semantic Types (built on Tier‑0)
  ├─ mem/block@1  (function/class/method capsule inputs)
  ├─ mem/tree@1   (workspace snapshot of blocks)
  ├─ mem/patch@1  (AST-aware diff)
  ├─ mem/conflict@1, mem/blob@1, mem/artifact@1
  └─ Tree‑sitter extractors (Python, JS)
Tier‑2  Porcelain (Git‑like) — unified commit
  ├─ Staging index (.mem/stage/index)
  ├─ mem/ftree@1 (file tree from index)
  ├─ mem commit  → file_tree + semantic tree + patch + effects
  └─ Branch/merge/reset/restore + semantic‑assisted merges
Tier‑3  Index/Analytics/MCP/IDE
  ├─ SQLite/FTS index, hybrid search
  ├─ Events → health/hotspots, stale checks
  └─ MCP server + VS Code extension

---

Quick Start

🛠 Prereqs

• CMake ≥ 3.20, C11 compiler (clang/gcc), Ninja (recommended)

• Libraries: blake3, zstd, cmocka

  • macOS: brew install blake3 zstd cmocka ninja
  • Ubuntu: sudo apt install libblake3-dev libzstd-dev libcmocka-dev ninja-build

⚙️ Configure & Build

cmake -S . -B build -G Ninja
cmake --build build
ctest --test-dir build -V        # run tests

🚀 First Repo

./build/bin/mem init /tmp/mem-demo
cd /tmp/mem-demo

# Store and retrieve a blob
echo "hello" > hello.txt
../build/bin/mem hash-object --write hello.txt
../build/bin/mem blob cat <OID> > roundtrip.txt
diff -u hello.txt roundtrip.txt  # (no output = identical)

# Pack, verify, garbage-collect
../build/bin/mem repack
../build/bin/mem fsck
../build/bin/mem gc --dry-run

🔎 Extract Code Blocks

# Print Python blocks
../build/bin/mem scan --lang python --path path/to/file.py

# Persist JavaScript blocks into .mem
../build/bin/mem scan --lang javascript --path src/ --write

scan emits mem/block@1 objects with stable semantic IDs; --write stores them in the ODB.

---

🕰️ Audit Trail & Conversation Context

mem provides complete traceability from AI agent conversations to code changes through a shadow git repository (.mem-git) that tracks every tool use with full context.

Shadow Repository

Every workspace automatically maintains a .mem-git directory containing:

• A complete git repository tracking all file changes
• checkpoint.json in each commit with conversation metadata
• User queries, assistant responses, tool parameters, and session info

Commands

Commit plumbing

The low-level history plumbing mirrors familiar Git workflows while emitting mem/commit@1 objects:

• mem commit-tree <tree-oid> [-p <parent> ...] [-m <msg>] [--patch <patch-oid> ...] writes a commit object and prints its OID.
• mem update-ref <ref> <oid> atomically updates .mem/refs/heads/<ref> using lockfiles and fsync.
• mem rev-parse <ref|oid> resolves ref names and abbreviated IDs via mem_refs_resolve.
• mem log [--limit N] [<ref>] walks parent links and prints commit metadata (file_tree, tree, patch, effects_ref, message).

Use these commands to script Tier-2 porcelain flows or integration tests before the higher-level mem commit command lands.

mem log [options]

Show chronological history of all changes with conversation context.

# Basic log (compact, no diffs)
mem log --max-count 10

# Show inline diffs for file changes
mem log --show-diff --max-count 5

# Filter by session
mem log --session <session-id>

# Show only commit hashes and titles
mem log --oneline

Output includes:

• 💬 User queries that triggered changes
• 🔧 Tools used (Edit, Write, Bash, etc.)
• 📋 Full conversation context (User ↔ Assistant exchanges)
• 🔗 Session ID and PPID for grouping related changes
• 📝 File changes with optional color-coded diffs

mem history <file>

Show complete modification history for a specific file with full conversation context.

# See all changes to a file and WHY they were made
mem history src/main.py

# Trace the evolution of a configuration file
mem history config.json

Each entry shows:

• User's request: The original query that led to the change
• Assistant's reasoning: What the AI was thinking when making the change
• Tool details: Exact parameters passed to the tool
• Session context: When and in what context the change was made
• Code diff: Exact lines added/removed/modified

Example output:

History of src/tools/mem_cli.c

a1b2c3d4 - 5 minutes ago feat: Add new command (alice)
│ 💬 User: "add a history command to show file changes"
│ 🔧 Tool: Edit
│
│ 📋 Context:
│   User: add a history command to show file changes
│   Assistant: I'll implement the history command by adding...
│
│ 🔗 Session: abc123 (PPID: 12345)
│
│ 📝 Changes to src/tools/mem_cli.c:
│ +static int cmd_history(int argc, char **argv) {
│ +    // implementation
│ +}

This provides end-to-end traceability: you can see exactly what user request led to each code change, understand the AI's reasoning, and view the exact modifications.

mem blame <file>

Show line-by-line attribution with conversation context (similar to git blame).

mem blame src/server.js

Shows which user query and tool call last modified each line.

Use Cases

1. Understanding Changes: See why a file was modified and what conversation led to it
2. Debugging: Trace a bug back to the original user request that introduced it
3. Code Review: Understand the intent behind changes, not just the diff
4. Audit Trail: Complete record of all AI-assisted modifications
5. Knowledge Transfer: New team members can see the reasoning behind code

Integration with Rapala

When used with the mem-intelligent-committer hook in Rapala (AI coding assistant):

• Every Edit/Write operation automatically creates a shadow git commit
• Conversation context is extracted and stored in checkpoint.json
• mem history and mem log provide instant access to the decision trail
• No manual intervention needed—fully automatic

---

CLI Cheat Sheet (current)

Tier‑0 (shipped)

Command	What it does
mem init [PATH]	Create .mem repo at PATH
mem hash-object <file> [--write]	Print BLAKE3 OID (and store if --write)
mem blob put/cat	Store or retrieve raw bytes
mem refs set/ls/del	Manage refs (.mem/refs/{blobs,anchors})
mem repack	Pack loose objects into .pack/.idx
mem gc [--dry-run]	Prune unreachable loose objects (packs immutable)
mem fsck	Verify loose + packs + idx integrity
mem ls-objects	List loose/packed object OIDs

Audit Trail & History (shipped)

Command	What it does
mem log [--max-count N] [--show-diff]	Show chronological history with conversation context
mem history <file>	Show complete modification history for a file with WHY it changed
mem blame <file>	Line-by-line attribution with conversation context

Tier‑1 (shipped)

Command	What it does
mem scan --lang {python|javascript} --path <file|dir> [--write]	Extract blocks to stdout or store them as mem/block@1
mem snapshot --langs py,js --root <dir> [--write]	Construct mem/tree@1 (semantic workspace snapshot)
mem diff --from <treeA> --to <treeB> [--write]	Generate mem/patch@1 (AST-aware diff)
mem weave <patch> --base-tree <tree>	Apply semantic patch to the staging index (and optionally materialize)
mem query [filters] [--format json|oids]	List mem/envelope@1 objects (packed and loose) with tag/attr filters

Tier‑2 (shipped)

Command	What it does
mem status/add/rm/mv/ls-files	Manage the staging index
mem write-tree / mem read-tree	Persist or restore mem/ftree@1 snapshots
mem commit [-m] [...]	Unified commit: file tree + semantic tree + patch + effects
mem branch/switch/tag/reset/restore/merge	Branching, checkout, reset, restore, semantic-assisted merge
mem effects show <commit>	Display commit side effects (capsule refresh, stale blocks)

Tier‑2 polish (in progress)

Command	Intent
mem propose ...	Store semantic proposals with rationale & fingerprints
mem stale-check ...	Report stale-context score and policy gates for proposals
mem evaluate ...	Run configured eval/policy commands and capture artifacts
mem events tail	Stream recent mem/event@1 entries

Tier‑3 (planned)

Command / Tool	Planned capability
mem index scan [--since ...]	Ingest ODB objects into the SQLite analytics store
mem index stats [--timings]	Report indexed row counts and last scan performance
mem embed import --csv ...	Load block embeddings for hybrid/vector search
mem search "query" [--hybrid ...]	Ranked FTS (and optional ANN) search over semantic blocks
mem health compute / mem hotspots / mem heatmap	Compute and surface health/hotspot analytics
mem stale-check --json (extended)	Full stale-context scoring backed by SQLite + embeddings
memd (JSON-RPC daemon)	Agent/IDE bridge exposing search, context, health APIs

---

Design Guarantees

• Deterministic OIDs: same bytes → same 64‑hex ID (BLAKE3‑256).
• Atomic updates: write‑temp → fsync file & dir → rename.
• Immutability: objects are append‑only; packs are immutable.
• Safe parsing: all on‑disk formats have magic/version, big‑endian length fields, bounds checks, and CRCs.
• No foot‑guns by default: GC never removes packed content; refs are length/hex validated.

---

Roadmap & Checklists

Tier‑0 — Object Database / Blob Core ✅

• Loose objects (magic/version, CRC, zstd frames)
• Packs + idx (fanout, OID‑sorted, mmap reader)
• Refs/anchors + HEAD
• CLI plumbing (hash-object, blob, repack, gc, fsck)
• Test suite (hex/hash/loose/pack/fsck)

Tier‑1 — Semantic Layer ✅

• Canonical JSON envelopes (mem/block@1, …)
• Block extraction (Tree‑sitter Python, JS)
• Stable semantic IDs (blake3(canonical_source+kind+qualname))
• mem/tree@1 (snapshot) (P3)
• mem/patch@1 (AST diff) (P4)
• 3‑way merge (AST → text fallback)
• weave --to-index + optional materialize

Tier‑2 — Git‑like Porcelain ✅ (core)

• Staging index (.mem/stage/index) + status/add/rm/mv/ls-files
• mem/ftree@1, write-tree, read-tree
• Unified mem commit (file tree + semantic tree + patch + effects)
• Branch/switch/tag/reset/restore, semantic-assisted merge
• Effects propagation (mem/effects@1)

Tier‑2 — Polishing ⏳

• mem propose CLI & context fingerprints
• mem stale-check scoring + policy gates
• mem evaluate (policy/eval runner + artifacts)
• Event emission plumbing (mem events tail, DB ingestion)

Tier‑3 — Index / Analytics / MCP / IDE (Later)

• SQLite backbone (WAL, migrations, prepared-statement cache)
• Indexer ingestion (mem index scan, mem index stats) + FTS search
• Embeddings + hybrid/vector search (mem embed import, ANN support)
• Health analytics (mem health compute, mem hotspots, mem heatmap), stale-context scoring
• MCP daemon (memd) and JSON-RPC tools (mem searchBlocks, mem.healthReport, etc.)
• VS Code extension (search, context drawer, health view)
• Performance & security hardening passes

---

Safety & Hardening (dev defaults)

• Debug builds: -Wall -Wextra -Werror -Wpedantic -Wshadow -fsanitize=address,undefined
• Release builds: -fstack-protector-strong -D_FORTIFY_SOURCE=2
• I/O hygiene: openat + O_NOFOLLOW, strict 64‑hex OID validation, path normalization (no ..), temp+fsync+rename
• Format limits: max object size/window for zstd; bound checks on all offsets
• Tests to watch: corruption causes fsck to fail; whitespace‑only changes don’t change semantic IDs

Licenses

• Project: Private for Swarm Code
• Vendored: retain upstream licenses under third_party/*/LICENSE

---

Appendix: Object Types (Tier‑1 snapshot)

• mem/block@1 — semantic block (function/class/method)
• mem/tree@1 — workspace snapshot of blocks
• mem/patch@1 — AST‑aware diff
• mem/conflict@1 — merge conflict record
• mem/blob@1, mem/artifact@1 — metadata descriptors for raw blobs/artifacts
• (Tier‑2) mem/ftree@1 — file tree (bytes) from the staging index
• (Tier‑2) mem/effects@1 — commit side‑effects (capsules updated, stale marked)

---

North star: one mem commit that’s both Git‑strong (bytes) and agent‑aware (semantics). Everything in this repo marches toward that.