Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
267b27d
test(dreamer): capture native startup and input-budget regressions
joshyorko Oct 2, 2026
ed16ccc
style(tests): format Dreamer regression checkpoint
joshyorko Oct 2, 2026
45e3c1f
test(dreamer): reproduce full-wire exhaustion and native rejection
joshyorko Oct 2, 2026
565c407
fix(native): preserve readiness and bound scheduled Dreamer input
joshyorko Oct 2, 2026
eefc7ea
test(native): checkpoint managed-container readiness regression
joshyorko Oct 2, 2026
4972855
fix(native): preserve usable startup and bounded Dreamer previews
joshyorko Oct 2, 2026
0106eda
Merge remote-tracking branch 'origin/master' into fix/issue-260-dream…
joshyorko Oct 14, 2026
3bfe494
fix(test): qualify lifecycle checks under restricted procfs
joshyorko Oct 14, 2026
8c94d1a
fix(container): honor configured database path
joshyorko Oct 14, 2026
e2d9f66
fix: close bounded scheduler and container review gaps
joshyorko Oct 14, 2026
8314950
fix: address latest PR 261 review findings
joshyorko Oct 14, 2026
26713be
fix: preserve late scheduler frontier evidence
joshyorko Oct 14, 2026
4d9424f
fix: preserve unbounded simulated scheduler frontier
joshyorko Oct 14, 2026
01b24b2
fix: preserve completed scheduler frontiers and status
joshyorko Oct 14, 2026
e5bcd6b
fix: preserve scheduler mutation and rowid frontiers
joshyorko Oct 14, 2026
720f0f1
fix: make dream scheduler frontiers race-safe
joshyorko Oct 14, 2026
a97105e
fix: preserve scheduler mutations across frontiers
joshyorko Oct 14, 2026
49ea61e
fix: preserve scheduler frontiers across rowid reuse
joshyorko Oct 14, 2026
7717d40
Make scheduler rowid upgrade atomic
joshyorko Oct 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,39 @@ gate.
- Before merge or handoff, report the reviewed head, a link to the completed
connector review, unresolved findings, and required check status. Review
completion does not itself authorize merging or replace other merge gates.

## Isolated Linux lifecycle validation

Native lifecycle checks start detached fixture daemons. Run them with a reaping
PID 1 or the repository's child-subreaper launcher:

```sh
python3 scripts/run-with-child-reaper.py cargo test --test cli_smoke
python3 scripts/run-with-child-reaper.py scripts/v0.1-release-gate.sh
```

Use the launcher only for isolated synthetic checks. When the command exits or
is interrupted, it terminates and reaps the command's remaining descendants;
do not wrap a real operator service or `--include-dogfood` with it.

On restricted procfs mounts, the launcher falls back to scanning
`/proc/<pid>/status` for direct children. Process records that disappear or
cannot be read due to permissions are skipped; the focused test
`test_proc_status_fallback_skips_unreadable_status_entries` covers that case.

A non-reaping PID 1 can leave an exited daemon as a zombie. Linux `kill(pid, 0)`
still reports that PID as present, so `src/native_runtime.rs::process_alive`
cannot establish clean shutdown in that environment. Inspect `/proc/<pid>/status`
and PID 1 before classifying such a fixture failure as a startup regression.
Do not waive the shutdown assertion or change production lifecycle behavior to
make the test environment pass.

The agent that starts a release gate must keep ownership until it has the
command's exit receipt. Shared log files do not transfer ownership of an
agent-local execution session; do not hand off a still-running gate as verified.

Validate the launcher with
`python3 -B -m unittest discover -s tests -p test_child_reaper.py -v`.
Its `run` function owns the main command's exit status separately from adopted
child reaping and bounds signal cleanup. The canonical release gate includes
these launcher checks.
2 changes: 2 additions & 0 deletions docs/dakota-native-codex-instance.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ md() {
PATH="$PATH" \
TMPDIR="${TMPDIR:-/tmp}" \
CODEX_MEMORYD_HOME="$MEMORYD_CODEX_HOME" \
CODEX_MEMORYD_PROFILE=personal \
CODEX_MEMORYD_WORKSPACE=dakota \
codex-memoryd "$@"
}
```
Expand Down
5 changes: 4 additions & 1 deletion docs/dreamer-consolidation.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ rejects the transaction as stale.

A scheduled run that fills its bounded input window persists a per-stream
continuation cursor, including timestamp ties, rather than skipping to the run's
end. Later ticks consume the remaining pages before advancing the time watermark.
end. Every source stream uses that cursor's fixed upper frontier. Later ticks
consume the remaining pages before advancing the time watermark; a new time band
starts strictly after the prior frontier so boundary evidence is not selected
twice.
Runtime or candidate limits retain the previous cursor. A scheduled command's
watermark does not authorize replay of archived or withdrawn evidence.

Expand Down
68 changes: 59 additions & 9 deletions docs/dreamer-native-provider.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,61 @@ children in their process group. The native companion has no tool executor.
## Budgets and review

Scheduled command mode uses the existing typed preview job: one provider call,
zero provider retries, bounded input/output, and a deadline covering pipe I/O.
zero provider retries, bounded input/output, and one absolute deadline covering
evidence selection, every sizing retry, and pipe I/O. Retrying input selection
does not replenish fractional seconds already consumed.
The Linux runner uses nonblocking pipes, not detached reader/writer threads.
An oversized response is stopped at the smaller job/configured byte limit.
Failure does not advance the successful scheduled watermark. Process-group
children are terminated; orphan reaping remains the operating system's job.
Input byte/token checks include the complete serialized provider envelope and
response schema. If pre-dispatch sizing exceeds an input budget, scheduled
command mode reselects fewer whole evidence records before its single provider
call, under the same byte, token, candidate, call, and runtime limits; it does
not truncate evidence text. Safety screening still covers the entire source,
including its tail. If one selected record still cannot fit, the
command is not dispatched. A deterministic preview processes that one record,
records the input limit, and advances only that source-kind cursor so later
evidence remains eligible. Other provider failures do not advance the successful
scheduled watermark. A recovered deterministic fallback also records a later
status-visible `ok_with_limits` receipt while retaining the provider budget
rejection in audit history. Completed unbounded simulated-clock scans retain
their rowid frontier even when the simulated clock does not advance. Scheduler
frontiers use persisted logical rowids for visible turns, conclusions,
checkpoints, imported sources, and memory records. Existing rows retain their
SQLite rowid during migration, while later inserts receive a logical rowid
above every saved cursor frontier; hard-deleting a physical maximum can no
longer make a replacement insert invisible. Each bounded band caps every
source stream at its captured logical rowid frontier. Memory record contents
are materialized by the same filtered query that checks the frontier, so a
later pooled read cannot substitute a post-frontier update into the evidence
window. The next band also checks rows inserted after that cap, including rows
whose source timestamp was backdated below the prior band's end. The first
bounded band after an unbounded scan uses the same post-frontier catch-up, so arrivals
between that snapshot and the next band are not lost below the timestamp
watermark. Existing-row updates that affect scheduler-visible evidence and
eligibility changes allocate a per-record mutation sequence in the same SQLite
transaction: provenance upserts, supersession and invalidation, consolidation
reuse and undo, restore, and promotion. The scheduler captures that sequence
with the rowid frontier so an uncommitted update cannot be skipped because its
wall-clock timestamp predates a scan. Archiving and quarantine remove rows from
eligible selection; a later reactivation advances the sequence. Previously
selected active records may provide comparison context, but only a record
selected in the current evidence window can emit stale or supersession
candidates. Process-group children are terminated; orphan reaping remains the
operating system's job.

Managed readiness accepts `local_only`, or `degraded` only when the reported
store is writable SQLite at the current schema, the exact configured
profile/workspace pair exists, exposure/auth fields are loopback-only, and
the sole degradation reason matches a persisted failed preview Dreamer receipt.
Native readiness checks the configured host database path; managed containers
check both the host directory mounted at `/data` and the exact configured
`/data/<filename>` path passed to the daemon inside the container. Reusing a
container with a stale database filename is refused before it is started.
`/v1/status` reports `features.configured_scope` using an exact workspace query,
rather than combining global profile and workspace lists. A healthy empty store
can start before its first workspace is created.
Storage, schema, scope, authorization, exposure, malformed-status, and other
degradation failures still block startup.

The scheduled command path is **preview-only**. Candidates preserve scoped
source references and remain subject to normal MemoryD validation and review.
Expand All @@ -67,17 +117,17 @@ accepted or useful.
## Verification

```sh
python -m unittest discover -s tests -p test_dreamer_native_provider.py -v
python3 -m unittest discover -s tests -p test_dreamer_native_provider.py -v
cargo test --lib
cargo test --test dream_command_scheduler --test command_no_http_fallback --test command_job_policy --test config_env
```

The command tests are Linux-scoped. Use a disposable database for actual
The command tests are Linux-scoped. Native lifecycle acceptance also needs a
reaping PID 1 or `scripts/run-with-child-reaper.py`; see `AGENTS.md` for the
isolated-check commands and the zombie-PID counterexample. The launcher cleans
up only its owned test descendants and preserves the main command's exit code.
Use a disposable database for actual
subscription tests; `dream --scheduled` exercises the scheduler's configured
provider, not a separate shell smoke test. Verify `command` provenance, the
selected model, validated `dream_provider_` observations, and zero accepted
memory writes. Keep raw receipts and profile paths private.

Three `dream_jobs` fixture failures also reproduce on the unchanged base:
HTTP fixtures used in HTTPS-only Provider mode, and a configured-credential
endpoint mismatch. They are not waived security checks or a green full suite.
5 changes: 5 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,11 @@ directory of the persistent SQLite database at `/data`, preserving SQLite
sidecar files such as WAL and shared-memory files. Docker or Podman must be
installed and available to the CLI.

When `--db` or `CODEX_MEMORYD_DB` selects a custom database, managed container
startup mounts that database's parent directory and configures the matching
`/data/<filename>` path inside the container. The host root cannot be used as
the database parent; choose a dedicated data directory instead.

To connect Codex to the container-backed read-only MCP server:

```zsh
Expand Down
8 changes: 8 additions & 0 deletions migrations/0014_scheduler_mutation_frontier.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- Commit-ordered mutation markers for scheduler-visible memory reactivation.
CREATE TABLE IF NOT EXISTS active_memory_mutation_counter (
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
sequence INTEGER NOT NULL
);

INSERT OR IGNORE INTO active_memory_mutation_counter(singleton, sequence)
VALUES (1, 0);
97 changes: 97 additions & 0 deletions migrations/0015_scheduler_nonreusable_source_rowids.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
-- Stable scheduler row IDs decouple saved cursors from SQLite rowid reuse.
CREATE TABLE IF NOT EXISTS scheduler_source_rowid_counter (
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
sequence INTEGER NOT NULL CHECK (sequence >= 0)
);

INSERT OR IGNORE INTO scheduler_source_rowid_counter(singleton, sequence)
VALUES (1, 0);

CREATE TABLE IF NOT EXISTS scheduler_source_rowids (
stream TEXT NOT NULL,
source_key TEXT NOT NULL,
scheduler_rowid INTEGER NOT NULL CHECK (scheduler_rowid > 0),
PRIMARY KEY (stream, source_key),
UNIQUE (stream, scheduler_rowid)
);

INSERT OR IGNORE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'visible_turns', id, rowid FROM visible_turns;
INSERT OR IGNORE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'conclusions', id, rowid FROM conclusions;
INSERT OR IGNORE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'checkpoints', id, rowid FROM checkpoints;
INSERT OR IGNORE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'memory_sources', id, rowid FROM memory_sources;
INSERT OR IGNORE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'memory_records', id, rowid FROM memory_records;

UPDATE scheduler_source_rowid_counter
SET sequence = MAX(
sequence,
COALESCE((SELECT MAX(scheduler_rowid) FROM scheduler_source_rowids), 0)
)
WHERE singleton = 1;

CREATE INDEX IF NOT EXISTS idx_scheduler_source_rowids_rowid
ON scheduler_source_rowids(stream, scheduler_rowid);

CREATE TRIGGER IF NOT EXISTS visible_turns_scheduler_rowid_insert
AFTER INSERT ON visible_turns BEGIN
UPDATE scheduler_source_rowid_counter SET sequence = sequence + 1 WHERE singleton = 1;
INSERT OR REPLACE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'visible_turns', NEW.id, sequence
FROM scheduler_source_rowid_counter WHERE singleton = 1;
END;
CREATE TRIGGER IF NOT EXISTS visible_turns_scheduler_rowid_delete
AFTER DELETE ON visible_turns BEGIN
DELETE FROM scheduler_source_rowids WHERE stream = 'visible_turns' AND source_key = OLD.id;
END;

CREATE TRIGGER IF NOT EXISTS conclusions_scheduler_rowid_insert
AFTER INSERT ON conclusions BEGIN
UPDATE scheduler_source_rowid_counter SET sequence = sequence + 1 WHERE singleton = 1;
INSERT OR REPLACE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'conclusions', NEW.id, sequence
FROM scheduler_source_rowid_counter WHERE singleton = 1;
END;
CREATE TRIGGER IF NOT EXISTS conclusions_scheduler_rowid_delete
AFTER DELETE ON conclusions BEGIN
DELETE FROM scheduler_source_rowids WHERE stream = 'conclusions' AND source_key = OLD.id;
END;

CREATE TRIGGER IF NOT EXISTS checkpoints_scheduler_rowid_insert
AFTER INSERT ON checkpoints BEGIN
UPDATE scheduler_source_rowid_counter SET sequence = sequence + 1 WHERE singleton = 1;
INSERT OR REPLACE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'checkpoints', NEW.id, sequence
FROM scheduler_source_rowid_counter WHERE singleton = 1;
END;
CREATE TRIGGER IF NOT EXISTS checkpoints_scheduler_rowid_delete
AFTER DELETE ON checkpoints BEGIN
DELETE FROM scheduler_source_rowids WHERE stream = 'checkpoints' AND source_key = OLD.id;
END;

CREATE TRIGGER IF NOT EXISTS memory_sources_scheduler_rowid_insert
AFTER INSERT ON memory_sources BEGIN
UPDATE scheduler_source_rowid_counter SET sequence = sequence + 1 WHERE singleton = 1;
INSERT OR REPLACE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'memory_sources', NEW.id, sequence
FROM scheduler_source_rowid_counter WHERE singleton = 1;
END;
CREATE TRIGGER IF NOT EXISTS memory_sources_scheduler_rowid_delete
AFTER DELETE ON memory_sources BEGIN
DELETE FROM scheduler_source_rowids WHERE stream = 'memory_sources' AND source_key = OLD.id;
END;

CREATE TRIGGER IF NOT EXISTS memory_records_scheduler_rowid_insert
AFTER INSERT ON memory_records BEGIN
UPDATE scheduler_source_rowid_counter SET sequence = sequence + 1 WHERE singleton = 1;
INSERT OR REPLACE INTO scheduler_source_rowids(stream, source_key, scheduler_rowid)
SELECT 'memory_records', NEW.id, sequence
FROM scheduler_source_rowid_counter WHERE singleton = 1;
END;
CREATE TRIGGER IF NOT EXISTS memory_records_scheduler_rowid_delete
AFTER DELETE ON memory_records BEGIN
DELETE FROM scheduler_source_rowids WHERE stream = 'memory_records' AND source_key = OLD.id;
END;
Loading
Loading