chore: regenerate SQLx/cache artifacts and harden local Postgres workflows by Sinity · Pull Request #98 · Sinity/sinex

Sinity · 2026-04-09T09:50:50Z

Summary

I started this branch to refresh the SQLx cache and quickly ran into a more fundamental problem: the repository still assumed a lot of implicit local Postgres behavior that was never written down consistently. I used the regeneration work as a forcing function to standardize the developer shell, make helper scripts point at real package boundaries, and remove stale workflow/docs scaffolding that still described older habits. Along the way I refreshed a number of adjacent surfaces—CLI operator guidance, gateway/runtime follow-on cleanup, NixOS database options, and bundle-generation helpers—because they all sat in the same “how do I work on this repo locally?” path. The result is a branch that is broader than a pure .sqlx refresh, but much more coherent for anyone trying to regenerate schemas, rebuild caches, or validate the current stack from a fresh shell.

Motivation

The branch exists because SQLx regeneration was not really a standalone maintenance task anymore.

Regenerating .sqlx or the checked-in schema artifacts depended on knowing which database to point at, which shell to enter, which helper commands were current, and which files in the repository were historical noise rather than living instructions. Some of the local scripts inferred crate names incorrectly. The shell banner told you very little about whether Postgres was actually ready. The README, PR template, and workflow docs still had language from earlier architecture and process phases. And there was no checked-in .env.example giving a new contributor a minimal, concrete starting point.

I also hit an architectural pivot inside the branch itself. Early on I deliberately removed some host-specific build tuning from .cargo/config.toml to make the defaults portable and obvious. Later in the branch I re-enabled mold and more aggressive linker flags, but this time as an explicit, reviewable choice after the underlying local Postgres and task flow had been cleaned up. That “remove hidden tuning first, then add back the parts we still want” arc is representative of the whole branch.

Changes grouped by concern

1. Make the local shell and database story explicit

I added a real .env.example and extended scripts/dev-env-banner.sh so the shell tells you what database it expects, whether that database is up, which toolchain you are on, and which commands are supposed to start the main processes. That may sound small, but it changes the onboarding loop from “guess until Cargo or psql fails” to “the shell immediately tells you whether the environment is sane.”

I also simplified .cargo/config.toml. The old version mixed general workspace settings with aggressive per-host tuning and did not make the local Postgres assumptions obvious. The new version exports the common development DB settings directly:

PGHOST=/run/postgresql
PGUSER=sinity
PGDATABASE=sinex_dev
DATABASE_URL=postgresql:///sinex_dev?host=/run/postgresql

That makes schema generation, SQLx preparation, and direct psql inspection all converge on the same defaults.

2. Fix the scripts developers actually lean on

scripts/smart-check.sh and scripts/smart-watch.sh were relying on brittle path heuristics. I rewrote them to map changed files to real package names, including the e2e test package, so a developer editing one crate does not accidentally get a workspace-wide check or, worse, a check of the wrong package.

I also spent time on scripts/combine-files-batch.sh. The branch adds skip-path logic, optional symlink following, deterministic ordering, and richer output products such as tokei_plus_gitlog.md and commit-diff shards. The point is not “more artifacts for their own sake”; it is that after schema/cache churn, we should be able to regenerate analysis bundles in a repeatable way without touching unrelated files.

Example:

./scripts/combine-files-batch.sh . combined-bundles

now produces the source/test/doc bundles plus the extra repo-stat and diff outputs in one pass.

3. Remove stale repository scaffolding

The SQLx refresh made it harder to ignore older repo furniture that no longer matched the active workflow. I deleted the stale Claude LSP configuration and the older pre-commit artifacts, refreshed the PR template, and updated the workflow README so they describe the current schema/test flow instead of an older abstraction-check story.

The README also got a meaningful cleanup: it now calls out devenv, makes the RPC/auth guidance more concrete, and documents the combined-bundle workflow as a local analysis helper rather than an ad hoc script.

I treated this as part of the same problem because stale docs and stale hooks are just another kind of broken local workflow. They waste the same time and create the same false confidence as a stale SQLx cache.

4. Carry the follow-on operator/runtime cleanups that the branch exposed

Several adjacent surfaces were already out of sync with the local/dev shell story, so I fixed them while the branch was touching the same files.

The CLI now prints more useful RPC guidance and adds aggregated DLQ metrics instead of forcing operators to hand-assemble that view from raw tables. Gateway/replay/native-messaging work landed in the same window, and the shared test harness was pushed further across CLI/Nix integration checks and service/runtime regressions. I also tightened the NixOS database module so the provisioning story matches the shell defaults and parallel nextest load more closely.

This is the part of the branch that looks the least like a traditional “cache refresh,” but it is still coherent when viewed from the operator/developer loop: the same people who regenerate .sqlx locally are also the people who need accurate CLI guidance, current docs, and a shell that matches the database module’s defaults.

Testing

The branch includes a mix of direct tests and workflow-facing validation. The script changes are exercised indirectly through the local shell/task flow and the updated CLI/Nix integration checks that were moved onto sinex_test_utils. The services and runtime follow-on work adds analytics/search regressions, document-ingestor coverage cleanup, processor-runtime checkpoint coverage, and other targeted test updates in the crates that were touched. For pure documentation and stale-artifact removals, the protection comes from the repo being simpler and from the updated workflow/docs paths being the only remaining path.

The main gap is that several of the local-workflow improvements are operational rather than unit-testable in isolation. For those, the value is in making the one supported path explicit and eliminating the older alternatives that drifted.

Notes

The branch intentionally mixes “regenerate cache artifacts” with “remove stale local scaffolding” and “refresh current workflow docs.” I would rather have one coherent maintenance window that leaves the repository easier to use than a perfectly narrow cache-only change that leaves the local story confusing.

I also consider the tuning pivot important context: I removed hidden host-specific build tuning first, then put back the parts we still wanted once they were explicit and documented. The same principle guided the rest of the branch: delete the implicit or stale path, then keep the explicit one.

Ref #97

Add the current `.sqlx` query metadata needed for Nix and offline validation after the recent schema and query changes. Co-Authored-By: Claude <noreply@anthropic.com>

Teach the helper scripts to derive more accurate changed-crate sets so incremental check and watch runs follow the files that actually changed.

- Add skip-path handling, optional symlink following, and explicit output ordering to `scripts/combine-files-batch.sh`. - Keep regenerated source/test/doc bundles stable between runs so analysis artifacts stop churning.

… `.env` template Add a checked-in `.env.example` that documents the expected local database variables, then introduce `scripts/dev-env-banner.sh` to render a TTY-only shell banner when entering the dev environment. The banner reports PostgreSQL readiness with `pg_isready`, surfaces the active toolchain/process hints, and prints common devenv commands. This turns shell entry into an explicit onboarding surface instead of relying on remembered local setup steps.

…stgres env defaults Simplify `.cargo/config.toml` by removing machine-shaped performance defaults (`jobs`, linker mold flag, and high codegen-unit tuning) that should not be hard-coded for every contributor. In exchange, add shared Postgres environment variables (`PGHOST`, `PGUSER`, `PGDATABASE`, `DATABASE_URL`) so cargo-invoked workflows resolve to the expected local database by default. The commit narrows cargo config to portable workspace defaults after the surrounding devshell cleanup.

Change the `ci-parallel` nextest profile from a pinned `test-threads = 18` to `test-threads = "num-cpus"`. This removes an implicit workstation-specific assumption and lets each CI runner size parallelism to its own capacity, while still allowing explicit override via `NEXTEST_TEST_THREADS`. The change is consistent with nearby commits that remove hard-coded local tuning from shared repository config.

Remove three abandoned repo-management files: `.claude/cclsp.json`, `.githooks/pre-commit`, and `.pre-commit-config.yaml`. Those artifacts referenced old validation flows (`cargo fmt`, schema scripts, ad-hoc shell checks) that no longer represented the active workflow, so keeping them created misleading maintenance paths. Deleting them reduces toolchain drift and avoids duplicate, outdated hook enforcement next to the refreshed CI and PR process.

- Update workflow docs and the PR template to match the current nextest/schema regeneration flow. - Remove older checklist language that still assumed earlier abstraction and test commands.

Tighten the top-level architecture description and quick-start guidance so the README matches the JetStream-first runtime and current VM test commands.

- Extend `exo` with aggregated DLQ metrics and clearer RPC/database failure guidance. - Make operator output more useful when the gateway or local database is unavailable.

- Extend the gateway control surface with replay RPC methods. - Carry the native-messaging support changes needed to exercise the same path locally.

…rsistence tests Harden ingestd startup and ingestion paths by replacing ad-hoc SQL with `sqlx::query!` in both config smoke checks and batch event insertion. The consumer now builds owned vectors for bulk `UNNEST` inserts, uses typed `RETURNING id::uuid`, and converts rows back to ULIDs without `Row` index extraction. Test updates remove unnecessary `TestContext` parameters from pure config tests and align schema-order validator assertions with the current non-`Result` API. This keeps ingestd validation and persistence behavior consistent with typed-query patterns.

Expand the processor-runtime CLI around checkpoint parsing and scan control, and add runner tests that exercise the updated execution paths.

- Tighten the heartbeat, annex, and confirmation plumbing in the SDK so they match the current ingestion/runtime assumptions. - Reduce older flow-specific branching while the surrounding cache/schema work is already touching the same code.

Adjust the canonical migration, clean up schema exports, and refresh the serde, validation, and ULID test surface around the schema crate.

…ce tests to shared harness conventions Deepen service-layer testing across search, analytics, query, and integration suites. Prepared-search checks move to `#[sinex_test]` and explicit `TestResult`, analytics fixtures get stronger typed time handling and a new concurrency regression that verifies requests do not starve when one query holds a single-connection pool, and integration tests adopt unified `TestResult` signatures. The query-service fixture command text is also updated to current nextest usage. Together this raises confidence around pagination, time-window filtering, and cross-service behavior under realistic load patterns.

- Clean up fixture ergonomics, benchmark support, and helper documentation in `sinex-test-utils`. - Keep the shared harness aligned with the current local workflow after the shell/database cleanup.

…cal checks to `sinex_test` Align the analytics automaton crate with shared testing and typing patterns by adding `sinex-test-utils` as a dev dependency, switching local unit checks from raw `#[test]` to `#[sinex_test]`, and returning explicit `color_eyre::Result<()>` in those test cases. The production path also narrows one helper signature to `CoreResult<Vec<Event<JsonValue>>>`, keeping error typing consistent with core conventions. This commit is mostly reliability hygiene so automaton behavior and tests evolve with the same harness used elsewhere.

…acked blob persistence Extend `ClipboardWatcher` with optional blob-manager wiring: initialize a git-annex repository, create a `BlobManager`, and ingest large clipboard payloads as annex blobs instead of skipping them. Metadata now records storage strategy (`inline` vs `annex_blob`) and blob identifiers, while source material inserts use the enriched metadata and stricter temporal precision tagging. The patch also adds focused tests for large-content persistence and degraded operation without a direct DB pool, making clipboard ingestion behavior explicit across both storage paths.

Create a crate-local document-ingestor test entry point and trim the older scattered setup so direct-ingestion behavior is easier to exercise.

Collapse the filesystem watcher validation logic into `FilesystemConfig` itself and align the config and unified-processor tests with the tighter bounds.

…o `sinex_test` Bring the search automaton in line with current crate conventions by adding `sinex-test-utils` for local tests, converting internal checks from raw `#[test]` to `#[sinex_test]`, and returning explicit `color_eyre::Result<()>` in those cases. The production helper signature is also tightened from a concrete `SinexError` return to the core `Result` alias. This is a consistency-and-reliability pass so search automaton behavior and test ergonomics match neighboring satellites.

Build on the watcher scaffolding by wiring more system watcher behavior, exporting the related modules, and adding processor/systemd integration coverage.

…y watcher state Rework terminal processor state, history-watcher context, command processing, and tests so the terminal satellite continues its move toward a single unified processor path.

… dependency update helpers Delete three stale maintenance scripts that no longer match the active toolchain and workflow: `scripts/dependency_graph.py`, `scripts/test-analytics.sh`, and `scripts/update_deps.sh`. The removed helpers depended on ad-hoc cargo flows, custom analytics directories, and manual crates.io polling that had been superseded by newer workspace tooling. Removing them reduces dead operational surface and prevents contributors from following commands that no longer represent supported project maintenance practices.

Add `schemas/README.md` to explain the generated JSON payload schemas, how they are produced, and how to validate them after code changes.

…rackers Annotate `docs/TODO.md` and `docs/TODO_audit_status.md` with explicit historical-context notes explaining that some entries still reference pre-pivot sensd/gRPC ingestion. The new banner text points readers to `docs/way.md` for the current JetStream-only architecture. This keeps old backlog items for audit traceability while preventing contributors from mistaking transitional references for active implementation direction.

Add historical context to the older vision and analysis documents so readers can distinguish archived planning from the active architecture.

…s and current DB ownership Refresh the high-level architecture/security notes under `docs/misc-including-high-level-overviews-and-plans/` so they stop pointing at the old `sinex-db` crate layout and instead reference `crate/lib/sinex-core` repositories and query helpers. The patch updates both prose mandates and concrete extracted-path references (`db/security.rs`, `db/pool.rs`, repository files). This keeps umbrella documentation aligned with the actual crate boundaries after the core/database refactor sequence.

…tifacts Insert explicit JetStream-transition context notes across the test-suite report corpus (`TEST_ANALYSIS_README`, summary, inventory, and infrastructure analysis). The added banners clarify that gRPC and transactional-outbox references in those artifacts describe a migration-era snapshot, not the live architecture. This preserves archival reports without letting readers infer that dual-ingest pathways are still current behavior.

… context Add a top-level title and historical-context banner to `docs/testing-gap-analysis.md` explaining that portions of the gap list were captured during a dual-path (gRPC + JetStream) transition period. The note explicitly redirects readers to `docs/way.md` for current ingestion behavior. This prevents misinterpretation of legacy gap entries while keeping the document useful for audit trail and follow-up planning.

…orkflow outcomes Update `docs/way_2.md` from aspirational checklist to status-tracked record by marking previously open tasks as done and adding concrete completion notes. The commit documents that schema regeneration contributor guidance, historical doc sweeps, CI profile alignment, query-lint guardrails, and legacy scaffolding removals have landed. This turns `way_2` into an accurate progress ledger for architecture/documentation convergence rather than a stale list of unresolved intentions.

Adjust the flake wiring, database module defaults, module imports, and secrets plumbing so Sinex services provision PostgreSQL and related state more predictably.

Rewrite the top-level CLI and Nix smoke tests to use `TestContext` / `TestResult` conventions and add DLQ plus confirmation command coverage that matches the surrounding CLI work.

- Sweep the core schema/replay/repository layers to keep their APIs consistent after the regeneration and workflow changes. - Trim mismatched naming and stale assumptions instead of carrying them into later branches.

Add `.devenv/` to `.gitignore` so local environment state generated by devenv no longer pollutes `git status`. The diff is tiny, but it removes recurring worktree noise during active refactor/build churn and lowers the chance of accidentally staging environment-local artifacts. This keeps reviews focused on intentional source changes rather than transient local runtime files.

…y churn Regenerate `Cargo.lock` after the surrounding build and retry adjustments, capturing a sizable resolver shift: new HTTP/TLS transport stack entries (for example `reqwest 0.12`, `hyper-rustls`, `quinn`, `tokio-rustls 0.26`), plus workspace dependency realignment such as added `sinex-test-utils` links and trimmed document-ingestor/test dependency edges. The commit is mechanical but important for reproducibility because it freezes the exact crate graph implied by the preceding infrastructure edits.

Turn the faster linker and related build settings back on in `.cargo/config.toml` after the earlier CI and toolchain adjustments settled.

Increase the development/test PostgreSQL connection budget and memory/parallelism settings in the Nix module. Raise the test-utils pool budget accordingly so heavily parallel nextest runs stop failing from connection exhaustion. Co-Authored-By: Claude <noreply@anthropic.com>

chatgpt-codex-connector · 2026-04-09T09:50:55Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

coderabbitai · 2026-04-09T09:50:57Z

Important

Review skipped

Too many files!

This PR contains 216 files, which is 66 over the limit of 150.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c2890365-c0c0-418b-b934-0f939821bcd4

📥 Commits

Reviewing files that changed from the base of the PR and between c82449a and 16af1b5.

⛔ Files ignored due to path filters (2)

Cargo.lock is excluded by !**/*.lock, !**/*.lock, !**/Cargo.lock
flake.lock is excluded by !**/*.lock, !**/*.lock, !**/flake.lock

📒 Files selected for processing (216)

.cargo/config.toml
.claude/cclsp.json
.config/nextest.toml
.env.example
.githooks/pre-commit
.github/pull_request_template.md
.github/workflows/README.md
.github/workflows/ci.yml
.github/workflows/sqlx-cache.yml
.github/workflows/sqlx-check.yml
.gitignore
.pre-commit-config.yaml
README.md
cli/exo.py
crate/core/sinex-gateway/Cargo.toml
crate/core/sinex-gateway/doc/overview.md
crate/core/sinex-gateway/doc/rpc_server.md
crate/core/sinex-gateway/src/cascade_analyzer.rs
crate/core/sinex-gateway/src/handlers.rs
crate/core/sinex-gateway/src/lib.rs
crate/core/sinex-gateway/src/native_messaging.rs
crate/core/sinex-gateway/src/replay_control.rs
crate/core/sinex-gateway/src/rpc_server.rs
crate/core/sinex-gateway/tests/cascade_analyzer_cycle_test.rs
crate/core/sinex-gateway/tests/cascade_analyzer_tests.rs
crate/core/sinex-gateway/tests/replay_state_machine_tests.rs
crate/core/sinex-gateway/tests/service_container_test.rs
crate/core/sinex-ingestd/src/config.rs
crate/core/sinex-ingestd/src/jetstream_consumer.rs
crate/core/sinex-ingestd/tests/config_security_tests.rs
crate/core/sinex-ingestd/tests/validator_schema_order_tests.rs
crate/lib/sinex-core/src/db/distributed_locking.rs
crate/lib/sinex-core/src/db/integrity.rs
crate/lib/sinex-core/src/db/mod.rs
crate/lib/sinex-core/src/db/replay/state_machine.rs
crate/lib/sinex-core/src/db/repositories/events.rs
crate/lib/sinex-core/src/db/repositories/source_materials.rs
crate/lib/sinex-core/src/db/repositories/state.rs
crate/lib/sinex-core/src/types/bin/sinex-schema.rs
crate/lib/sinex-core/src/types/events/payloads/filesystem.rs
crate/lib/sinex-core/src/types/events/payloads/mod.rs
crate/lib/sinex-core/src/types/events/payloads/process.rs
crate/lib/sinex-core/src/types/events/payloads/shell.rs
crate/lib/sinex-core/src/types/events/test_helpers.rs
crate/lib/sinex-core/src/types/utils/timestamp_helpers.rs
crate/lib/sinex-core/tests/adversarial/attack_simulation_test.rs
crate/lib/sinex-core/tests/adversarial/chaos_engineering_test.rs
crate/lib/sinex-core/tests/adversarial/concurrency_test.rs
crate/lib/sinex-core/tests/adversarial/security_test.rs
crate/lib/sinex-core/tests/adversarial/ulid_edge_cases_test.rs
crate/lib/sinex-core/tests/concurrency/checkpoint_concurrency_test.rs
crate/lib/sinex-core/tests/concurrency/mod.rs
crate/lib/sinex-core/tests/database.rs
crate/lib/sinex-core/tests/error.rs
crate/lib/sinex-core/tests/integration/checkpoint_consistency_test.rs
crate/lib/sinex-core/tests/integration/pipeline_integration_test.rs
crate/lib/sinex-core/tests/integration/provenance_test.rs
crate/lib/sinex-core/tests/integration/test_automation_integration_test.rs
crate/lib/sinex-core/tests/integration_tests.rs
crate/lib/sinex-core/tests/payloads_clipboard.rs
crate/lib/sinex-core/tests/performance/bottleneck_identification_test.rs
crate/lib/sinex-core/tests/performance/concurrent_load_test.rs
crate/lib/sinex-core/tests/performance/database_performance_test.rs
crate/lib/sinex-core/tests/performance/jetstream_performance_test.rs
crate/lib/sinex-core/tests/performance/large_payload_test.rs
crate/lib/sinex-core/tests/performance/memory_usage_test.rs
crate/lib/sinex-core/tests/performance/mod.rs
crate/lib/sinex-core/tests/performance/regression_detection_test.rs
crate/lib/sinex-core/tests/performance/resource_exhaustion_test.rs
crate/lib/sinex-core/tests/property/event_model_fuzzing_test.rs
crate/lib/sinex-core/tests/property/path_sanitization_property_test.rs
crate/lib/sinex-core/tests/property/schema_property_test.rs
crate/lib/sinex-core/tests/property/time_range_property_test.rs
crate/lib/sinex-core/tests/property/ulid_property_test.rs
crate/lib/sinex-core/tests/property_tests.rs
crate/lib/sinex-core/tests/repositories_common.rs
crate/lib/sinex-core/tests/repositories_schema_management.rs
crate/lib/sinex-core/tests/sanitization.rs
crate/lib/sinex-core/tests/security.rs
crate/lib/sinex-core/tests/security/ulid_attack_test.rs
crate/lib/sinex-core/tests/security/unicode_attack_test.rs
crate/lib/sinex-core/tests/system/mod.rs
crate/lib/sinex-core/tests/system/performance_test.rs
crate/lib/sinex-core/tests/system/reliability_test.rs
crate/lib/sinex-core/tests/system/stress_test.rs
crate/lib/sinex-core/tests/system/temporal_chaos_test.rs
crate/lib/sinex-core/tests/unit/database_test.rs
crate/lib/sinex-core/tests/unit/event_type_system_test.rs
crate/lib/sinex-core/tests/unit/preflight_test.rs
crate/lib/sinex-core/tests/unit/version_system_test.rs
crate/lib/sinex-core/tests/validation_config.rs
crate/lib/sinex-core/tests/validation_core.rs
crate/lib/sinex-processor-runtime/Cargo.toml
crate/lib/sinex-processor-runtime/src/cli.rs
crate/lib/sinex-processor-runtime/tests/processor_runner.rs
crate/lib/sinex-satellite-sdk/Cargo.toml
crate/lib/sinex-satellite-sdk/doc/overview.md
crate/lib/sinex-satellite-sdk/src/acquisition_manager.rs
crate/lib/sinex-satellite-sdk/src/annex/blob_manager.rs
crate/lib/sinex-satellite-sdk/src/automaton_event_handler.rs
crate/lib/sinex-satellite-sdk/src/confirmation_handler.rs
crate/lib/sinex-satellite-sdk/src/dlq_retry.rs
crate/lib/sinex-satellite-sdk/src/heartbeat.rs
crate/lib/sinex-satellite-sdk/src/jetstream_consumer.rs
crate/lib/sinex-satellite-sdk/src/lease_manager.rs
crate/lib/sinex-satellite-sdk/src/runtime/stream/runtime_state.rs
crate/lib/sinex-satellite-sdk/src/stage_as_you_go.rs
crate/lib/sinex-satellite-sdk/tests/ingestion_helpers.rs
crate/lib/sinex-satellite-sdk/tests/integration/blob_manager_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/checkpoint_persistence_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/config_environment_validation_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/critical_failure_modes_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/event_generation_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/satellite_architecture_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/satellite_coordination_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/satellite_lifecycle_test.rs
crate/lib/sinex-satellite-sdk/tests/integration/version_migration_test.rs
crate/lib/sinex-satellite-sdk/tests/material_acquisition.rs
crate/lib/sinex-satellite-sdk/tests/property/checkpoint_property_test.rs
crate/lib/sinex-satellite-sdk/tests/replay_control.rs
crate/lib/sinex-satellite-sdk/tests/security/path_validation_test.rs
crate/lib/sinex-satellite-sdk/tests/stage_as_you_go_requires_db.rs
crate/lib/sinex-satellite-sdk/tests/system/external_test.rs
crate/lib/sinex-schema/Cargo.toml
crate/lib/sinex-schema/src/migrations/m20241028_000001_create_canonical_schema.rs
crate/lib/sinex-schema/src/schema/mod.rs
crate/lib/sinex-schema/tests/serde_tests.rs
crate/lib/sinex-schema/tests/ulid_property_from_main_tests.rs
crate/lib/sinex-schema/tests/validation_tests.rs
crate/lib/sinex-services/src/search.rs
crate/lib/sinex-services/tests/analytics_service_test.rs
crate/lib/sinex-services/tests/query_service_test.rs
crate/lib/sinex-services/tests/service_integration_test.rs
crate/lib/sinex-test-utils/IMPROVEMENTS.md
crate/lib/sinex-test-utils/macros/src/lib.rs
crate/lib/sinex-test-utils/src/bench_context.rs
crate/lib/sinex-test-utils/src/bench_results.rs
crate/lib/sinex-test-utils/src/channel_behavior_utils.rs
crate/lib/sinex-test-utils/src/database_pool.rs
crate/lib/sinex-test-utils/src/db_common.rs
crate/lib/sinex-test-utils/src/error_testing.rs
crate/lib/sinex-test-utils/src/fixture_generator.rs
crate/lib/sinex-test-utils/src/fixtures.rs
crate/lib/sinex-test-utils/src/lib.rs
crate/lib/sinex-test-utils/src/property_testing.rs
crate/lib/sinex-test-utils/src/resources.rs
crate/lib/sinex-test-utils/src/satellite_management_utils.rs
crate/lib/sinex-test-utils/src/static_fixtures.rs
crate/lib/sinex-test-utils/src/test_context.rs
crate/lib/sinex-test-utils/src/timing_utils.rs
crate/lib/sinex-test-utils/tests/basic_enhanced_test.rs
crate/lib/sinex-test-utils/tests/database_pool_tests.rs
crate/lib/sinex-test-utils/tests/deployment_scenario_utils_tests.rs
crate/lib/sinex-test-utils/tests/fully_integrated_test.rs
crate/lib/sinex-test-utils/tests/integration/collector_test.rs
crate/lib/sinex-test-utils/tests/integration/configuration_test.rs
crate/lib/sinex-test-utils/tests/integration/multi_source_integration_test.rs
crate/lib/sinex-test-utils/tests/integration/stream_processing_test.rs
crate/lib/sinex-test-utils/tests/macro_conversion_examples.rs
crate/lib/sinex-test-utils/tests/modern_test_infrastructure_example.rs
crate/lib/sinex-test-utils/tests/rstest_integration_example.rs
crate/satellites/sinex-analytics-automaton/Cargo.toml
crate/satellites/sinex-analytics-automaton/src/lib.rs
crate/satellites/sinex-desktop-satellite/Cargo.toml
crate/satellites/sinex-desktop-satellite/src/clipboard.rs
crate/satellites/sinex-document-ingestor/Cargo.toml
crate/satellites/sinex-document-ingestor/tests/.gitkeep
crate/satellites/sinex-document-ingestor/tests/direct_ingestion_test.rs
crate/satellites/sinex-fs-watcher/src/unified_processor.rs
crate/satellites/sinex-fs-watcher/tests/config_validation_tests.rs
crate/satellites/sinex-fs-watcher/tests/unified_processor_tests.rs
crate/satellites/sinex-search-automaton/Cargo.toml
crate/satellites/sinex-search-automaton/src/lib.rs
crate/satellites/sinex-system-satellite/Cargo.toml
crate/satellites/sinex-system-satellite/doc/README.md
crate/satellites/sinex-system-satellite/src/lib.rs
crate/satellites/sinex-system-satellite/src/unified_processor.rs
crate/satellites/sinex-system-satellite/tests/system_processor_watchers.rs
crate/satellites/sinex-system-satellite/tests/systemd_integration_tests.rs
crate/satellites/sinex-terminal-satellite/Cargo.toml
crate/satellites/sinex-terminal-satellite/src/unified_processor.rs
docs/TODO.md
docs/TODO_audit_status.md
docs/historical/FINAL_COMPREHENSIVE_SINEX_UNDERSTANDING.md
docs/historical/SINEX_CRITICAL_ASSESSMENT_PART_THREE.md
docs/historical/VISION.md
docs/historical/_experimental_research_analysis.md
docs/historical/_git_history_development_analysis.md
docs/historical/_implementation_plans_analysis.md
docs/historical/hdraft_v2.md
docs/misc-including-high-level-overviews-and-plans/_new_ideas_discussion.md
docs/misc-including-high-level-overviews-and-plans/_security_privacy_architecture_analysis.md
docs/misc-including-high-level-overviews-and-plans/_top_down_sinex_description.md
docs/misc-including-high-level-overviews-and-plans/_understanding.md
docs/test-suite-report/TEST_ANALYSIS_README.md
docs/test-suite-report/TEST_ANALYSIS_SUMMARY.txt
docs/test-suite-report/TEST_FILE_INVENTORY.md
docs/test-suite-report/TEST_INFRASTRUCTURE_ANALYSIS.md
docs/testing-gap-analysis.md
docs/way_2.md
flake.nix
nixos/modules/database.nix
nixos/modules/default.nix
nixos/modules/secrets.nix
schemas/README.md
scripts/combine-files-batch.sh
scripts/dependency_graph.py
scripts/dev-env-banner.sh
scripts/generate-llm-context.py
scripts/smart-check.sh
scripts/smart-watch.sh
scripts/test-analytics.sh
scripts/update_deps.sh
tests/cli_missing_commands.rs
tests/e2e/tests/cli_smoke_test.rs
tests/e2e/tests/nix_module_integration_test.rs

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

`.coderabbit.yaml` has unrecognized properties

CodeRabbit is using all valid settings from your configuration. Unrecognized properties (listed below) have been ignored and may indicate typos or deprecated fields that can be removed.

⚠️ Parsing warnings (1)

Validation error: Unrecognized key(s) in object: 'finishing_touches'

⚙️ Configuration instructions

Please see the configuration documentation for more information.
You can also validate your configuration using the online YAML validator.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feature/chore/sqlx-cache-local-postgres

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…flows (#98) Tighten the local developer loop around SQLx/schema regeneration by standardizing Postgres defaults, cleaning stale scaffolding, and updating the surrounding scripts, docs, and CI guidance to match the current repository shape. local workflow and scripts: - Add `.env.example` and extend `scripts/dev-env-banner.sh` so the shell shows database readiness, expected process commands, and the environment variables developers are supposed to rely on instead of tribal knowledge. - Rework `.cargo/config.toml` to drop host-specific build tuning, export shared local Postgres defaults (`PGHOST`, `PGUSER`, `PGDATABASE`, `DATABASE_URL`), and keep the fast-path build flags explicit rather than hidden in per-machine setup. - Fix `scripts/smart-check.sh` and `scripts/smart-watch.sh` so they target actual package names instead of guessing from path fragments, which keeps incremental local checks from silently falling back to the wrong scope. - Expand `scripts/combine-files-batch.sh` with skip-path handling, optional symlink following, deterministic ordering, and extra outputs (`tokei_plus_gitlog.md`, diff shards) so local analysis bundles are reproducible and easier to regenerate after cache/schema churn. repository cleanup and docs: - Delete stale `.claude`/pre-commit artifacts, update `.github/pull_request_template.md`, refresh workflow docs, and rewrite the README sections that still described older architecture or workflow assumptions. - Move more docs onto the current Postgres/devenv/schema story, including the schema regeneration workflow, historical-context notes, and the NixOS database provisioning guidance. - Ignore `.devenv/` state and refresh the lockfile after the toolchain/dependency churn so local setup stops producing noisy, unexplained diffs. CLI, gateway, runtime, and tests: - Extend `cli/exo.py` with better RPC fallback guidance plus aggregated DLQ metrics and related operator output, which makes the CLI more useful when the RPC layer or local database is unavailable. - Carry follow-on gateway/runtime/schema/test cleanups that were exposed by the regenerated cache work, including replay/native-messaging support, typed SQLx checks, processor-runtime/checkpoint coverage, and shared harness updates. - Re-enable mold/aggressive linker flags and raise local Postgres defaults for parallel nextest once the earlier “drop host-specific tuning” pass made those knobs explicit and reviewable.

Copilot

Pull request overview

This PR refreshes and standardizes “local dev + Postgres + SQLx offline” workflows across the repo, while also aligning tests, scripts, and NixOS module behavior with the current JetStream-first architecture and updated crate boundaries.

Changes:

Standardize local dev environment defaults (explicit .env.example, dev shell banner, Cargo env defaults, smarter check/watch scripts).
Update NixOS module database provisioning (extra DBs, extension setup, migration binary rename) and align CI Postgres images / nextest profiles.
Sweep tests/docs to use the #[sinex_test] harness + TestResult, remove stale scaffolding, and update schema/operator documentation.

Reviewed changes

Copilot reviewed 213 out of 218 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/e2e/tests/nix_module_integration_test.rs	Switch e2e tests to `TestResult` and simplify signatures.
tests/e2e/tests/cli_smoke_test.rs	Convert CLI smoke test to `#[sinex_test]` + `TestResult`.
tests/cli_missing_commands.rs	Update CLI command tests to use `color_eyre::Result` and inline `Command`.
scripts/update_deps.sh	Remove unused dependency update helper script.
scripts/test-analytics.sh	Remove legacy test analytics script.
scripts/smart-watch.sh	Improve crate/package detection (incl. e2e package) for targeted watch builds.
scripts/smart-check.sh	Improve changed-package detection and run targeted `cargo check -p`.
scripts/dev-env-banner.sh	Add interactive dev shell banner with DB/toolchain/process hints.
scripts/dependency_graph.py	Remove legacy dependency graph generator.
scripts/combine-files-batch.sh	Add skip-paths, optional symlink following, deterministic collection improvements.
schemas/README.md	Document Rust→JSON→Postgres schema flow and non-Rust contributor workflow.
README.md	Update feature bullets and crate layout; add schema regeneration guidance.
nixos/modules/secrets.nix	Gate agenix integration config and expose secrets paths/export snippet.
nixos/modules/default.nix	Add `extraDatabases`; update migration binary default to `sinex-schema`.
nixos/modules/database.nix	Provision multiple DBs; adjust max_connections logic; add extension auto-setup; tune settings.
flake.nix	Update migration packaging notes to reflect `sinex-schema`.
docs/way_2.md	Mark workflow/doc tasks complete; update narrative around current architecture.
docs/TODO.md	Add historical-context banner re: pre–JetStream references.
docs/TODO_audit_status.md	Add historical-context banner re: pre–JetStream references.
docs/testing-gap-analysis.md	Add historical-context banner and heading normalization.
docs/test-suite-report/TEST_INFRASTRUCTURE_ANALYSIS.md	Add historical-context banner; trim trailing whitespace.
docs/test-suite-report/TEST_FILE_INVENTORY.md	Add historical-context banner.
docs/test-suite-report/TEST_ANALYSIS_SUMMARY.txt	Add historical-context note.
docs/test-suite-report/TEST_ANALYSIS_README.md	Add historical-context banner.
docs/misc-including-high-level-overviews-and-plans/_understanding.md	Update extracted-path references to new crate locations.
docs/misc-including-high-level-overviews-and-plans/_top_down_sinex_description.md	Update “query centralization” references from sinex-db→sinex-core.
docs/misc-including-high-level-overviews-and-plans/_security_privacy_architecture_analysis.md	Update path references to sinex-core security location.
docs/misc-including-high-level-overviews-and-plans/_new_ideas_discussion.md	Update “query centralization” / crate naming references.
docs/historical/VISION.md	Add explicit historical note banner.
docs/historical/SINEX_CRITICAL_ASSESSMENT_PART_THREE.md	Add explicit historical note banner.
docs/historical/hdraft_v2.md	Add explicit historical note banner / front-matter tweak.
docs/historical/FINAL_COMPREHENSIVE_SINEX_UNDERSTANDING.md	Add explicit historical note banner.
docs/historical/_implementation_plans_analysis.md	Add explicit historical note banner.
docs/historical/_git_history_development_analysis.md	Add explicit historical note banner.
docs/historical/_experimental_research_analysis.md	Add explicit historical note banner.
crate/satellites/sinex-terminal-satellite/Cargo.toml	Add `tempfile` dev-dependency.
crate/satellites/sinex-system-satellite/tests/systemd_integration_tests.rs	Convert tests to `#[sinex_test]` + `TestResult`.
crate/satellites/sinex-system-satellite/tests/system_processor_watchers.rs	Fix event_type comparison to use `as_str()`.
crate/satellites/sinex-system-satellite/src/unified_processor.rs	Add watcher snapshot + unit test for watcher initialization.
crate/satellites/sinex-system-satellite/src/lib.rs	Re-export `WatcherSnapshot`.
crate/satellites/sinex-system-satellite/doc/README.md	Document watcher overview and readiness snapshot.
crate/satellites/sinex-system-satellite/Cargo.toml	Add `sinex-test-utils` dev-dependency.
crate/satellites/sinex-search-automaton/src/lib.rs	Adopt `CoreResult`; convert tests to `#[sinex_test]`.
crate/satellites/sinex-search-automaton/Cargo.toml	Add `sinex-test-utils` dev-dependency.
crate/satellites/sinex-fs-watcher/tests/unified_processor_tests.rs	Remove legacy config fields from tests.
crate/satellites/sinex-fs-watcher/tests/config_validation_tests.rs	Remove legacy validation tests tied to removed fields.
crate/satellites/sinex-fs-watcher/src/unified_processor.rs	Remove legacy config fields + update validation tests to `#[sinex_test]`.
crate/satellites/sinex-document-ingestor/tests/direct_ingestion_test.rs	Make runtime mutable for test flow.
crate/satellites/sinex-document-ingestor/tests/.gitkeep	Keep tests dir tracked.
crate/satellites/sinex-document-ingestor/Cargo.toml	Update description; drop unused deps; add dev-deps.
crate/satellites/sinex-desktop-satellite/Cargo.toml	Add `sinex-test-utils` dev-dependency.
crate/satellites/sinex-analytics-automaton/src/lib.rs	Adopt `CoreResult`; convert tests to `#[sinex_test]`.
crate/satellites/sinex-analytics-automaton/Cargo.toml	Add `sinex-test-utils` dev-dependency.
crate/lib/sinex-test-utils/tests/rstest_integration_example.rs	Switch tests to `TestResult`.
crate/lib/sinex-test-utils/tests/modern_test_infrastructure_example.rs	Switch tests to `TestResult`.
crate/lib/sinex-test-utils/tests/macro_conversion_examples.rs	Switch tests to `TestResult`; remove unused ctx in one test.
crate/lib/sinex-test-utils/tests/integration/stream_processing_test.rs	Switch to `TestResult`.
crate/lib/sinex-test-utils/tests/integration/multi_source_integration_test.rs	Switch to `TestResult`.
crate/lib/sinex-test-utils/tests/integration/configuration_test.rs	Switch to `TestResult`.
crate/lib/sinex-test-utils/tests/integration/collector_test.rs	Switch to `TestResult`.
crate/lib/sinex-test-utils/tests/fully_integrated_test.rs	Switch to `TestResult`; convert sync tests to `#[sinex_test]`.
crate/lib/sinex-test-utils/tests/database_pool_tests.rs	Improve error mapping when verifying pool state.
crate/lib/sinex-test-utils/tests/basic_enhanced_test.rs	Switch to `TestResult`.
crate/lib/sinex-test-utils/src/static_fixtures.rs	Update docs/paths; unify Result types; update tests to `TestResult`.
crate/lib/sinex-test-utils/src/resources.rs	Normalize to crate `Result<T>` alias signatures.
crate/lib/sinex-test-utils/src/lib.rs	Introduce `TestResult` alias + `ProptestCasesGuard`; export `TestResult` in prelude.
crate/lib/sinex-test-utils/src/fixture_generator.rs	Use `TestResult` for generator helpers and verification.
crate/lib/sinex-test-utils/src/error_testing.rs	Remove unused ctx param from async test.
crate/lib/sinex-test-utils/src/database_pool.rs	Increase default connection budget to align with Nix module min max_connections.
crate/lib/sinex-test-utils/src/channel_behavior_utils.rs	Switch one sync test to `TestResult`; remove unused ctx params in async tests.
crate/lib/sinex-test-utils/macros/src/lib.rs	Allow `TestResult` return type; relax proptest return-type enforcement.
crate/lib/sinex-test-utils/IMPROVEMENTS.md	Update paths to new crate locations; whitespace fix.
crate/lib/sinex-services/tests/service_integration_test.rs	Switch to `TestResult`.
crate/lib/sinex-services/tests/query_service_test.rs	Update embedded command string to `cargo nextest run`.
crate/lib/sinex-services/src/search.rs	Convert unit tests to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-schema/tests/validation_tests.rs	Switch to `TestResult`.
crate/lib/sinex-schema/tests/ulid_property_from_main_tests.rs	Convert property tests to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-schema/src/schema/mod.rs	Remove `sensd` schema exports.
crate/lib/sinex-schema/Cargo.toml	Add `tracing` dependency; formatting fix.
crate/lib/sinex-satellite-sdk/tests/system/external_test.rs	Switch to `TestResult`.
crate/lib/sinex-satellite-sdk/tests/stage_as_you_go_requires_db.rs	Update temporal_ledger column names and ULID cast syntax.
crate/lib/sinex-satellite-sdk/tests/replay_control.rs	Convert tests to `#[sinex_test]` and return `Result`.
crate/lib/sinex-satellite-sdk/tests/property/checkpoint_property_test.rs	Remove unused ctx param in async test.
crate/lib/sinex-satellite-sdk/tests/material_acquisition.rs	Fix move/borrow issues by cloning policy and reusing strings by ref.
crate/lib/sinex-satellite-sdk/tests/integration/satellite_lifecycle_test.rs	Switch to `TestResult`.
crate/lib/sinex-satellite-sdk/tests/integration/satellite_coordination_test.rs	Switch to `TestResult`.
crate/lib/sinex-satellite-sdk/tests/integration/satellite_architecture_test.rs	Switch to `TestResult`.
crate/lib/sinex-satellite-sdk/tests/integration/event_generation_test.rs	Switch to `TestResult`; update command strings.
crate/lib/sinex-satellite-sdk/tests/integration/critical_failure_modes_test.rs	Remove unused ctx param in some tests; keep Result types.
crate/lib/sinex-satellite-sdk/tests/integration/config_environment_validation_test.rs	Remove unused ctx param.
crate/lib/sinex-satellite-sdk/tests/integration/checkpoint_persistence_test.rs	Switch to `TestResult`.
crate/lib/sinex-satellite-sdk/tests/ingestion_helpers.rs	Convert sync tests to async `#[sinex_test]` returning Result.
crate/lib/sinex-satellite-sdk/src/stage_as_you_go.rs	Switch temporal_ledger insert to `query!` and updated column names/casts.
crate/lib/sinex-satellite-sdk/src/runtime/stream/runtime_state.rs	Remove job manager API from runtime state.
crate/lib/sinex-satellite-sdk/src/lease_manager.rs	Convert tests to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-satellite-sdk/src/jetstream_consumer.rs	Convert test to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-satellite-sdk/src/dlq_retry.rs	Convert test to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-satellite-sdk/src/confirmation_handler.rs	Convert tests to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-satellite-sdk/src/automaton_event_handler.rs	Convert tests to `#[sinex_test]` + `TestResult`.
crate/lib/sinex-satellite-sdk/src/acquisition_manager.rs	Switch ledger insert to `query!` and ULID cast syntax.
crate/lib/sinex-satellite-sdk/doc/overview.md	Remove gRPC mentions; document confirmations/DLQ subjects.
crate/lib/sinex-satellite-sdk/Cargo.toml	Add `libc` dependency.
crate/lib/sinex-processor-runtime/src/cli.rs	Expand heartbeat auto-spawn to Scan/Explore; add unit tests.
crate/lib/sinex-processor-runtime/Cargo.toml	Add `async-trait` dev-dependency.
crate/lib/sinex-core/tests/validation_core.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/validation_config.rs	Remove unused ctx params.
crate/lib/sinex-core/tests/unit/database_test.rs	Remove unused ctx param.
crate/lib/sinex-core/tests/system/temporal_chaos_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/system/stress_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/system/reliability_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/system/performance_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/system/mod.rs	Update documentation to prefer nextest commands.
crate/lib/sinex-core/tests/security/unicode_attack_test.rs	Switch helper result type to `TestResult`.
crate/lib/sinex-core/tests/security/ulid_attack_test.rs	Switch to `TestResult`; convert sync test to async sinex_test.
crate/lib/sinex-core/tests/security.rs	Remove unused ctx params.
crate/lib/sinex-core/tests/sanitization.rs	Remove unused ctx params.
crate/lib/sinex-core/tests/repositories_schema_management.rs	Convert sync test to async `#[sinex_test]`.
crate/lib/sinex-core/tests/repositories_common.rs	Remove unused ctx param.
crate/lib/sinex-core/tests/property/ulid_property_test.rs	Switch commented-out test signature to `TestResult`.
crate/lib/sinex-core/tests/property/time_range_property_test.rs	Use `#[sinex_test(proptest)]` and `TestResult`.
crate/lib/sinex-core/tests/property/path_sanitization_property_test.rs	Use `#[sinex_test(proptest)]` and `TestResult`.
crate/lib/sinex-core/tests/property_tests.rs	Switch property tests to `TestResult`.
crate/lib/sinex-core/tests/performance/resource_exhaustion_test.rs	Switch helper return types to `TestResult`; remove unused ctx params in benches.
crate/lib/sinex-core/tests/performance/regression_detection_test.rs	Remove unused ctx param from async test.
crate/lib/sinex-core/tests/performance/mod.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/performance/memory_usage_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/performance/large_payload_test.rs	Switch helper return types to `TestResult`; remove unused ctx params in benches.
crate/lib/sinex-core/tests/performance/jetstream_performance_test.rs	Switch helper return types to `TestResult`; remove unused ctx params in benches.
crate/lib/sinex-core/tests/performance/database_performance_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/performance/concurrent_load_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/performance/bottleneck_identification_test.rs	Remove unused ctx params in benches.
crate/lib/sinex-core/tests/payloads_clipboard.rs	Fix integer type hints for builder; remove unused ctx params.
crate/lib/sinex-core/tests/integration/test_automation_integration_test.rs	Switch helper and tests to `TestResult`.
crate/lib/sinex-core/tests/integration/provenance_test.rs	Switch to `TestResult`; remove unused imports; make one created event unused.
crate/lib/sinex-core/tests/integration/pipeline_integration_test.rs	Update embedded command string to `cargo nextest run`.
crate/lib/sinex-core/tests/integration/checkpoint_consistency_test.rs	Switch to `TestResult` for many helpers/tests.
crate/lib/sinex-core/tests/integration_tests.rs	Switch to `TestResult` in key tests.
crate/lib/sinex-core/tests/database.rs	Remove unused ctx param.
crate/lib/sinex-core/tests/concurrency/mod.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/concurrency/checkpoint_concurrency_test.rs	Switch helper return types to `TestResult`.
crate/lib/sinex-core/tests/adversarial/ulid_edge_cases_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/adversarial/security_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/adversarial/concurrency_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/adversarial/chaos_engineering_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/tests/adversarial/attack_simulation_test.rs	Switch to `TestResult`.
crate/lib/sinex-core/src/types/utils/timestamp_helpers.rs	Remove deprecated fallback timestamp converters.
crate/lib/sinex-core/src/types/events/test_helpers.rs	Update comment to sinex-core naming.
crate/lib/sinex-core/src/types/events/payloads/shell.rs	Remove unused import; enable test helpers for `test` builds too.
crate/lib/sinex-core/src/types/events/payloads/process.rs	Add typed ProcessStatus enum + degraded/failed payload types.
crate/lib/sinex-core/src/types/events/payloads/mod.rs	Remove re-export of internal macro.
crate/lib/sinex-core/src/types/events/payloads/filesystem.rs	Add `test_default` helpers for created/modified payloads.
crate/lib/sinex-core/src/db/repositories/state.rs	Add fallback operation start if stored procedure missing.
crate/lib/sinex-core/src/db/repositories/source_materials.rs	Convert updates to `sqlx::query!` with ULID/UUID casts.
crate/lib/sinex-core/src/db/repositories/events.rs	Fix ULID/UUID casting in insert; convert tests to `#[sinex_test]`.
crate/lib/sinex-core/src/db/mod.rs	Switch max_connections check to `query_scalar!` and parsing.
crate/lib/sinex-core/src/db/integrity.rs	Rename `IntegrityTestResults` to `IntegrityResults`.
crate/lib/sinex-core/src/db/distributed_locking.rs	Switch to sqlx macros; adjust advisory lock queries/returns.
crate/core/sinex-ingestd/tests/validator_schema_order_tests.rs	Fix validator usage (no `?` on non-Result validate call).
crate/core/sinex-ingestd/tests/config_security_tests.rs	Remove unused ctx params in config security tests.
crate/core/sinex-ingestd/src/jetstream_consumer.rs	Switch bulk insert to `query!` and owned buffers; adjust RETURNING parsing.
crate/core/sinex-ingestd/src/config.rs	Switch DB healthcheck query to `query!`.
crate/core/sinex-gateway/tests/service_container_test.rs	Switch to `TestResult`; adjust Result naming to avoid import conflict.
crate/core/sinex-gateway/tests/cascade_analyzer_tests.rs	Convert tests to `#[sinex_test]`.
crate/core/sinex-gateway/tests/cascade_analyzer_cycle_test.rs	Add prereq check for stored proc; skip tests when missing.
crate/core/sinex-gateway/src/lib.rs	Export new `native_messaging` module.
crate/core/sinex-gateway/src/handlers.rs	Convert tests to `#[sinex_test]` + `TestResult`.
crate/core/sinex-gateway/src/cascade_analyzer.rs	Convert tests to `#[sinex_test]` + `TestResult`.
crate/core/sinex-gateway/doc/rpc_server.md	Document new concurrency/timeout/body-size guards.
crate/core/sinex-gateway/doc/overview.md	Update example startup code to current RPC server API.
crate/core/sinex-gateway/Cargo.toml	Add tower features; add dev-deps (reqwest/which); enable hyper client feature.
.pre-commit-config.yaml	Remove legacy pre-commit config.
.gitignore	Ignore `.devenv/` directory.
.github/workflows/sqlx-check.yml	Pin Postgres image to timescaledb pg16.
.github/workflows/sqlx-cache.yml	Pin Postgres image to timescaledb pg16.
.github/workflows/README.md	Update local workflow execution notes; remove abstractions workflow mention.
.github/workflows/ci.yml	Install ripgrep; use reliable nextest profile; apply reliable profile to coverage.
.github/pull_request_template.md	Update checklists to current repo patterns (nextest, schema generation, repo boundaries).
.githooks/pre-commit	Remove legacy git hook script.
.env.example	Add tracked example env file for local DB defaults.
.config/nextest.toml	Use `num-cpus` in ci-parallel profile.
.claude/cclsp.json	Remove legacy Claude LSP config.
.cargo/config.toml	Add Postgres env defaults; adjust commentary; remove build jobs env var.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-09T09:58:04Z

+        let acquired: bool = sqlx::query_scalar!("SELECT pg_try_advisory_lock($1)", lock_id)
            .fetch_one(&mut *connection)
-            .await?;
+            .await?
+            .unwrap_or(false);



sqlx::query_scalar! for pg_try_advisory_lock should return a non-null bool, so the trailing .unwrap_or(false) is likely a compile error. Either remove the unwrap, or change the binding to Option<bool> (or adjust the SQL to explicitly force non-null) so the type matches.

Copilot · 2026-04-09T09:58:04Z

+        let acquired: bool = sqlx::query_scalar!("SELECT pg_try_advisory_lock($1)", lock_id)
            .fetch_one(&mut *connection)
-            .await?;
+            .await?
+            .unwrap_or(false);



Same issue as above: .unwrap_or(false) after query_scalar!("SELECT pg_try_advisory_lock...") will not compile if the macro returns bool (non-null). Consider removing the unwrap or explicitly selecting a nullable expression and binding to Option<bool>.

Copilot · 2026-04-09T09:58:04Z

+        let released: bool = sqlx::query_scalar!("SELECT pg_advisory_unlock($1)", lock_id)
            .fetch_one(&mut *connection)
-            .await?;
+            .await?
+            .unwrap_or(false);



sqlx::query_scalar! for pg_advisory_unlock should return a non-null bool, so .unwrap_or(false) is likely a compile error here as well. Remove the unwrap or bind to Option<bool> if you intentionally want null-tolerant behavior.

Copilot · 2026-04-09T09:58:05Z

    // Query PostgreSQL max_connections setting
-    let max_connections_row = sqlx::query("SHOW max_connections")
+    let max_connections_str = sqlx::query_scalar!("SHOW max_connections")
        .fetch_one(&temp_pool)
-        .await?;
+        .await?
+        .unwrap_or_else(|| "0".to_string());

-    let postgres_max_connections: i32 = max_connections_row.try_get("max_connections")?;
+    let postgres_max_connections: i32 = max_connections_str
+        .parse()
+        .map_err(|e| eyre!("Failed to parse max_connections value: {e}"))?;


sqlx::query_scalar!("SHOW max_connections") is expected to return a non-null String, so calling .unwrap_or_else(...) is likely a compile error. If you want a safe fallback, either bind to Option<String> (with a query that can be null) or just parse the returned String directly (or use current_setting('max_connections')).

Copilot · 2026-04-09T09:58:05Z

+    async fn fallback_start_replay_operation(
+        &self,
+        operator: &str,
+        scope: JsonValue,
+        _scope_window: Option<PgRange<DateTime<Utc>>>,
+    ) -> DbResult<Uuid> {
+        let uuid: Uuid = sqlx::query_scalar!(
+            r#"
+            INSERT INTO core.operations_log (
+                operation_type,
+                operator,
+                scope,
+                result_status
+            ) VALUES ($1, $2, $3::jsonb, 'running')
+            RETURNING id::uuid as "id!: Uuid"
+            "#,


fallback_start_replay_operation ignores the provided scope_window entirely. If the schema has core.operations_log.scope_window (it does in the canonical schema), this fallback should persist the window too so replay operations remain fully auditable/traceable when the stored procedure is missing.

Copilot · 2026-04-09T09:58:05Z

  postgresqlPkg = db.package;
-  postgresqlPackages =
-    if postgresqlPkg ? pkgs then postgresqlPkg.pkgs else pkgs.postgresql16Packages;
+  postgresqlPackages = pkgs.postgresql16Packages;

  extensionPackages = unique (
    optionals (postgresqlPackages ? timescaledb) [ postgresqlPackages.timescaledb ]
    ++ optionals (postgresqlPackages ? pgvector) [ postgresqlPackages.pgvector ]
+    ++ optionals (postgresqlPackages ? pg_jsonschema) [ postgresqlPackages.pg_jsonschema ]
    ++ optionals (postgresqlPackages ? pgx_ulid) [ postgresqlPackages.pgx_ulid ]
  );


postgresqlPackages is hard-coded to pkgs.postgresql16Packages even though services.postgresql.package is forced to db.package. If someone selects a non-16 PostgreSQL package, the extension packages (timescaledb/pgvector/pg_jsonschema/pgx_ulid) may be built against a different major version than the running server.

Copilot · 2026-04-09T09:58:05Z

  perServiceConnections = max 1 db.connectionPool.maxConnections;
+  # Ensure reasonable minimum for dev/test workloads (500) even with no services
+  # Production with services will compute higher based on service count
+  minConnectionsForTests = 500;
  baselineConnections = totalServiceCount * perServiceConnections + 50;
-  computedMaxConnections = max (perServiceConnections + 10) baselineConnections;
+  computedMaxConnections = max minConnectionsForTests (max (perServiceConnections + 10) baselineConnections);



computedMaxConnections now enforces a minimum of 500 connections for all configurations. This is likely too high for small/production hosts and can significantly increase Postgres memory usage. Consider making the minimum conditional on a dev/test mode (or configurable via an option), rather than hard-coding it as a global default.

Copilot · 2026-04-09T09:58:06Z

+    (mkIf (db.enable && db.autoSetup) {
+      systemd.services.postgresql-setup.script = lib.mkAfter ''
+        ensure_extension() {
+          local dbName="$1"
+          local extName="$2"
+          echo "[sinex] ensuring extension ''${extName} for ''${dbName}"
+          psql -v ON_ERROR_STOP=1 -d "$dbName" -c "CREATE EXTENSION IF NOT EXISTS \"$extName\"" >/dev/null
+        }
+
+        for dbName in ${concatStringsSep " " (map escapeShellArg allDatabases)}; do
+          ensure_extension "$dbName" "timescaledb"
+          ensure_extension "$dbName" "pg_jsonschema"
+          ensure_extension "$dbName" "vector"
+          ensure_extension "$dbName" "ulid"
+        done
+      '';


The postgresql-setup script unconditionally runs CREATE EXTENSION for timescaledb/pg_jsonschema/vector/ulid for every database. If any of these extensions aren’t available in the selected Postgres package, setup will fail due to ON_ERROR_STOP=1. Consider generating this loop from extensionPackages (or gating each extension on availability) so auto-setup remains robust.

…flows (#98) Tighten the local developer loop around SQLx/schema regeneration by standardizing Postgres defaults, cleaning stale scaffolding, and updating the surrounding scripts, docs, and CI guidance to match the current repository shape. local workflow and scripts: - Add `.env.example` and extend `scripts/dev-env-banner.sh` so the shell shows database readiness, expected process commands, and the environment variables developers are supposed to rely on instead of tribal knowledge. - Rework `.cargo/config.toml` to drop host-specific build tuning, export shared local Postgres defaults (`PGHOST`, `PGUSER`, `PGDATABASE`, `DATABASE_URL`), and keep the fast-path build flags explicit rather than hidden in per-machine setup. - Fix `scripts/smart-check.sh` and `scripts/smart-watch.sh` so they target actual package names instead of guessing from path fragments, which keeps incremental local checks from silently falling back to the wrong scope. - Expand `scripts/combine-files-batch.sh` with skip-path handling, optional symlink following, deterministic ordering, and extra outputs (`tokei_plus_gitlog.md`, diff shards) so local analysis bundles are reproducible and easier to regenerate after cache/schema churn. repository cleanup and docs: - Delete stale `.claude`/pre-commit artifacts, update `.github/pull_request_template.md`, refresh workflow docs, and rewrite the README sections that still described older architecture or workflow assumptions. - Move more docs onto the current Postgres/devenv/schema story, including the schema regeneration workflow, historical-context notes, and the NixOS database provisioning guidance. - Ignore `.devenv/` state and refresh the lockfile after the toolchain/dependency churn so local setup stops producing noisy, unexplained diffs. CLI, gateway, runtime, and tests: - Extend `cli/exo.py` with better RPC fallback guidance plus aggregated DLQ metrics and related operator output, which makes the CLI more useful when the RPC layer or local database is unavailable. - Carry follow-on gateway/runtime/schema/test cleanups that were exposed by the regenerated cache work, including replay/native-messaging support, typed SQLx checks, processor-runtime/checkpoint coverage, and shared harness updates. - Re-enable mold/aggressive linker flags and raise local Postgres defaults for parallel nextest once the earlier “drop host-specific tuning” pass made those knobs explicit and reviewable.

Sinity and others added 30 commits November 12, 2025 20:30

chore(sqlx): commit regenerated SQLX cache files

5fbc5b0

Add the current `.sqlx` query metadata needed for Nix and offline validation after the recent schema and query changes. Co-Authored-By: Claude <noreply@anthropic.com>

fix(scripts): improve smart-check and smart-watch crate targeting

7875c8f

Teach the helper scripts to derive more accurate changed-crate sets so incremental check and watch runs follow the files that actually changed.

refactor(scripts): make bundle batching deterministic

83ae67d

- Add skip-path handling, optional symlink following, and explicit output ordering to `scripts/combine-files-batch.sh`. - Keep regenerated source/test/doc bundles stable between runs so analysis artifacts stop churning.

ci(workflows): rewrite PR guidance around nextest and schemas

5429f79

- Update workflow docs and the PR template to match the current nextest/schema regeneration flow. - Remove older checklist language that still assumed earlier abstraction and test commands.

docs(readme): clarify architecture and VM workflow sections

0331bd5

Tighten the top-level architecture description and quick-start guidance so the README matches the JetStream-first runtime and current VM test commands.

feat(cli): add DLQ metrics and actionable RPC fallback hints

df9b427

- Extend `exo` with aggregated DLQ metrics and clearer RPC/database failure guidance. - Make operator output more useful when the gateway or local database is unavailable.

feat(gateway): add replay control RPCs and native messaging

32cdaa1

- Extend the gateway control surface with replay RPC methods. - Carry the native-messaging support changes needed to exercise the same path locally.

feat(processor-runtime): extend checkpoint parsing and runner coverage

0fe68f8

Expand the processor-runtime CLI around checkpoint parsing and scan control, and add runner tests that exercise the updated execution paths.

refactor(schema): tune migrations, exports, and tests

863db64

Adjust the canonical migration, clean up schema exports, and refresh the serde, validation, and ULID test surface around the schema crate.

refactor(test-utils): tighten fixtures, benches, and helper docs

1095190

- Clean up fixture ergonomics, benchmark support, and helper documentation in `sinex-test-utils`. - Keep the shared harness aligned with the current local workflow after the shell/database cleanup.

test(document-ingestor): clean up direct-ingestion coverage

0fc2612

Create a crate-local document-ingestor test entry point and trim the older scattered setup so direct-ingestion behavior is easier to exercise.

refactor(fs-watcher): simplify config validation and related tests

3d6fddc

Collapse the filesystem watcher validation logic into `FilesystemConfig` itself and align the config and unified-processor tests with the tighter bounds.

feat(system-satellite): expand watcher coverage and tests

e7db94b

Build on the watcher scaffolding by wiring more system watcher behavior, exporting the related modules, and adding processor/systemd integration coverage.

refactor(terminal-satellite): refine the unified processor and histor…

81e962d

…y watcher state Rework terminal processor state, history-watcher context, command processing, and tests so the terminal satellite continues its move toward a single unified processor path.

docs(schema): document the schema regeneration workflow

bc09e85

Add `schemas/README.md` to explain the generated JSON payload schemas, how they are produced, and how to validate them after code changes.

docs(historical): annotate legacy design documents

3a0f819

Add historical context to the older vision and analysis documents so readers can distinguish archived planning from the active architecture.

Sinity and others added 9 commits November 12, 2025 21:28

refactor(nixos): tighten database provisioning modules

67a145c

Adjust the flake wiring, database module defaults, module imports, and secrets plumbing so Sinex services provision PostgreSQL and related state more predictably.

test: move CLI and Nix integration checks onto sinex_test_utils

9353c30

Rewrite the top-level CLI and Nix smoke tests to use `TestContext` / `TestResult` conventions and add DLQ plus confirmation command coverage that matches the surrounding CLI work.

refactor(core): reconcile schema, replay, and repository surfaces

ccac7ba

- Sweep the core schema/replay/repository layers to keep their APIs consistent after the regeneration and workflow changes. - Trim mismatched naming and stale assumptions instead of carrying them into later branches.

build: re-enable mold and the aggressive linker flags

e2ee5d7

Turn the faster linker and related build settings back on in `.cargo/config.toml` after the earlier CI and toolchain adjustments settled.

Sinity added this to the Era 5: Schema, SQLx, and CI hardening milestone Apr 9, 2026

Sinity added type:chore Build, CI, tooling, or dependency changes area:core sinex-core: event model, schema, replay area:gateway sinex-gateway: JSON-RPC query/control labels Apr 9, 2026

Copilot AI review requested due to automatic review settings April 9, 2026 09:50

Sinity added area:cli sinexctl: CLI tool area:schema Schema, migrations, database area:nix NixOS modules, flake, packaging area:tests Test infrastructure and harnesses area:docs Documentation and architecture docs labels Apr 9, 2026

Copilot started reviewing on behalf of Sinity April 9, 2026 09:51

View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Sinity added the faux-pull-request PR created as part of a history rewrite process label Apr 10, 2026

Sinity closed this Apr 10, 2026

Sinity deleted the feature/chore/sqlx-cache-local-postgres branch April 16, 2026 00:19

Navigation Menu

Repository navigation

chore: regenerate SQLx/cache artifacts and harden local Postgres workflows - #98

chore: regenerate SQLx/cache artifacts and harden local Postgres workflows#98
Sinity wants to merge 39 commits into
masterfrom
feature/chore/sqlx-cache-local-postgres

Sinity commented Apr 9, 2026

Uh oh!

chatgpt-codex-connector Bot commented Apr 9, 2026

Uh oh!

coderabbitai Bot commented Apr 9, 2026

Review skipped

`.coderabbit.yaml` has unrecognized properties

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Sinity commented Apr 9, 2026

Summary

Motivation

Changes grouped by concern

1. Make the local shell and database story explicit

2. Fix the scripts developers actually lean on

3. Remove stale repository scaffolding

4. Carry the follow-on operator/runtime cleanups that the branch exposed

Testing

Notes

Uh oh!

chatgpt-codex-connector Bot commented Apr 9, 2026

Uh oh!

coderabbitai Bot commented Apr 9, 2026

Review skipped

.coderabbit.yaml has unrecognized properties

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`.coderabbit.yaml` has unrecognized properties