Repository navigation
Conversation
…maries Add a workspace rustc wrapper that rejects direct test builds by default, expand nextest timeout tuning, and teach schema synchronization to report created, updated, and unchanged payload schemas.
Remove the rustc-wrapper test block and update the workspace guidance so nextest remains the supported path without hard-disabling native cargo test runs.
…st/docs markdown packs Add `scripts/combine-files-batch.sh`, a repo-wide bundling utility that classifies text files into sources/tests/docs, applies explicit exclusion globs, sorts by path-priority heuristics, and writes `combined-*.md` reports with metadata headers, token estimates, table-of-contents anchors, and fenced file contents. The script is designed for offline review/analysis workflows where large coherent snapshots are needed without interactive prompts. `.gitignore` is updated to ignore generated `combined-bundles/` output. Net effect: repeatable one-command packaging of project surfaces for audit and context-transfer tasks.
Merge the consecutive atoms 293-a, 293-b into one semantic transition. Use the merge to absorb tightly-coupled fixups, docs, and test fallout that only exist to complete the same behavioral cutover. This merged commit belongs to the runtime_replay_jetstream_band band and should read as one forward-moving change instead of a staircase of tiny follow-ups.
- Move the analytics and search processors onto the shared runtime/service shape. - Normalize the surrounding wiring so replay and query work can reuse the same processor boundaries.
Move the hand-written payload builder methods into the proc macro so the payload modules stop carrying duplicated with_* and build helper implementations.
…search Introduce reusable Pagination and TimeRange helpers, move search filtering into EventRepository, and update the affected services and tests to use the shared query types.
Track the remaining ingestd, ledger, cascade, and replay SQL call sites that still need to move under repository or Postgres-side helpers.
…ries Add the shared `Pagination` type, move repository call sites onto it, and replace the remaining SeaQuery-based time-series paths with direct repository SQL. Update the surrounding integration coverage to match the new paging and query surface.
…y helpers Introduce reusable fixture constants in `sinex-test-utils`, then modernize the core query helpers and related event tests to consume the shared data.
Switch the integration tests to the shared constant set so repository and type-safety expectations stop re-declaring the same fixture values.
…factor Extend replay control with explicit range handling, keep the cascade logic on the repository side, and refresh the generated SQLx metadata for the new query surface.
Fold the cascade query helpers into the event repository itself, rewire the gateway analyzer to use the new home, and update the repository migration plan.
…y/SQL and reliability gaps Create `docs/TODO.md` as a concrete post-survey action ledger rather than a high-level note. The file enumerates 17 scoped tasks across gateway hardening, blob/content pipeline wiring, system watcher integration, heartbeat quality, encryption rollout, schema compatibility tooling, and test restoration. Each item includes target files, implementation steps, and fail-first validation expectations. In sequence context this commit documents unfinished follow-up work immediately before broader SeaQuery-helper removal, preserving an executable checklist so cleanup and reliability debt stays visible and test-driven.
Delete the unused `seaquery_helpers` module and trim the corresponding exports and tests now that the repository layer no longer depends on those wrappers.
Drop unnecessary `sea-query` dependencies from `sinex-core`, `sinex-services`, and `sinex-test-utils`, leaving the schema crate as the only consumer while the SQL cleanup continues.
Replace the large flake shell hook and `justfile` task layer with a dedicated `devenv.nix` so packages, tasks, and processes live in one development-environment definition.
- Remove the obsolete abstractions workflow, add `scripts/check_forbidden_patterns.sh`, and regenerate schemas during CI validation. - Pin the Timescale image and move test execution onto `cargo nextest` so schema drift is caught before merge.
…t code Delete the old sensd schema and sensor/job-management infrastructure, and remove the remaining document-ingestor and SDK hooks that depended on the pre-JetStream pipeline.
… and move schema workflows onto sinex-schema CLI Rebuild the entire `schemas/v1` artifact set from current Rust `EventPayload` definitions, including broad source/event namespace normalization (dot-style event filenames, reorganized source directories, and removal of stale legacy contracts), plus many newly emitted payload schemas. Add `schemas/v1/registry.json` as a generated manifest of `(source, event_type, version, path, content_hash)` entries for drift tracking and deterministic sync. Update `schemas/README.md` to mark JSON files as generated outputs, and rewrite helper scripts (`schema-dev`, `deploy-schemas`, `check-schema-compatibility`) to use `sinex-schema` CLI generate/sync/validate flows with explicit DB/env preflight checks instead of older ad-hoc SQL and global-install behavior.
- Add targeted regressions around replay-control resilience and blob-route behavior. - Keep the failures explicit so the remaining hardening work is tracked by executable tests instead of TODO text.
- Update the architecture/progress documents to match the schema CLI, replay control surface, and the current repository layout. - Remove language that still implied parallel sensd-era paths were active.
Add workflow checks that validate generated schema artifacts stay current and update the architecture note to state that Rust payload definitions remain the source of truth.
Sinity
added
area:core
Sinity
added
type:feat
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Sinity
added a commit
that referenced
this pull request
Apr 9, 2026
Move schema generation, replay control, and repository querying onto explicit source-of-truth paths so generated contracts, database state, and CLI control surfaces stop drifting. sinex-core: - Add `crate/lib/sinex-core/src/db/repositories/schema_management.rs::sync_discovered_schemas()` and `SchemaSyncResult` so schema discovery can compare content hashes against `sinex_schemas.event_payload_schemas` and report created/updated/unchanged rows instead of blind upserts. - Add `crate/lib/sinex-core/src/types/query.rs` with shared `Pagination` and `TimeRange` types, then thread them through `db/repositories/events.rs` and related repository helpers so event search and time-series queries stop carrying bespoke limit/offset logic. - Add `crate/lib/sinex-core/src/db/replay/state_machine.rs` and move replay state transitions into `sinex-core`, which gives the gateway a durable operation model instead of ad-hoc metadata blobs and inline SQL. - Replace leftover SeaQuery-driven time-series code in `db/repositories/events.rs` and remove the obsolete helper paths, which narrows dynamic SQL to repository code that is already covered by the core query layer. sinex-gateway and CLI: - Add `crate/core/sinex-gateway/src/replay_control.rs` and wire new replay handlers through `handlers.rs`, `rpc_server.rs`, and `service_container.rs` so replay planning, preview, approval, execution, cancellation, and status all ride a single `sinex.control.replay` control path. - Rewrite `cli/replay_commands.py` around the RPC-backed control plane and extend `cli/rpc_client.py` with replay methods, which turns `exo replay` into a stateful operator surface instead of a local planner with direct database assumptions. - Move cascade expansion helpers toward repository-backed execution in `cascade_analyzer.rs`, reducing hand-built SQL in the gateway and keeping event-topology logic closer to `EventRepository`. schema workflow, tooling, and docs: - Move schema generation/sync into `crate/lib/sinex-core/src/types/bin/sinex-schema.rs`, regenerate `schemas/v1/**` plus `registry.json`, and update `schemas/README.md`/`docs/way_2.md` to say `derive(EventPayload)` definitions are authoritative while `schemas/` stays generated output. - Add `scripts/check_forbidden_patterns.sh`, update `.github/workflows/ci.yml` and schema validation jobs to regenerate schemas and fail on a dirty `schemas/` tree, which makes schema drift visible in CI instead of after deployment. - Migrate local workflow setup to `devenv.nix`, remove the old abstractions workflow, and prune the remaining sensd/job-management scaffolding from the SDK/schema/ingestor surface so the repository describes one active architecture instead of a half-retired dual path.
|
Important Review skippedToo many files! This PR contains 180 files, which is 30 over the limit of 150. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: ⛔ Files ignored due to path filters (120)
📒 Files selected for processing (180)
You can disable this status message by setting the Use the checkbox below for a quick retry:
Note
|
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR makes schema generation/sync and replay orchestration first-class by centralizing “source-of-truth” behavior in sinex-core, updating callers to shared query types, and enforcing generated schema lock-step in CI while continuing the cleanup away from SeaQuery-era scaffolding.
Changes:
- Introduces shared
Pagination/TimeRangequery types and migrates repository callers/tests to use them. - Adds schema discovery sync reporting (
SchemaSyncResult) and updates schema/CI workflows to treatschemas/as generated artifacts. - Adds gateway replay control RPC surface and related client/server plumbing; removes additional legacy SeaQuery/sensor/job-manager remnants.
Reviewed changes
Copilot reviewed 147 out of 308 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| crate/satellites/sinex-analytics-automaton/Cargo.toml | Adds lib config and SQLx dependency for automaton |
| crate/lib/sinex-test-utils/tests/integration/event_processing_integration_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-test-utils/tests/fully_integrated_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-test-utils/src/test_context.rs | Adds backward-compatible pool accessor |
| crate/lib/sinex-test-utils/src/satellite_runtime.rs | Refactors test runtime wiring and types |
| crate/lib/sinex-test-utils/src/property_testing.rs | Updates queries to shared Pagination |
| crate/lib/sinex-test-utils/src/lib.rs | Exposes new modules/constants/runtime helpers |
| crate/lib/sinex-test-utils/src/fixtures.rs | Updates queries to shared Pagination |
| crate/lib/sinex-test-utils/src/db_common.rs | Normalizes ULID binding/casts for bootstrap cleanup |
| crate/lib/sinex-test-utils/src/database_pool.rs | Adds ULID shim installation + extension checks updates |
| crate/lib/sinex-test-utils/src/constants.rs | New shared fixture identifiers/constants |
| crate/lib/sinex-test-utils/IMPROVEMENTS.md | Updates docs note about event ID casting |
| crate/lib/sinex-test-utils/Cargo.toml | Removes SeaQuery dependency |
| crate/lib/sinex-services/tests/service_integration_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-services/Cargo.toml | Removes SeaQuery dependency |
| crate/lib/sinex-schema/src/schema/processors.rs | Adds ScopeWindow column definition |
| crate/lib/sinex-schema/doc/schema_design.md | Updates schema docs for id column naming |
| crate/lib/sinex-satellite-sdk/tests/support/mod.rs | Re-exports shared runtime from test-utils |
| crate/lib/sinex-satellite-sdk/tests/stage_as_you_go_requires_db.rs | Adds tests for stage-as-you-go DB coupling |
| crate/lib/sinex-satellite-sdk/tests/replay_stage_flow.rs | Updates replay tests for progress callback and runtime config |
| crate/lib/sinex-satellite-sdk/tests/processor_shutdown_leak_test.rs | Adds shutdown/background task regression test |
| crate/lib/sinex-satellite-sdk/tests/material_acquisition.rs | Adds concurrent acquisition isolation test |
| crate/lib/sinex-satellite-sdk/tests/job_manager_sensors.rs | Removes legacy job-manager sensor tests |
| crate/lib/sinex-satellite-sdk/tests/integration/checkpoint_persistence_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-satellite-sdk/tests/heartbeat_metrics_regression.rs | Adds heartbeat metrics/logging regression tests |
| crate/lib/sinex-satellite-sdk/tests/config_loading_tests.rs | Renames config fields in tests |
| crate/lib/sinex-satellite-sdk/src/sensors/tree_watch.rs | Removes legacy sensor implementation |
| crate/lib/sinex-satellite-sdk/src/sensors/mod.rs | Removes legacy sensor module exports |
| crate/lib/sinex-satellite-sdk/src/sensors/append_stream.rs | Removes legacy sensor implementation |
| crate/lib/sinex-satellite-sdk/src/replay/service.rs | Switches replay queries to Pagination |
| crate/lib/sinex-satellite-sdk/src/lib.rs | Removes job_manager/sensors exports; adds heartbeat sink export |
| crate/lib/sinex-satellite-sdk/src/jetstream_consumer.rs | Removes unused test-only confirmed handler scaffold |
| crate/lib/sinex-satellite-sdk/doc/README.md | Updates test runtime documentation reference |
| crate/lib/sinex-macros/src/event_payload.rs | Adds builder method generation in derive macro |
| crate/lib/sinex-core/tests/unit_tests.rs | Updates queries to shared Pagination |
| crate/lib/sinex-core/tests/unit/event_type_system_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-core/tests/unit/database_test.rs | Updates docs/examples for Pagination |
| crate/lib/sinex-core/tests/system/temporal_chaos_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-core/tests/system/stress_test.rs | Fixes SQLx typing + affected rows handling |
| crate/lib/sinex-core/tests/system/performance_test.rs | Updates SQL to use id column |
| crate/lib/sinex-core/tests/security/unicode_attack_test.rs | Updates SQL to use id column and ULID casts |
| crate/lib/sinex-core/tests/security/ulid_attack_test.rs | Adds ULID collision and time-skew security tests |
| crate/lib/sinex-core/tests/seaquery_helpers.rs | Removes SeaQuery helper tests |
| crate/lib/sinex-core/tests/repositories_common.rs | Replaces SeaQuery builder usage with raw SQL string |
| crate/lib/sinex-core/tests/process_status_test.rs | Adds invalid enum string rejection test |
| crate/lib/sinex-core/tests/performance/database_performance_test.rs | Updates primary key query to use id/ULID cast |
| crate/lib/sinex-core/tests/performance/checkpoint_performance_test.rs | Updates checkpoint load API usage |
| crate/lib/sinex-core/tests/integration/work_queue_test.rs | Improves JetStream error handling and ACK API usage |
| crate/lib/sinex-core/tests/integration/validation_cache_test.rs | Migrates to injected TestContext + pool usage |
| crate/lib/sinex-core/tests/integration/timestamp_test.rs | Updates queries to shared Pagination |
| crate/lib/sinex-core/tests/integration/test_automation_integration_test.rs | Standardizes Result import + Pagination |
| crate/lib/sinex-core/tests/integration/subscription_service_test.rs | Standardizes Result import |
| crate/lib/sinex-core/tests/integration/state_management_test.rs | Standardizes Result import |
| crate/lib/sinex-core/tests/integration/single_writer_enforcement_test.rs | Standardizes Result import and prelude usage |
| crate/lib/sinex-core/tests/integration/resource_management_test.rs | Standardizes Result import |
| crate/lib/sinex-core/tests/integration/pipeline_integration_test.rs | Refactors concurrency setup + Pagination |
| crate/lib/sinex-core/tests/integration/ingest_service_test.rs | Standardizes Result import + Pagination |
| crate/lib/sinex-core/tests/database.rs | Updates queries to shared Pagination |
| crate/lib/sinex-core/tests/concurrency/checkpoint_concurrency_test.rs | Fixes SQLx ULID typing for checkpoint helpers |
| crate/lib/sinex-core/tests/adversarial/ulid_edge_cases_test.rs | Fixes SQLx ULID typing in adversarial test |
| crate/lib/sinex-core/tests/adversarial/enhanced_boundary_test.rs | Refactors concurrency test to propagate errors |
| crate/lib/sinex-core/tests/adversarial/boundary_test.rs | Updates SQL to use id column |
| crate/lib/sinex-core/src/types/query.rs | Adds shared Pagination and TimeRange types |
| crate/lib/sinex-core/src/types/mod.rs | Exposes query module and re-exports types |
| crate/lib/sinex-core/src/types/events/payloads/window.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/system.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/shell.rs | Removes ad-hoc builders; keeps/adjusts impl stubs |
| crate/lib/sinex-core/src/types/events/payloads/rpc.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/process.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/filesystem.rs | Removes ad-hoc builders; trims test helpers |
| crate/lib/sinex-core/src/types/events/payloads/document.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/desktop.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/clipboard.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/types/events/payloads/blob.rs | Removes ad-hoc builder impls (macro-generated now) |
| crate/lib/sinex-core/src/lib.rs | Removes SeaQuery re-exports; adds EventRepositoryTx export |
| crate/lib/sinex-core/src/db/seaquery_helpers.rs | Removes SeaQuery ULID helper module |
| crate/lib/sinex-core/src/db/repositories/state.rs | Adds replay scope window range + ULID generation typing |
| crate/lib/sinex-core/src/db/repositories/schema_management.rs | Adds discovered-schema sync and sync result reporting |
| crate/lib/sinex-core/src/db/repositories/mod.rs | Exposes new repository types/rows |
| crate/lib/sinex-core/src/db/repositories/events_extensions.rs | Updates method signature to accept Pagination |
| crate/lib/sinex-core/src/db/repositories/common.rs | Refactors EventSearchFilters to new query surfaces |
| crate/lib/sinex-core/src/db/replay/mod.rs | Adds replay state machine module re-exports |
| crate/lib/sinex-core/src/db/query_helpers.rs | Replaces SeaQuery usage with SQLx QueryBuilder |
| crate/lib/sinex-core/src/db/mod.rs | Adds integrity/validation modules; removes SeaQuery helpers |
| crate/lib/sinex-core/Cargo.toml | Removes SeaQuery; adds features + dev dependencies |
| crate/core/sinex-ingestd/tests/pipeline_resilience_test.rs | Adds ingest pipeline latency/idempotency resilience tests |
| crate/core/sinex-ingestd/tests/jetstream_e2e_integration_test.rs | Updates queries to shared Pagination |
| crate/core/sinex-ingestd/tests/figment_config_tests.rs | Ensures env isolation for config loading test |
| crate/core/sinex-ingestd/src/prelude.rs | Updates schema/validation re-exports |
| crate/core/sinex-ingestd/src/lib.rs | Re-exports sync result and schema info from core |
| crate/core/sinex-ingestd/src/jetstream_consumer.rs | Simplifies payload validation handling |
| crate/core/sinex-ingestd/src/config.rs | Fixes config file merging behavior |
| crate/core/sinex-gateway/tests/replay_control_serialization_tests.rs | Adds replay control JSON round-trip tests |
| crate/core/sinex-gateway/tests/replay_control_resilience_test.rs | Adds replay control availability/regression test |
| crate/core/sinex-gateway/tests/native_messaging_auth_test.rs | Adds extension allow-list enforcement test |
| crate/core/sinex-gateway/tests/gateway_secret_management_test.rs | Adds admin token file env requirement test |
| crate/core/sinex-gateway/tests/cascade_analyzer_cycle_test.rs | Expands cascade analyzer test coverage |
| crate/core/sinex-gateway/tests/blob_route_security_test.rs | Adds unauth/quota blob route security test |
| crate/core/sinex-gateway/tests/blob_event_forwarding_test.rs | Adds blob event forwarding regression test |
| crate/core/sinex-gateway/src/service_container.rs | Adds replay state machine + replay control wiring |
| crate/core/sinex-gateway/src/rpc_server.rs | Adds replay RPC method dispatch and client retrieval |
| crate/core/sinex-gateway/src/main.rs | Registers new replay modules |
| crate/core/sinex-gateway/src/lib.rs | Exposes replay control module |
| crate/core/sinex-gateway/src/handlers.rs | Adds replay RPC handlers + parsing helpers |
| crate/core/sinex-gateway/Cargo.toml | Adds async-nats/uuid/futures deps; adjusts dev-deps |
| cli/tests/test_replay_planner.py | Adds replay planner tests (including stub expectations) |
| cli/tests/test_dlq_cli_commands.py | Adds CLI command existence tests |
| cli/tests/test_cli_error_guidance.py | Adds CLI error guidance tests for 401/429 |
| cli/rpc_client.py | Adds replay control RPC client methods |
| cli/exo.py | Updates SQL to use id column instead of event_id |
| cli/completion.py | Updates completion query to use id column |
| TESTING.md | Updates dev/test guidance to devenv/nextest |
| README.md | Updates dev shell + task guidance; adds DB defaults notes |
| Cargo.toml | Re-enables automaton crates in workspace |
| .github/workflows/sqlx-check.yml | Runs migrations via schema crate; uses nextest |
| .github/workflows/sqlx-cache.yml | Updates migration path to schema crate |
| .github/workflows/schema-validation.yml | Pins Timescale version; validates schema CLI/generation |
| .github/workflows/schema-management.yml | Generates schemas via script; deploy gating improvements |
| .github/workflows/schema-compatibility.yml | Updates schema compatibility checks to new paths/scripts |
| .github/workflows/ci.yml | Adds forbidden-pattern + schema regen enforcement; uses nextest |
| .github/workflows/abstractions.yml | Removes old abstraction workflow |
| .config/nextest.toml | Formatting-only blank line additions |
| .cargo/config.toml | Adds rustc workspace wrapper; tweaks env section placement |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| pipelining = true | ||
| # Use all available CPU cores for compilation | ||
| jobs = 24 | ||
| rustc-workspace-wrapper = "scripts/rustc_wrapper.sh" |
There was a problem hiding this comment.
The PR description notes that the rustc wrapper-based enforcement was backed out, but the workspace config re-introduces a rustc-workspace-wrapper. If this wrapper is only meant for CI enforcement, gate it behind an env var/profile-specific config (or remove it from repo-wide config) to avoid surprising local builds and platform-specific failures when the script is missing/non-executable.
Comment on lines
+32
to
+36
| def test_replay_planner_nats_target_publishes(): | ||
| planner = ReplayPlanner(db_url="postgresql://ignored") | ||
|
|
||
| planner._replay_event(_sample_event(), "nats") | ||
| pytest.fail("Replay planner's NATS target is still a stub; expected it to publish events") |
There was a problem hiding this comment.
This test is guaranteed to fail unconditionally due to the final pytest.fail(...). If the intent is to document an unimplemented feature without failing CI, mark it as xfail/skipped (with a tracking issue) or assert the current stub behavior instead; otherwise, implement the NATS publish path and assert on an observable effect (e.g., a mocked publisher call).
Suggested change
| def test_replay_planner_nats_target_publishes(): | |
| planner = ReplayPlanner(db_url="postgresql://ignored") | |
| planner._replay_event(_sample_event(), "nats") | |
| pytest.fail("Replay planner's NATS target is still a stub; expected it to publish events") | |
| @pytest.mark.xfail( | |
| reason="Replay planner's NATS target is not implemented yet; replace with an observable publish assertion once available", | |
| strict=False, | |
| ) | |
| def test_replay_planner_nats_target_publishes(): | |
| planner = ReplayPlanner(db_url="postgresql://ignored") | |
| planner._replay_event(_sample_event(), "nats") |
Comment on lines
+85
to
+96
| let control_client = match spawn_replay_control( | ||
| replay.clone(), | ||
| &std::env::var("SINEX_NATS_URL").unwrap_or_else(|_| "nats://localhost:4222".into()), | ||
| ) | ||
| .await | ||
| { | ||
| Ok(client) => Some(client), | ||
| Err(err) => { | ||
| warn!(error = %err, "Replay control bus disabled (NATS connection failed)"); | ||
| None | ||
| } | ||
| }; |
There was a problem hiding this comment.
The container currently downgrades replay-control startup failures to None while continuing to return Ok(ServiceContainer { .. }). That contradicts the expectation encoded in replay_control_resilience_test (service creation should fail when replay control is unavailable) and also risks silently running without the control plane. Decide on one contract: either (mandatory) return an error here when replay control is required, or (optional) keep it optional but update the test and make downstream RPC methods return a clearer, actionable error when replay_control is None.
Suggested change
| let control_client = match spawn_replay_control( | |
| replay.clone(), | |
| &std::env::var("SINEX_NATS_URL").unwrap_or_else(|_| "nats://localhost:4222".into()), | |
| ) | |
| .await | |
| { | |
| Ok(client) => Some(client), | |
| Err(err) => { | |
| warn!(error = %err, "Replay control bus disabled (NATS connection failed)"); | |
| None | |
| } | |
| }; | |
| let control_client = Some( | |
| spawn_replay_control( | |
| replay.clone(), | |
| &std::env::var("SINEX_NATS_URL") | |
| .unwrap_or_else(|_| "nats://localhost:4222".into()), | |
| ) | |
| .await | |
| .map_err(|err| { | |
| SinexError::service("Failed to start replay control") | |
| .with_source(err.to_string()) | |
| })?, | |
| ); |
Comment on lines
+4
to
+11
| fn gateway_requires_admin_token_secret() -> color_eyre::eyre::Result<()> { | ||
| let secret = std::env::var("SINEX_GATEWAY_ADMIN_TOKEN_FILE"); | ||
| assert!( | ||
| secret.is_ok(), | ||
| "Gateway should source admin tokens via agenix-managed files (set SINEX_GATEWAY_ADMIN_TOKEN_FILE)" | ||
| ); | ||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
This test will fail in any environment that doesn't set SINEX_GATEWAY_ADMIN_TOKEN_FILE (including typical CI runners) regardless of whether the runtime code correctly handles secrets. Consider turning this into either (a) a config validation unit test that exercises your config loader with a temporary env var, or (b) an integration test gated behind #[ignore], a feature flag, or a CI-only environment guard.
Comment on lines
+13
to
+23
| fn require_git_annex() { | ||
| if which::which("git-annex").is_err() { | ||
| panic!("git-annex must be installed to run blob route security tests"); | ||
| } | ||
| } | ||
|
|
||
| #[sinex_test] | ||
| async fn blob_routes_should_enforce_auth_and_quota( | ||
| ctx: TestContext, | ||
| ) -> color_eyre::eyre::Result<()> { | ||
| require_git_annex(); |
There was a problem hiding this comment.
Hard-panicking when git-annex is absent makes this test suite environment-dependent and can break CI or contributor workflows. Prefer conditionally skipping (e.g., return early with Ok(()) after eprintln! / tracing::warn!) or gating behind an #[ignore]/feature so the absence of an external binary doesn't fail unrelated changes.
Suggested change
| fn require_git_annex() { | |
| if which::which("git-annex").is_err() { | |
| panic!("git-annex must be installed to run blob route security tests"); | |
| } | |
| } | |
| #[sinex_test] | |
| async fn blob_routes_should_enforce_auth_and_quota( | |
| ctx: TestContext, | |
| ) -> color_eyre::eyre::Result<()> { | |
| require_git_annex(); | |
| fn require_git_annex() -> bool { | |
| which::which("git-annex").is_ok() | |
| } | |
| #[sinex_test] | |
| async fn blob_routes_should_enforce_auth_and_quota( | |
| ctx: TestContext, | |
| ) -> color_eyre::eyre::Result<()> { | |
| if !require_git_annex() { | |
| eprintln!( | |
| "skipping blob route security test because git-annex is not installed" | |
| ); | |
| return Ok(()); | |
| } |
Comment on lines
+33
to
+35
| if matches!(until, TimeHorizon::Continuous) { | ||
| self.running.store(true, Ordering::SeqCst); | ||
| } |
There was a problem hiding this comment.
The test sets running = true during scan(...) but HangingProcessor does not override shutdown() to reset it. Unless the trait default shutdown() mutates this field (unlikely), this assertion is not satisfiable and will fail deterministically. Implement shutdown() in HangingProcessor to clear the flag (or change the test to assert on an observable shutdown side-effect that the runtime actually provides).
Comment on lines
+96
to
+101
| processor.shutdown().await?; | ||
|
|
||
| assert!( | ||
| !processor.running.load(Ordering::SeqCst), | ||
| "Continuous scanners should clear their running state during shutdown" | ||
| ); |
There was a problem hiding this comment.
The test sets running = true during scan(...) but HangingProcessor does not override shutdown() to reset it. Unless the trait default shutdown() mutates this field (unlikely), this assertion is not satisfiable and will fail deterministically. Implement shutdown() in HangingProcessor to clear the flag (or change the test to assert on an observable shutdown side-effect that the runtime actually provides).
Comment on lines
+38
to
+43
| let metrics = emitter.create_heartbeat_metrics(None); | ||
|
|
||
| assert!( | ||
| metrics.cpu_usage_percent > 0.0, | ||
| "Heartbeat metrics should report actual CPU usage instead of the current hard-coded 0.0 value" | ||
| ); |
There was a problem hiding this comment.
Asserting cpu_usage_percent > 0.0 is inherently flaky even with a correct implementation (CPU sampling can legitimately yield ~0 for short intervals, CI can run on constrained/virtualized environments, and scheduling noise dominates). Consider injecting a metrics provider/clock into HeartbeatEmitter so tests can deterministically supply a non-zero reading, or assert on structural properties (e.g., field present and within [0, 100]) rather than a strictly-positive runtime value.
Comment on lines
+35
to
+42
| assert!( | ||
| default_limit > 0, | ||
| "default pagination limit must be positive" | ||
| ); | ||
| assert!( | ||
| max_limit >= default_limit, | ||
| "max pagination limit must be >= default limit" | ||
| ); |
There was a problem hiding this comment.
Pagination is documented as clamping invalid inputs, but these assert!s will panic at runtime for invalid bounds. Since with_default/with_bounds are public APIs, prefer returning a Result<Self, SinexError> (or using debug_assert! with a safe fallback) so callers don't get unexpected process-level panics from configuration/user-provided defaults.
Comment on lines
+36
to
+38
| static BOOTSTRAP_MATERIAL_ID: Lazy<Ulid> = Lazy::new(|| { | ||
| Ulid::from_str("014D2PF2DBSQQZXQ5TK1V58CGG").expect("valid bootstrap material id") | ||
| }); |
There was a problem hiding this comment.
This introduces a duplicated bootstrap ULID constant (a similar BOOTSTRAP_MATERIAL_ID exists in src/db_common.rs). To avoid divergence, centralize this identifier in one place (e.g., sinex_test_utils::constants) and reference it from both modules.
Suggested change
| static BOOTSTRAP_MATERIAL_ID: Lazy<Ulid> = Lazy::new(|| { | |
| Ulid::from_str("014D2PF2DBSQQZXQ5TK1V58CGG").expect("valid bootstrap material id") | |
| }); | |
| static BOOTSTRAP_MATERIAL_ID: Lazy<Ulid> = | |
| Lazy::new(|| (*crate::db_common::BOOTSTRAP_MATERIAL_ID).clone()); |
Sinity
added
the
faux-pull-request
Sinity
added a commit
that referenced
this pull request
Jul 13, 2026
Move schema generation, replay control, and repository querying onto explicit source-of-truth paths so generated contracts, database state, and CLI control surfaces stop drifting. sinex-core: - Add `crate/lib/sinex-core/src/db/repositories/schema_management.rs::sync_discovered_schemas()` and `SchemaSyncResult` so schema discovery can compare content hashes against `sinex_schemas.event_payload_schemas` and report created/updated/unchanged rows instead of blind upserts. - Add `crate/lib/sinex-core/src/types/query.rs` with shared `Pagination` and `TimeRange` types, then thread them through `db/repositories/events.rs` and related repository helpers so event search and time-series queries stop carrying bespoke limit/offset logic. - Add `crate/lib/sinex-core/src/db/replay/state_machine.rs` and move replay state transitions into `sinex-core`, which gives the gateway a durable operation model instead of ad-hoc metadata blobs and inline SQL. - Replace leftover SeaQuery-driven time-series code in `db/repositories/events.rs` and remove the obsolete helper paths, which narrows dynamic SQL to repository code that is already covered by the core query layer. sinex-gateway and CLI: - Add `crate/core/sinex-gateway/src/replay_control.rs` and wire new replay handlers through `handlers.rs`, `rpc_server.rs`, and `service_container.rs` so replay planning, preview, approval, execution, cancellation, and status all ride a single `sinex.control.replay` control path. - Rewrite `cli/replay_commands.py` around the RPC-backed control plane and extend `cli/rpc_client.py` with replay methods, which turns `exo replay` into a stateful operator surface instead of a local planner with direct database assumptions. - Move cascade expansion helpers toward repository-backed execution in `cascade_analyzer.rs`, reducing hand-built SQL in the gateway and keeping event-topology logic closer to `EventRepository`. schema workflow, tooling, and docs: - Move schema generation/sync into `crate/lib/sinex-core/src/types/bin/sinex-schema.rs`, regenerate `schemas/v1/**` plus `registry.json`, and update `schemas/README.md`/`docs/way_2.md` to say `derive(EventPayload)` definitions are authoritative while `schemas/` stays generated output. - Add `scripts/check_forbidden_patterns.sh`, update `.github/workflows/ci.yml` and schema validation jobs to regenerate schemas and fail on a dirty `schemas/` tree, which makes schema drift visible in CI instead of after deployment. - Migrate local workflow setup to `devenv.nix`, remove the old abstractions workflow, and prune the remaining sensd/job-management scaffolding from the SDK/schema/ingestor surface so the repository describes one active architecture instead of a half-retired dual path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I consolidated three threads that had been drifting independently: generated payload schemas, replay orchestration, and repository-backed query/cascade logic. The center of gravity now sits in
sinex-core: schema discovery can sync itself intosinex_schemas.event_payload_schemas, replay state lives in a reusable state machine, and pagination/time-range query types are shared instead of being redefined in each caller. On the gateway side I replaced the old local replay planner flow with a control-plane client/server pair backed bysinex.control.replay, so the CLI and the service now speak the same vocabulary for plan/preview/approve/execute/status. I also used the branch to finish removing leftover SeaQuery and sensd-era scaffolding, regenerate the v1 payload contracts, and make CI treatschemas/as generated output that must stay in lock-step with the Rust payload definitions.The branch looks broad because these pieces were entangled in practice. Schema generation was still half file-oriented and half database-oriented, replay control lived in gateway-specific code with direct SQL, and some of the query/cascade surfaces still reflected earlier transport and repository experiments. I wanted the squash to leave behind a single, explicit architecture rather than land a narrow schema feature on top of old indirection.
Motivation
The immediate problem was schema drift. Rust
EventPayloadderives, generated JSON files underschemas/, and rows insinex_schemas.event_payload_schemaswere all being treated as important, but none of them was cleanly authoritative. That makes simple questions harder than they should be: when a payload changes, which artifact should change first, who writes the database row, and how do we tell whether the database is stale or just equivalent?Replay control had a similar problem. The gateway had useful replay/cascade work in progress, but it was trapped in a planner/executor split that assumed local control and mixed raw SQL, transient state, and CLI-specific presentation. At the same time,
sinex-corealready had the right place to own operation state and event topology. I wanted to move replay orchestration onto that core surface before more operators or automata started depending on the gateway’s internal shape.There was one clear false start during the branch: I briefly blocked direct
cargo testinvocations with arustcwrapper and then backed that out in the very next commit when it became obvious that the enforcement was too blunt for local work. I kept the useful testing/config work, but not the wrapper-based policy. That pivot is why the branch starts with a testing/schema commit and then moves quickly toward repository, replay, and CI changes.Changes grouped by concern
1. Schema generation and database sync
The most important addition is the explicit sync path in
SchemaManagementRepository:Instead of treating schema sync as “rewrite whatever is in the database,” the repository now loads the active schema map, computes content hashes for discovered schemas, and records whether each item was created, updated, or left unchanged. That gives the CLI and future automation something meaningful to report, and it prevents churn when the generated content is equivalent.
I paired that with a stronger generator in
crate/lib/sinex-core/src/types/bin/sinex-schema.rs. The generator now emits the schema files and the registry manifest from the discovered Rust payload contracts, and it can sync the discovered set into Postgres when asked. The docs now say the quiet part out loud: Rust payload definitions are the source of truth, whileschemas/v1/**exists as generated distribution material for GitOps and downstream tooling.I also regenerated the v1 payload contracts and added the registry manifest in the same branch so the code, generated JSON, and CI expectations all agree on day one.
2. Replay control and cascade execution
The gateway now has a real replay control surface rather than a local-only planner.
crate/core/sinex-gateway/src/replay_control.rsintroduces a NATS-backed control client/server pair that speaks in terms ofReplayOperation,ReplayScope, and the replay state machine that now lives insinex-core.On the CLI side,
cli/replay_commands.pywas rewritten around that RPC model. The replay UX is now explicitly stateful:That matters because it gives us a single operational contract. The gateway owns the control plane, the CLI becomes a client of that plane, and the persistent replay state sits in
sinex-corerather than inside the CLI’s process.I also pushed cascade helpers toward repository-backed execution. The analyzer still has gateway-specific presentation logic, but the actual event graph operations now sit closer to
EventRepository, which is where we can reuse them for other control or repair flows later.3. Query and repository cleanup
I added shared
PaginationandTimeRangetypes undercrate/lib/sinex-core/src/types/query.rsand used them to simplify event search and time-series queries. This is not just tidiness: it removes one more reason for services and CLI code to carry their own query math and one more excuse for raw SQL to creep back into callers.I also removed more of the leftover SeaQuery-era code. The branch doesn’t claim that every repository path is perfect now, but it does narrow the active data-access surface to the repository layer that
sinex-corealready owns. That makes follow-on work like repository cacheing, state helpers, and schema management much easier to reason about.4. CI, devenv, and architecture cleanup
Once schemas became explicitly generated, CI had to enforce that contract. The GitHub Actions changes now regenerate schemas during validation and fail if the working tree under
schemas/changes. I also addedscripts/check_forbidden_patterns.shso the branch can remove old workflow/documentation patterns and keep them from silently reappearing.I migrated the local workflow to
devenv.nixin the same branch because the schema and replay work kept touching the same developer loops: generate schemas, run validation, run tests, inspect the DB. The repo is easier to understand when there is a single supported shell and a single supported generation path.Finally, I removed the last bits of sensd and legacy job-management scaffolding that were still sitting beside the active JetStream/repository architecture. That cleanup is important context for the branch: the schema source-of-truth work would still be confusing if the repo continued to present multiple half-active ingestion/control paths.
Testing
I covered the branch in a few layers.
The core repository and query changes are exercised through updated integration suites that were switched onto shared fixture constants, plus the new replay/core repository plumbing that lands under
sinex-coreand the gateway replay handlers. The schema pipeline is covered both by the generator/sync path itself and by the CI enforcement that regenerates schemas and rejects stale checked-in artifacts. The gateway side also gained fail-first regression coverage for still-open gaps such as replay resilience and blob-route security so those behaviors are documented as executable expectations rather than TODO text.I’m calling out that last part deliberately: this squash includes some tests that describe gaps rather than close them. That was intentional. I wanted the branch to leave the repository with clearer architecture and sharper failure signals, not just silently move code around.
Notes
I reverted the direct
cargo testblocking wrapper almost immediately; the enforcement goal was valid, but that mechanism was too invasive for local work.The branch also intentionally spans code, generated artifacts, and docs. I did that because schema source-of-truth only becomes real when all three agree: the Rust derive path, the generated
schemas/tree, and the CI/docs language contributors see first.The next useful follow-up is to keep pulling ad-hoc SQL out of replay/checkpoint edge cases and finish consolidating any remaining schema cache reads behind repository helpers. This branch gets the architectural center right; the remaining work is mostly cleanup around that center.
Ref #95