Repository navigation
feat(dbt): manifest.json -> SysGraph (digital-thread box-1 front-end) - #30
Merged
Merged
Conversation
Add DbtManifest, DbtManifestMetadata, DbtNode, and DbtDependsOn structs with serde deserialization support for parsing dbt manifest.json artifacts. Includes test fixture validating parsing of metadata, model nodes, and source nodes.
- Add DbtLiftConfig placeholder struct for future configuration options
- Add dbt_element_id() to convert dbt unique_id to ElementId with "dbt:" prefix
- Add lower_nodes() to transform qualifying nodes (models, seeds, snapshots, sources)
into OntologicalNode entries in a SysGraph with UfoStereotype::Kind("DbtModel")
- Add tests for node lowering and resource type filtering
- Fix import order (UfoStereotype after ontology, before sysgraph) - Break long dep_exists line to stay within rustfmt's line width
…nodes C1 (critical): lower_edges walked manifest.nodes unfiltered and checked dep_exists against the unfiltered maps too, while lower_nodes only lowers model/seed/snapshot/source resource types. Real manifests have test.* nodes whose depends_on.nodes point at models, so lower_edges emitted edges from nodes lower_nodes never created (and, symmetrically, could emit an edge to a present-but-non-qualifying node like analysis.*) -- SysGraph::dangling_edges was non-empty on any real-world manifest. Both functions now derive their node set from a single qualifying_nodes() helper; DanglingDependency still hard-errors, just against the correct set. Also, per this review round: - I1: fixed the two cargo fmt diffs in this file. - I2: added a REALISTIC_BUILD_FIXTURE (source + dependent model + a test.* node depending on the model) and a regression test asserting dangling_edges().is_empty() -- the shape every prior fixture lacked. - I3/I4: documented every public item and field, and expanded the module doc comment with the v12-only, dbt: identity, hard-error, and acquisition-neutral decisions a consumer needs. - I5: implemented the spec's promised debug-only label suffix (`name (schema)`), which also resolves the FIXTURE's own label collision between model.jaffle_shop.customers and source.jaffle_shop.raw.customers (both previously labeled "customers").
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements kr0ki's
docs/superpowers/specs/2026-09-20-dbt-manifest-digital-thread-design.md(piece 1 ofPLAN-KR0KI-006) — a newufo_types::dbtmodule that parses a dbtmanifest.jsonartifact and lowers it intoufo_types::sysgraph::SysGraph, the crate's existing canonical UFO semantic graph. Unconditional (no feature flag — no risky dependency, unlikereqif).Built via subagent-driven development: 4 tasks (parsing,
Upgradeimpl for dbt's own versioned wire schema, node lowering, edge lowering + entry point), each independently reviewed, plus a final whole-branch review that caught and fixed one real cross-task bug before merge (see below).Architecture
impl Upgrade for DbtManifest— not onSysGraphdirectly (Rust coherence only allows oneimpl Upgrade for SysGraphacross the whole crate, and Kubernetes/Rust-source front-ends will need that slot too). This is the crate's first realUpgradeconsumer.ElementIdisdbt:{unique_id}— no new provenance field added anywhere.parse_and_lift(bytes, config) -> Result<SysGraph, DbtLiftError>is acquisition-neutral — bytes in, no file I/O/network/credentials.ontology.rsorsysgraph.rs.Caught by the final whole-branch review (fixed before this PR)
Node lowering filtered to qualifying resource types (model/seed/snapshot/source); edge lowering didn't apply the same filter — any real dbt manifest with
test.*nodes produced dangling edges. Fixed by deriving both passes from one shared qualifying-node set; added a regression fixture shaped like realdbt buildoutput. Also fixed in the same pass: acargo fmtgap, a documentation pass (this module now matchesreqif.rs's doc-comment bar), and the spec's promised debug-only label suffix (disambiguates same-named models/sources by schema).Test plan
cargo test— 263/263 passcargo clippy --all-targets -- -D warnings— cleancargo fmt --all -- --check— clean