Skip to content

feat(locate): report semantic column references for safe drops #236

Description

@orian

Problem

After #234 and #235, table/column lookup can answer which resolved models contain a selected column. It cannot answer which modeled objects and expressions reference that column.

A user preparing to drop a known resolved column needs a reverse-reference report across every manifest composition, layer, and node dump. Text search is insufficient: it produces unrelated matches, misses semantic references through qualification and aliases, and cannot account for inheritance, patches, overrides, or drops.

The current validator dependency graph is primarily object-level. Diff and plan can emit a DROP COLUMN operation without proving that the column is unused.

Proposed interface

Extend locate with a reference mode:

hclexp locate \
  -manifest manifest.hcl \
  -layer-root ./schema \
  -tables posthog.events \
  -columns person_properties \
  -references \
  -format json

Both selectors remain required. Reference mode should report reverse references to every selected resolved column rather than only reporting model membership.

Model-loading semantics

Reference lookup must retain the model-first behavior:

  1. A manifest produces one resolved model per role/environment stack.
  2. The comma-separated -layer list is loaded in declared order as one resolved model.
  3. Every HCL file under -dump is loaded as an independent node model.
  4. Independent models load through the existing bounded parallel worker pool. A 30-node cluster must be loaded concurrently, while output remains deterministic.
  5. References are analyzed only after composition and resolution, so inherited, patched, modified, overridden, and dropped state is respected.

References to cover

Within a table:

  • primary_key and order_by
  • partition_by, sample_by, and table TTL
  • column DEFAULT, MATERIALIZED, EPHEMERAL, ALIAS, and column TTL expressions
  • indexes
  • projections
  • CHECK and ASSUME constraints
  • engine arguments that name columns

Across objects:

  • materialized-view and view queries, including qualified names and aliases
  • SELECT * and table.* wildcard projections
  • materialized-view destination shape
  • Distributed proxy/remote column contracts
  • Buffer and TimeSeries forwarding/target contracts where applicable
  • dictionary source queries and attribute expressions where they can reference the selected table column

The analysis must resolve SQL scopes rather than matching identifier text. The same column name on an unrelated table is not a reference.

Provenance and output

Each result should identify:

  • target database, table, and column
  • model source: manifest role/environment, ordered layer stack, or dump file/node
  • referring object type and qualified name
  • reference kind, such as order_by, projection, view_query, or distributed_contract
  • relevant expression or query
  • authored file, line, and layer when source provenance is available

References shared by several compositions should be deduplicated as source sites while retaining every affected model placement.

JSON output should be stable and machine-readable. It should distinguish definite references from unresolved safety blockers.

Fail-closed behavior

The command must not claim that a drop is safe when analysis is incomplete.

  • Raw SQL that may reference the target is an explicit unresolved blocker.
  • Unparseable or ambiguously scoped SQL is an explicit unresolved blocker.
  • SELECT * and table.* are explicit references unless the analyzer can prove otherwise.
  • A selector pattern matching no target in any supplied model is an error, preventing typo-based false safety.
  • External application queries are outside the model and must be called out in the result/documentation.

The strongest valid conclusion is no references found in the supplied model universe, not globally safe to drop.

Suggested exit behavior:

  • 0: every selector matched and no modeled references or unresolved blockers were found
  • 1: references/blockers exist, a selector matched nothing, or model loading/analysis failed
  • 2: invalid flag combination

Acceptance criteria

  • End-to-end manifest test where a column is inherited from a base layer and referenced from different files/layers and role/environment compositions.
  • End-to-end test covering references from table expressions, projections, constraints, views, materialized views, and proxy contracts.
  • Negative test proving the same identifier on an unrelated table is not reported.
  • Wildcard, raw SQL, ambiguous query, and unparseable query tests demonstrate fail-closed output.
  • Patch/override/drop tests demonstrate that only references present in each final resolved model are attributed to that model.
  • A subprocess end-to-end test covers text and JSON output plus exit status.
  • A 30-node dump test preserves the parallel-loading guarantee and deterministic output.

Follow-up to #234 and #235.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions