Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ npm install @pondpilot/flowscope-core
Analyze a query:

```typescript
import { initWasm, analyzeSql } from '@pondpilot/flowscope-core';
import { analyzeSql, initWasm, nodesInStatement } from '@pondpilot/flowscope-core';

await initWasm();

Expand All @@ -145,7 +145,13 @@ const result = await analyzeSql({
dialect: 'postgres',
});

console.log(result.statements[0]);
console.log('All graph nodes:', result.nodes);

const statement = result.statements[0];
console.log(
'First statement nodes:',
nodesInStatement(result, statement.statementIndex)
);
```

## Completion API
Expand Down
57 changes: 44 additions & 13 deletions crates/flowscope-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Core SQL lineage analysis engine for FlowScope.

## Structure

```
```text
src/
├── analyzer.rs # Main analysis orchestration
├── analyzer/
Expand Down Expand Up @@ -58,30 +58,61 @@ use flowscope_core::{analyze, AnalyzeRequest, Dialect};
fn main() {
let request = AnalyzeRequest {
sql: "SELECT u.name, o.id FROM users u JOIN orders o ON u.id = o.user_id".to_string(),
files: None,
dialect: Dialect::Postgres,
schema: None, // Optional schema metadata
file_path: None,
source_name: Some("example.sql".to_string()),
options: None,
schema: None,
#[cfg(feature = "templating")]
template_config: None,
};

let result = analyze(&request);

// Access table lineage
for statement in result.statements {
println!("Tables: {:?}", statement.nodes);
println!("Edges: {:?}", statement.edges);
// The complete graph spans every statement.
println!("Nodes: {:?}", result.nodes);
println!("Edges: {:?}", result.edges);

// Use the helpers to inspect one statement's portion of the flat graph.
for statement in &result.statements {
let nodes: Vec<_> = result
.nodes_in_statement(statement.statement_index)
.collect();
let edges: Vec<_> = result
.edges_in_statement(statement.statement_index)
.collect();
println!("Statement {} nodes: {nodes:?}", statement.statement_index);
println!("Statement {} edges: {edges:?}", statement.statement_index);
}
}
```

### Linting

```rust
use flowscope_core::linter::{Linter, LintConfig, LintDocument};

let config = LintConfig::default();
let linter = Linter::new(config);
let document = LintDocument::new(sql, dialect);
let issues = linter.check_document(&document);
use flowscope_core::{analyze, AnalysisOptions, AnalyzeRequest, Dialect, LintConfig};

let request = AnalyzeRequest {
sql: "select * from users".to_string(),
files: None,
dialect: Dialect::Postgres,
source_name: None,
options: Some(AnalysisOptions {
lint: Some(LintConfig::default()),
..Default::default()
}),
schema: None,
#[cfg(feature = "templating")]
template_config: None,
};

let result = analyze(&request);
let lint_issues: Vec<_> = result
.issues
.iter()
.filter(|issue| issue.code.starts_with("LINT_"))
.collect();
println!("Lint issues: {lint_issues:?}");
```

## Testing
Expand Down
2 changes: 2 additions & 0 deletions crates/flowscope-core/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#![doc = include_str!("../README.md")]

pub mod analyzer;
pub mod completion;
pub mod error;
Expand Down
6 changes: 6 additions & 0 deletions crates/flowscope-wasm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ When `options.lint.enabled` is `true`, lint diagnostics are included in the `iss
```json
{
"statements": [ ... ],
"nodes": [ ... ],
"edges": [ ... ],
"issues": [ ... ],
"summary": {
"hasErrors": false,
Expand All @@ -38,6 +40,10 @@ When `options.lint.enabled` is `true`, lint diagnostics are included in the `iss
}
```

`statements` contains per-statement metadata. The flat lineage graph lives in
the top-level `nodes` and `edges` arrays; each graph item lists its participating
statements in `statementIds`.

### `analyze_sql(sql: string) -> string`

**Legacy/Deprecated.** Simple API that takes a raw SQL string and returns a basic JSON list of tables. Use `analyze_sql_json` for full features.
Expand Down
14 changes: 14 additions & 0 deletions docs/api-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,20 @@ export interface StatementMeta {
}
```

`StatementMeta` contains metadata only. Use the public helpers to project the
top-level graph down to a statement without reimplementing the `statementIds`
filter:

```typescript
import { edgesInStatement, nodesInStatement } from '@pondpilot/flowscope-core';

for (const statement of result.statements) {
const nodes = nodesInStatement(result, statement.statementIndex);
const edges = edgesInStatement(result, statement.statementIndex);
console.log({ statement, nodes, edges });
}
```

### Node & Edge

```typescript
Expand Down
20 changes: 15 additions & 5 deletions docs/guides/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@ yarn add @pondpilot/flowscope-core
## Basic Usage

```typescript
import { initWasm, analyzeSql } from '@pondpilot/flowscope-core';
import {
analyzeSql,
edgesInStatement,
initWasm,
nodesInStatement,
} from '@pondpilot/flowscope-core';

await initWasm();

Expand Down Expand Up @@ -52,12 +57,17 @@ if (result.summary.hasErrors) {
console.error('Analysis failed:', result.issues);
}

for (const stmt of result.statements) {
console.log(`Statement ${stmt.statementIndex}: ${stmt.statementType}`);
console.log('Edges:', stmt.edges.length);
for (const statement of result.statements) {
const nodes = nodesInStatement(result, statement.statementIndex);
const edges = edgesInStatement(result, statement.statementIndex);

console.log(`Statement ${statement.statementIndex}: ${statement.statementType}`);
console.log('Nodes:', nodes.length);
console.log('Edges:', edges.length);
}

console.log('Global nodes:', result.globalLineage.nodes.length);
console.log('All graph nodes:', result.nodes.length);
console.log('All graph edges:', result.edges.length);
```

## Disabling Column Lineage
Expand Down
18 changes: 17 additions & 1 deletion packages/core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,30 @@ npm install @pondpilot/flowscope-core
## Usage

```typescript
import { initWasm, analyzeSql } from '@pondpilot/flowscope-core';
import {
analyzeSql,
edgesInStatement,
initWasm,
nodesInStatement,
} from '@pondpilot/flowscope-core';

await initWasm();

const result = await analyzeSql({
sql: 'SELECT * FROM users',
dialect: 'duckdb',
});

console.log('All graph nodes:', result.nodes);
console.log('All graph edges:', result.edges);

for (const statement of result.statements) {
console.log({
metadata: statement,
nodes: nodesInStatement(result, statement.statementIndex),
edges: edgesInStatement(result, statement.statementIndex),
});
}
```

Bundlers such as Vite resolve the package-owned WASM URL automatically. Pass
Expand Down
2 changes: 1 addition & 1 deletion packages/core/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"test:wasm-browser": "node ../../scripts/test_wasm_browser.mjs",
"test:watch": "vitest",
"clean": "rm -rf dist",
"typecheck": "yarn check:generated-types && tsc --noEmit",
"typecheck": "yarn check:generated-types && tsc --noEmit && tsc --project tsconfig.type-tests.json",
"lint": "eslint src",
"lint:fix": "eslint src --fix"
},
Expand Down
6 changes: 5 additions & 1 deletion packages/core/src/analyzer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -213,12 +213,16 @@ async function ensureWasmReady(): Promise<void> {
*
* @example
* ```typescript
* import { analyzeSql, nodesInStatement } from '@pondpilot/flowscope-core';
*
* const result = await analyzeSql({
* sql: 'SELECT * FROM users JOIN orders ON users.id = orders.user_id',
* dialect: 'postgres'
* });
*
* console.log(result.statements[0].nodes); // Tables: users, orders
* console.log(result.nodes.map((node) => node.label)); // Complete flat graph
* const statement = result.statements[0];
* console.log(nodesInStatement(result, statement.statementIndex));
* console.log(result.summary.hasErrors); // false
* ```
*/
Expand Down
46 changes: 46 additions & 0 deletions packages/core/tests/documentation.type-test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
import {
analyzeSql,
edgesInStatement,
nodesInStatement,
type AnalyzeRequest,
type AnalyzeResult,
} from '../src';

async function documentationExample(): Promise<void> {
const request = {
sql: 'SELECT * FROM users JOIN orders ON users.id = orders.user_id',
dialect: 'postgres',
sourceName: 'example.sql',
} satisfies AnalyzeRequest;

const result = await analyzeSql(request);

console.log(result.nodes, result.edges);
for (const statement of result.statements) {
console.log(
nodesInStatement(result, statement.statementIndex),
edgesInStatement(result, statement.statementIndex)
);

// @ts-expect-error StatementMeta contains metadata, not graph collections.
console.log(statement.nodes);
// @ts-expect-error StatementMeta contains metadata, not graph collections.
console.log(statement.edges);
}

// @ts-expect-error The flat graph replaces the removed globalLineage wrapper.
console.log(result.globalLineage);
}

const staleRequest: AnalyzeRequest = {
sql: 'SELECT 1',
dialect: 'postgres',
// @ts-expect-error Use sourceName; filePath is not a current request field.
filePath: 'example.sql',
};

declare const typedResult: AnalyzeResult;
void typedResult.nodes;
void typedResult.edges;
void documentationExample;
void staleRequest;
8 changes: 8 additions & 0 deletions packages/core/tsconfig.type-tests.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": true,
"rootDir": "."
},
"include": ["tests/documentation.type-test.ts"]
}
6 changes: 6 additions & 0 deletions packages/core/wasm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ When `options.lint.enabled` is `true`, lint diagnostics are included in the `iss
```json
{
"statements": [ ... ],
"nodes": [ ... ],
"edges": [ ... ],
"issues": [ ... ],
"summary": {
"hasErrors": false,
Expand All @@ -38,6 +40,10 @@ When `options.lint.enabled` is `true`, lint diagnostics are included in the `iss
}
```

`statements` contains per-statement metadata. The flat lineage graph lives in
the top-level `nodes` and `edges` arrays; each graph item lists its participating
statements in `statementIds`.

### `analyze_sql(sql: string) -> string`

**Legacy/Deprecated.** Simple API that takes a raw SQL string and returns a basic JSON list of tables. Use `analyze_sql_json` for full features.
Expand Down
Loading