Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ members = [
"crates/pyrefly_types",
"crates/pyrefly_util",
"crates/tsp_types",
"static_analysis",
"static_analysis/pyrefly_exports",
"static_analysis/spmd",
"static_analysis/lifeguard",
"pyrefly",
"pyrefly_wasm",
]
Expand Down
4 changes: 3 additions & 1 deletion pyrefly/lib/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ mod compat;
#[doc(hidden)]
pub mod embed;
mod error;
mod export;
// Experimental export-only adapter for the static_analysis spike. This is not a stable API.
#[doc(hidden)]
pub mod export;
#[doc(hidden)]
pub mod lsp;
mod memory_project;
Expand Down
20 changes: 20 additions & 0 deletions static_analysis/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
[package]
name = "pyrefly_static_analysis"
version = "0.0.0"
edition = "2024"
license.workspace = true
repository.workspace = true
publish = false
description = "Experimental neutral Python frontend (scopes, operation CFG) and generic dataflow engine"

[dependencies]
pyrefly_python = { path = "../crates/pyrefly_python" }
pyrefly_util = { path = "../crates/pyrefly_util" }
ruff_python_ast = "0.0.14"
ruff_python_parser = "0.0.14"
ruff_source_file = "0.0.14"
ruff_text_size = "0.0.14"
starlark_map = "0.14.2"

[lints]
workspace = true
48 changes: 48 additions & 0 deletions static_analysis/DESIGN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Shared static-analysis frontend: experimental design

This branch is an open-source feasibility spike. The question is whether two substantially different analyses can share Python scope, evaluation-order, and flow machinery—not whether this branch replaces either production checker.

## Dependency direction

```text
Python syntax and lexical ownership
|
ordered operations, regions, CFG, symbol history
|
+----------+--------------+
| | |
SPMD Lifeguard history queries
domain domain without a domain
```

Pyrefly's type-specific `Bindings` and `Answers` are not the shared intermediate representation. A future Pyrefly binding emitter could consume this frontend and add inference dependencies downstream. The experimental export adapter depends on the existing checker crate but the core does not.

## Shared work versus domain work

The structured driver owns statement-level structure over the parsed AST: suite order, the decomposition of `if`/`elif`/`else` and loops, which arms run given a backend's static truth for their tests, and the loop that a `break` or `continue` leaves. Lowering is a backend of the driver that emits blocks and terminators. A backend owns its values and its policy for deciding tests; the driver never evaluates a test itself.

The frontend owns declarations, scope visibility, evaluation order, deferred versus immediate code regions, and explicit flow edges. Statement ranges provide a coarse grouping of ordered expression operations. Consumers do not reparse source or reconstruct scopes by walking the AST.

The engine owns frame bookkeeping, branch/loop propagation, joins, convergence limits, and deterministic diagnostic collection. A domain defines abstract values plus arbitrary state: heap/typestate facts, capability state, ambient meshes, effects, or other context. An operation may change existing facts; it need not have a useful return value.

Assignment preserves a value reference, not a link that follows later rebinding of another name. Lexical symbol ownership, reaching definitions, and object aliasing are distinct. Domains that track mutable objects must preserve object identities and account for escapes. Captured bindings and globals are activation-time cells, not permanently frozen definition-time values.

Ambient context is carried and joined along paths, rather than kept as a process-global mode. Scoped entry/exit and call contracts belong in analysis semantics. Unknown context or differing predecessor contexts must not be resolved by choosing one predecessor arbitrarily.

The reusable declaration scanner and lexical navigation engine are separate from a backend's value tables. Runtime ownership requests unpruned syntax; configured/type-aware declaration views can select positions and branches through explicit policy hooks. Name occurrences carry a syntactic role and do not stand in for runtime reads. Candidate payloads, initialization and type-specific filtering stay in the consumer.

Optional pre-check feedback is a separate interface keyed by source-qualified structural points. The facts interface carries explicit assumptions, diagnostic coverage and observed coverage; a conditional typing result is not unconditional execution proof. Point IDs are not CFG-block identities, and providers must reject stale source while consumers arrange fresh dependency/configuration snapshots. The shared frontend does not choose narrowing or merge policies when applying such facts.

## Expression ordering and inference ordering

Expression-flow consumption is useful for custom type systems, but runtime evaluation order is not type-inference dependency order. Ordinary assignment evaluates RHS before target expressions; augmented assignment evaluates its target/read before the RHS. Short-circuit operators and conditional expressions contain control flow.

A downstream type checker may introduce contextual/deferred dependencies without changing the shared execution structure. A statement-level client may summarize expression behavior; it must not treat a statement as effect-free merely because its interior is unexpanded.

## Evidence needed for generality

The consumers must both use the same neutral operations and engine. SPMD exercises tensor/gradient contracts and active-mesh context. Lifeguard exercises imported-object provenance, mutation, deferred function effects, and call summaries. A history-only client and a small typestate test probe additional reuse without adding a third production analysis.

Unsupported Python features and unconverged analyses are explicit incomplete outcomes. Passing a selected fixture subset is not full parity, and neither domain provides universal formal verification of arbitrary Python or custom kernels.

Temporary adapters into complete Pyrefly/Lifeguard forks are possible follow-up experiments; production migration is outside this spike.
108 changes: 108 additions & 0 deletions static_analysis/PYREFLY_ADAPTER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Pyrefly bindings as a downstream consumer

## Goal and evidence boundary

The long-term capability target is an exact match or superset of today's binding pass, without making `Bindings` or `Answers` the shared frontend API. Adoption by Pyrefly remains optional: a substantial performance cost could make sharing undesirable even if it is technically possible.

**The current spike does not establish that capability target.** Its neutral scopes, regions, ordered operations and CFG provide a plausible substrate, but its runtime-focused IR discards information required by the actual bindings pass. This document is an implementation sketch and gap map, not a completed migration or a performance result.

Existing Pyrefly references below are against upstream commit `0424eebb19345ca0b40727a15cfca9e759c05d6b`. The spike's files are the working implementation in this branch.

## Dependency direction

```text
one parse + shared declarations / scope / flow construction
|
neutral Program + optional lossless syntax view
|
Pyrefly-specific binding emitter
|
actual Bindings -> existing solver
```

The emitter belongs in Pyrefly or an optional adapter, not the generic core. Its job is to retain Pyrefly's binding tables, inference policy and result packaging while replacing frontend-facing work. Today's [Bindings::new](https://github.com/facebook/pyrefly/blob/0424eebb19345ca0b40727a15cfca9e759c05d6b/pyrefly/lib/binding/bindings.rs#L702-L897) combines that work with AST traversal, scope state, deferred name processing and exports.

Binding emission should **not** happen as side effects of `Domain::transfer`: the forward engine repeats transfers during fixpoint iteration and diagnostic replay. An emitter should reserve graph identities, obtain neutral flow relationships, and emit each computation once. It need not use the generic worklist engine as its dependency solver.

## Identity mapping

`SymbolId` denotes one name owned by one scope. It is not one definition or one typing computation. The emitter needs separate mappings:

- declaration/write occurrence -> `Key::Definition`;
- read occurrence -> `Key::BoundName` plus its selected value/narrowing dependency;
- scope/symbol -> annotation, capture and anywhere-summary bookkeeping;
- producing operation and merge port -> the computation supplying a value.

The spike's `Reg` is not SSA: both arms of a conditional expression can write the same temporary. A simple `(RegionId, Reg) -> Key` map would therefore lose producers. Use operation identities and predecessor ports.

Nor should every runtime operation become a separate `Binding::Expr`. Existing [NameAssign](https://github.com/facebook/pyrefly/blob/0424eebb19345ca0b40727a15cfca9e759c05d6b/pyrefly/lib/binding/binding.rs#L2327-L2357) retains a whole RHS expression so the existing solver can apply contextual typing. Runtime decomposition and typing-computation ownership must remain distinct.

## Typing dependencies stay downstream

Execution order is not inference order. An annotation or receiver can supply an expected type to the RHS; callable parameters constrain arguments; predicates generate narrowing dependencies; first-use inference can connect a definition to a later use.

The shared frontend should preserve source occurrences, expression relationships and declaration records. Pyrefly decides whether a particular occurrence is a type, metadata, ordinary value, or non-pinning read. This includes `cast`, type-like subscripts, and `Annotated` metadata. Retaining syntax does not require a second scope/control-flow walker: shared traversal supplies occurrence records and expression roots, while the existing solver retains its expression inference.

First-use resolution belongs in graph finalization. Pyrefly records uses with their computation owners, fills phis and then finalizes reads in insertion order. Runtime earliest-use order is insufficient; static/non-pinning reads and solve-time feedback differ. See [deferred name-use finalization](https://github.com/facebook/pyrefly/blob/0424eebb19345ca0b40727a15cfca9e759c05d6b/pyrefly/lib/binding/bindings.rs#L2027-L2117).

## Flow, termination and deferred references

CFG predecessors are inputs to Pyrefly phis, not finished phis. A branch's value can be paired with a typing computation that establishes whether that branch terminates. A dependency solving to `Never` may remove a branch from a typing merge; missing definitions generate deferred initialization checks.

Expose incoming-edge identities and definitions before generic joins discard them. The adapter adds branch tests/end computations, narrowing, exhaustiveness and loop-merge policies. Explicit runtime return/raise structure helps, but does not resolve type-dependent continuation.

Reserve identities before filling computations to support recursion and forward references. Runtime closure-cell reads are invocation-time facts; Pyrefly's static/anywhere lookup and forward typing references are separate adapter policies. Domain-free `History::reaching` marks possible writes from other regions with `Def::External`; such a set is not an exact local-only value history. It does not resolve invocation-time cross-region reaching definitions.

## Concrete example

```python
from typing import Callable


def pick(flag: bool, f: Callable[[int], int]) -> int:
if flag:
x: int = f(1)
else:
x = 0
return x
```

An experimental emitter would:

1. Reserve function, parameter, annotation and return identities.
2. Bind the condition's `flag` occurrence and add downstream narrowing/branch computations.
3. Emit two definition/`NameAssign` pairs for `x`. Keep `f(1)` as the true RHS computation; attach its `f` read to that assignment's usage context. Forward the canonical annotation to the other assignment under Pyrefly's policy.
4. Emit the merge with predecessor/continuation dependencies, then connect the return's `x` read to it.
5. Emit return-type and explicit-return computations and finalize deferred uses.

This maps to existing statement/function constructor seams, but **requires additional syntax preservation**: current runtime lowering does not retain the `x: int` declaration sufficiently to reproduce its complete binding output.

## Capability gap map

The gap map is not limited to constructs already marked with a runtime `Gap`:

- **Annotation syntax and occurrences:** annotation-only declarations can emit no runtime operation. `StaticExpr::Other` retains only a range; structured static forms lose some occurrence ranges and collection distinctions. Preserve declaration records and lossless annotation/type-parameter syntax.
- **Expression payloads and computation grouping:** bytes, complex/oversized numerical values and formatted-string structure are compressed. Target groups, keyword identifier ranges and enclosing RHS ownership are incomplete. Source slices are not a replacement for a parsed syntax view.
- **Scope/capture provenance:** declaration occurrences (`global`/`nonlocal`), static capture anchors and diagnostic ranges need fuller records. Operations now carry their lexical evaluation scope, and class/module loads have shared binding-presence/fallback semantics; those runtime facts still differ from Pyrefly's typing lookup policy.
- **Flow ports:** structural CFG edges exist, but joined frames/history sets discard incoming-edge provenance. Raised exits now retain source range, cause and state. Return terminators still need explicit/implicit origin and range metadata for exact binding parity.
- **Function/class metadata:** headers, suites, decorator roles, overload adjacency, annotation-defined fields and synthesized `__class__` plumbing are incomplete. Shared generic declaration structure belongs upstream; overload, `Self`, dataclass/framework synthesis and shape-DSL policy stay downstream.
- **Typing declarations:** PEP 695 scopes, alias RHS syntax, bounds/constraints/defaults and recursive alias anchors are outside v0. Ordinary function regions are not enough to reproduce them.
- **Patterns, exceptions, imports and packaging:** approximation gaps are not lossless representations. Exact binding/IDE parity also includes suppression ranges, scope traces, unused-symbol data and other result metadata.

## Smallest next experiment

Three frontend additions would make the seam testable:

1. An **optional lossless syntax view**, with occurrence IDs, original expression roots, exact ranges, declaration/header/suite records and separately represented typing syntax.
2. A **neutral predecessor-port query**, including incoming definitions of symbols and temporary values plus entry/unbound paths.
3. A **parse-reuse entry point** accepting the already parsed module. Optional collectors should not allocate when absent.

Then implement an opt-in emitter for names, assignments, one function, calls and branches. Compare canonicalized actual binding tables with `Bindings::new`: source identities, kinds, dependency edges, expectations and metadata, not merely final inferred types. Add first-use, annotation-only, loop/unbound, recursion/capture, termination, decoration, class and type-parameter fixtures incrementally. Unsupported constructs must fail coverage checks rather than fabricate parity.

## Performance decision

Benchmark optimized, comparable configurations: parse only; parse plus neutral frontend; frontend plus history; frontend plus syntax view/emitter; existing bindings. Separate parsing-included and parsing-excluded results, allocations and peak memory, with comparable AST lifetimes.

Preserve existing skipped-body/lazy-work optimizations where possible. Measure the no-domain/no-history/no-syntax-overlay path explicitly. Do not adopt a percentage threshold without a baseline and measured noise; capability success and performance acceptability are independent decisions.

A small parity-tested adapter would increase confidence in reusability. Until then, the two implemented domains establish a narrower result: shared analysis machinery for their supported subsets, not replacement capability for the full Pyrefly binding pass.
Loading
Loading