Validate Python MakeSsa D facade candidate - #158
Draft
yoff wants to merge 32 commits into
Draft
Conversation
Preparatory refactor for the shared-CFG dataflow migration. Adds the new Python SSA adapter additively, without changing any production behaviour. Library additions: - semmle.python.dataflow.new.internal.SsaImpl — Python SSA implementation built on the new (shared) CFG. Mirrors the Java SSA adapter (java/ql/lib/semmle/code/java/dataflow/internal/SsaImpl.qll): an InputSig is defined in terms of positional (BasicBlock, int) variable references, and the shared codeql.ssa.Ssa::Make<Location, Cfg, Input> module is then instantiated. SourceVariable is the AST-level Py::Variable. Variable references are looked up via the new CFG facade's NameNode.defines/uses/deletes predicates (added in the preceding PR), which themselves are one-line bridges to AST-level Name.defines/uses/deletes. Implicit-entry definitions are inserted for non-local/global/builtin reads, captured variables, and (when needed) parameters. Test additions: - library-tests/dataflow-new-ssa/ — exercises the new SSA over a representative test corpus and checks expected def/use chains. - library-tests/dataflow-new-ssa-vs-legacy/ — runs both new SSA and legacy ESSA over the same corpus and diffs the results, so any semantic divergence shows up as a test failure. Production impact: None. The new SSA adapter has zero callers in lib/ and src/ — the legacy ESSA SSA (semmle/python/essa/*) remains the default. The dataflow library is not migrated yet; that lands in a follow-up PR. Verified by: - All 367 lib + src + consistency-queries compile clean. - All 641 ControlFlow + PointsTo + dataflow + essa + consistency library-tests pass. - Both new dataflow-new-ssa[/vs-legacy] test packs pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- part of the ESSA adapter layer still refers to the raw SSA (now called Impl)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll) and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade (semmle.python.controlflow.internal.Cfg) and the new SSA adapter (semmle.python.dataflow.new.internal.SsaImpl), both introduced additively in the preceding PRs in this stack. This is the trunk-flip equivalent of the original draft PR github#21894 (kept around as documentation), rebased on top of the four preparatory PRs: P1: Remove AstNode.getAFlowNode() and rewrite callers (github#21919). P2: Qualify Flow.qll's AST references with Py:: prefix (github#21920). P3: Add new shared-CFG-backed control flow graph (github#21921). P4: Add new shared-SSA-backed SSA adapter (github#21923). The Python dataflow library (semmle/python/dataflow/new/) now imports the new CFG facade and SSA adapter. All CFG-typed predicates (ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are qualified with the Cfg:: prefix; SSA references switch from EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable. GuardNode is redesigned to use the new CFG's outcome-node model (isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock + flipped indirection. Only BarrierGuard<...> is preserved as public API. Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib, ...) are updated to take CFG nodes from the new facade. A handful of dataflow consistency tweaks for the new CFG: - Augmented-assignment targets are treated as both load and store. - 'from X import *' produces uncertain SSA writes for unknown names. - CFG nodes are canonicalised so dataflow does not see equivalent pre/post-order pairs as distinct nodes. Two AST tweaks for the new CFG: - AstNodeImpl: omit PEP 695 type-parameter names from FunctionDefExpr / ClassDefExpr children. - ImportResolution: drop the legacy essa import. Test churn (~175 files): reblessed library- and query-test .expected files reflect slightly different CFG granularity, different toString output, and a handful of true alert deltas in security queries. Verification: all 367 lib + src + consistency-queries compile clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's `dominates`/`strictlyDominates` predicates, which are declared `bindingset[this, that]` + `pragma[inline_late]` and are meant to be used as bound-pair membership checks. The facade wrappers dropped these annotations (using plain `pragma[inline]`), so even though the only callers — the `with` / `async with` taint steps in DataFlowPrivate.qll and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full O(nodes^2) relation over the (larger) shared-CFG node set. On some projects this dominated analysis time entirely (DCA showed e.g. ICTU/quality-time and biosimulations regressing ~75-160x). Restoring `bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns the predicate back into a bound-pair check and is result-preserving (only binding annotations change, the predicate body is unchanged). Reproduced on ICTU/quality-time: full python-security-extended suite went from stalling >20min on `strictlyDominates` to completing in ~6min; all ControlFlow and dataflow/coverage library tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the public expression adapter and apply the canonical QL annotation ordering required by the formatter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 529363f5-bc7d-4f0b-9f47-e03ba9aa0cdf
Prove that the query-shaped shared-SSA relation for direct truthiness guards is equivalent to the generic BarrierGuard abstraction on the modification-of-default-value regression corpus. This guards the performance specialization against semantic drift before changing production code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
Keep the modification-of-default-value query on shared SSA while expressing its two direct truthiness checks in a query-shaped predicate. The generic BarrierGuard abstraction causes the evaluator to materialize and rescan a 1,579,772,664-row def-use pair relation before applying branch control. Binding both concrete NameNode uses in one predicate lets the optimizer fuse the same joins with controlsBlock and persist only the 4,992 guarded uses. On FreeCAD@0def330, three prewarmed evaluator runs improve from 309.159-329.621s to 77.781-81.349s with byte-identical query results. A direct symmetric-difference evaluation returns zero rows, and the CommandInjection path-query control retains identical results, work, and plan hashes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
The legacy CFG (`Flow.qll`) and legacy ESSA (`Essa`/`SsaCompute`/ `SsaDefinitions`) were pinned into the always-on `Stages::AST` cached stage via `Stages::AST::ref()` and the matching `backref()` disjuncts. Because a cached stage is materialized as a unit once any of its predicates is demanded (and every query demands e.g. `Expr.toString()`), this forced the legacy CFG/ESSA to be computed for *every* query -- including the security/dataflow queries, which after the shared-CFG dataflow flip no longer depend on the legacy CFG at all. Since `Stages::AST::ref()` is `1 = 1`, removing it is result-preserving; it only changes stage scheduling. After this change the legacy CFG/ESSA is no longer materialised for queries that do not genuinely reference it. Verified on the full `python-security-extended` suite and on django: legacy CFG/ESSA families materialised drop from ~165 to 0 with byte-identical results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The previous capturedJumpStep shape introduced an independent Cfg::DefinitionNode and related it to the captured variable before nodeTo bound the relevant scope-entry definition. On substantial databases, the evaluator chose a plan that materialized a high-duplication store/variable join before applying the target entry and source-node constraints. Bind the scope-entry definition from nodeTo first, derive its source variable, and then match nodeFrom's DefinitionNode directly to that variable's store. This is relation-equivalent existential elimination: the old nodeFrom.asCfgNode() = def and def.getNode() = store constraints become nodeFrom.asCfgNode().(Cfg::DefinitionNode).getNode() = store, preserving the DefinitionNode type restriction and the unchanged enclosing-scope condition. On Airflow a9da0f7 with CodeQL 2.26.2, against exact github#21925 head 1a8e317: * The call-target diagnostic retains the identical 59,732-edge set while tuples joined fall from 1,032,403,207 to 66,969,953 (-93.5%), maximum duplication falls from 1,473,981 to 4,053, and evaluator wall time falls from 59.9s to 6.2s. * py/clear-text-logging-sensitive-data retains every result tuple (130 alerts, 256,708 path edges, 102,385 path nodes, and 116,976 subpaths) while tuples joined fall from 1,329,594,809 to 242,111,554 (-81.8%) and evaluator wall time falls from 94s to 14.4s. The preceding commit is intentionally a semantic call-target invariant diagnostic that passes on the unoptimized relation and after this rewrite. An inline MISSING/SPURIOUS red-state would assert an artificial semantic delta; this optimization must preserve every valid captured call target. These measurements cover one exact substantial Airflow database and two query shapes. No DCA was run, the fix does not reduce legitimate call-graph or path growth, and broader fleet performance remains to be confirmed separately. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5dfda5f5-08c8-481b-9ecb-299018701497
Exercise the three public AdjacentUses relations and compare them with their internal projection or recursive expansion contracts. The cache-placement defect changes evaluator specialization and work rather than semantic results, so a semantic contract snapshot is the stable red-state equivalent; wall-time assertions would be machine-dependent and flaky. This intentionally records no MISSING or SPURIOUS rows: the expected invariant is exact relation equality before and after cache placement changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 21ab8585-861f-42c9-a834-451604646c6b
The shared SSA module requires language adapters to cache predicates that they expose. The Python adapter exposed firstUse, adjacentUseUse, and useOfDef without restoring that cache boundary, unlike the legacy AdjacentUses implementation. On exact historical Salt, the missing boundary caused the same 6,313,793-row liveAtExit fixed point to be evaluated twice. The equivalent plans received distinct RA hashes (c6bc8xgji0uv6seurbhesjqd315 versus fabf1xs3jb6t67a2buq2iv8iof4 for unsafe deserialization, and c6bc8xgji0uv6seurbhesjqd315 versus 8270excv27ldlfrtk19ou81d206 for modification-of-default-value) because one inherited an unrelated cached-empty sentinel while the other used a literal empty base. Cache the three Python adapter relations rather than generic liveness. This restores the documented shared-SSA contract at the narrow language boundary and avoids imposing a 6.31M-row generic cache on every language instantiation. On current head 1a8e317 with exact saltstack/salt@d036b117, three matched prewarmed -j1 repeats reduced median evaluator time from 51.294s to 45.103s for unsafe deserialization and from 42.377s to 34.238s for modification-of-default-value. Median paired reductions were 6.428s and 8.247s. Joined tuples fell by 58,255,670 and 85,664,313; recursive pipeline runs fell by 1,999 and 3,015. Both queries retained the identical empty endpoint hash 2a514e093aae140a14f6bf77beebe1ad in every repeat. Historical exact controls also retained 483,922 definitions, 169,921 phi inputs, 390,548 first uses, 475,226 adjacent uses, and 123,231 semantic call edges with zero left-only or right-only rows. Historical Salt evaluator recovery was 16.5% and 19.9%. Cold prewarm evaluator time was neutral (106.609s to 106.620s), so this is a warm-query optimization rather than a claimed cold-cache speedup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 21ab8585-861f-42c9-a834-451604646c6b
Allow language adapters that expose one SSA instantiation through multiple cached API stages to persist the complete liveness fixed point once. Keep the existing demand-specialized factory as the default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
Use the opt-in shared SSA factory so Python reuses the same complete liveness relation across its staged public SSA and data-flow consumers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
Allow language-specific SSA instantiations that cross multiple cached API stages to cache the complete definition-rank and end-of-block reachability fixed points. Keep the default factory demand-specialized so unrelated instantiations retain their existing evaluator behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
Select the opt-in cached definition-reachability factory for Python main SSA. The independent capture-SSA instantiation continues to use the default uncached factory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
Cache the Python-local shared-CFG scope relation after semantic inputs and before staged SSA and dataflow consumers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
The canonical injects-to-Python-AST mapping is evaluated repeatedly across DCA prewarm and target stages. Cache getNode so this truthful mapping forms a reusable evaluation boundary without changing its semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 857236eb-3350-48ba-9bf1-6bf5a387191b
Late-inline the explicit-step after-value wrapper so AST and successor demand are bound before expanding control-flow nodes, without changing CFG semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Regenerate the typetracking import diagnostic expectation after rebasing the production flip onto the current-main shared-SSA baseline. The two exit-use definitions are now labeled as implicit; query behavior is otherwise unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 03758eab-d713-4ffb-87bd-64ceb863dd3b
Apply the sealed MakeSsa D production candidate exactly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Format the sealed facade adapter and cover ordinary and synthetic-exit uses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
yoff
force-pushed
the
yoff-freecad-guarded-use-hotspot
branch
from
September 12, 2026 22:12
80939b0 to
8780dc1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Standalone diagnostic layer on draft #156 (
yoff-freecad-guarded-use-hotspotat80939b08790e0de870e8a5014b95f4fdfe5ae961). This PR is intentionally not inserted into or appended to native stack github#155; #157 remains the existing top of that stack and is excluded from both DCA variants.The candidate is the sealed ordered A→B core-cache removal plus B→D Python facade/adapter correction. No broad raw-reachability cache or new shared API is introduced.
Sealed identity
17e9303e5b8d37912082e3388ed0907e6cfdfbdb1ff1857efe80cedc6d391895627b47f2feb90c9adc984f88935d6617e4c50b4167f79f7e3eacf06a5e9913c66344a7cdf1d869c68be73c6371df1916d428545186f8b0d03119c5e2d06d57a54681f6af496050b4ccd991bc6547f05ccf6322f30601ee4a568fb97fLocal evidence
Deterministic joined tuples versus A:
D beats prior C on every source. CPython and Nova repeats retained invariant joined tuples and BQRS. Exact BQRS, focused tests, real-database facade/use contracts, capture SSA, and ordinary/synthetic-exit non-overlap passed.
The private synthetic-exit cache materializes once with 201,609 / 136,740 / 245,694 / 218,282 rows for Airflow / Nova / Salt / CPython and receives a target-query cache hit. It does not contain ordinary reads. Retained cache remains 7,168–25,304 KiB below A with unchanged tuple-pool sizes.
Validation
dataflow-new-ssa/SsaTest.qldataflow-new-ssa/AdjacentUsesContract.qldataflow-new-ssa/FacadeAdapterContract.qldataflow-new-ssa-vs-legacy/CmpTest.qldataflow/variable-capture/CaptureTest.qldataflow/variable-capture/dataflow-capture-consistency.qlAll passed. The focused facade contract reports zero ordinary-read, end-of-block, phi-input, uncertain-write, adapter-use, facade-exit-read, and ordinary/synthetic-exit overlap mismatches; it observes 14 synthetic exit uses. Airflow CSRF exact-output control reproduced BQRS hash
05953605a10e3fd0994ef0f614ff5abd.