Skip to content

feat: add null_absent_columns() for cross-version schema tolerance - #398

Merged
abrignoni merged 1 commit into
mainfrom
feat/null-absent-columns
Aug 8, 2026
Merged

feat: add null_absent_columns() for cross-version schema tolerance#398
abrignoni merged 1 commit into
mainfrom
feat/null-absent-columns

Conversation

@abrignoni

Copy link
Copy Markdown
Owner

Apps add columns between releases. A query written against a newer store names a column an older one lacks, SQLite fails the whole statement with no such column, and the artifact reports nothing instead of the rows it could have returned. One missing column costs every row, and it fails quietly: an artifact that returns nothing looks exactly like a feature the user never used.

null_absent_columns(path, query) substitutes NULL for column references the database does not have.

Ported verbatim from iLEAPP, where it took ten modules from 132 error lines to 1 across 21 corpora and recovered rows on 56 artifact/corpus pairs (iLEAPP #1912).

Three properties that are not obvious, and each cost a debugging round

  1. It emits NULL AS <name>, not a bare NULL, where the reference is a select item in its own right. A bare NULL also drops the output column's name, and artifacts read rows by name, so the SQL starts succeeding and the artifact then dies on record['ZFOO'] with IndexError: No item with that key. Inside an expression the enclosing alias already names the column, so a plain NULL is correct there.
  2. The substitution is a single pass over one compiled alternation. A per-reference loop rewrites the alias it just inserted and produces NULL AS NULL. A sentinel does not save it either, because a non-word sentinel still satisfies the (?<![\w.]) lookbehind.
  3. Column comparison is case-insensitive. A case-sensitive check reported 9 missing columns on one Photos.sqlite where only 2 were real.

It resolves table aliases from the FROM/JOIN clauses, only touches references it can attribute to exactly one table, and returns the query unchanged if the schema cannot be read.

Scope

No call sites in this repo yet. This makes the helper available; it does not change any artifact's behaviour. Wiring it into specific artifacts is a separate change that deserves its own before/after corpus sweep.

Verified the ported function behaves identically to iLEAPP's in this core: same 97 rows on a real ThreeBars.sqlite missing ZLOWQUALITY, column name preserved, value None.

Worth noting for context: these five ilapfuncs.py files are less synchronised than "shared core" suggests — 43 top-level functions are common to all five, with 7 to 28 unique to each. So this is a deliberate addition rather than a sync.

check_claim_language.py, check_html_safety.py and lint_changed.py pass. Lint reports one fewer warning than before, because the helper uses re and silenced a pre-existing unused-import warning.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Apps add columns between releases. A query written against a newer store names a
column an older one lacks, SQLite fails the whole statement with "no such
column", and the artifact reports nothing rather than the rows it could have
returned. One missing column costs every row.

null_absent_columns(path, query) substitutes NULL for references the database
does not have. It keeps the column in place and keeps its name, because
artifacts read rows by name and a bare NULL renames the output column.

Ported verbatim from iLEAPP, where it took ten modules from 132 error lines to 1
across 21 corpora and recovered rows on 56 artifact/corpus pairs (PR #1912).

No call sites in this repo yet: this makes the helper available, it does not
change any artifact's behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@abrignoni
abrignoni merged commit 3ca4b33 into main Aug 8, 2026
4 checks passed
@abrignoni
abrignoni deleted the feat/null-absent-columns branch August 8, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant