Repository navigation
Declarative filter operation for safe-mode collection filtering #293
Description
Activity
Summary
This is a well-scoped design proposal, not an epic — no linked papers/external resources to check. I dug into the referenced issues and the current code to ground the open questions below. Two pieces of prior art on this exact terrain aren't cross-linked yet and are worth pulling in.
Prior art: #187 and #188 cover adjacent ground, unimplemented
Both filed by you, both still open, both predate this issue by ~3 months:
- Add filter_rows expression to ClassDerivation for row-level filtering #187 "Add
filter_rowsexpression toClassDerivation" — a class-level row filter (skip whole source rows before processing), evaluated with the existing safe-expr infra. Explicitly notes composability with a future DuckDB backend (Flesh out DuckDBTransformer as a full execution backend #151) translating it to SQL WHERE. - Implement multi-row aggregation, group-by, and collection membership (in) operator #188 "Implement multi-row aggregation, group-by, and
inoperator" — notesAggregationOperationis defined in the model but has zero runtime implementation inObjectTransformer. I confirmed this is still true:aggregation_operationdoesn't appear anywhere inobject_transformer.pytoday. So theaggregation_operation: {operator: FIRST}sketch in this issue would need that wiring built regardless of whether FIRST/LAST land asAggregationTypevalues or a separateSelectionOperation— extending the enum alone changes nothing at runtime.
The distinction worth naming explicitly: #187's
filter_rowsfilters which source rows become target instances (class-level, whole-row). This issue'sFilterOperationfilters elements inside an already-selected multivalued slot (slot-level, sub-row). Related concerns, different granularity — worth linking all three issues together so whoever picks this up doesn't design the operator enum twice.The DuckDB/WHERE framing may be over-scoping the immediate unblock
I checked
has_important_life_eventsinpersoninfo-to-agent.transform.yamland the source schema: it's a locally inlined multivalued slot onPerson, populated straight from the source object — nojoins:involved anywhere in this transform spec. So the concrete thing blocking #270's flagship example is a plain in-memory list-comprehension-shaped filter, not a cross-table query.I also checked the DuckDB/join engine code that exists in the tree (
join_engine.py,duckdb_transformer.py,sql_compiler.py, ~500 lines total) — there's currently no WHERE/predicate/filter concept anywhere in it. So "align the operator enum with what the DuckDB path can express" is aspirational alignment with a facet that doesn't exist yet, not alignment with landed work. That's still the right instinct for avoiding two dialects later, but it means the object-pathFilterOperationdoesn't actually have an implementation to wait on — the dependency this issue's last open question raises ("does this wait on the joins/query work settling?") looks like it can be answered no for the object path specifically, provided the operator semantics (null handling, enum coercion) are specified narrowly enough now to stay a compatible subset whenever the SQL side catches up.On the open questions
- Operator scope: given Hidden slot derivations silently skip range mapping and collection reshaping #292 (hidden-slot reshaping) and Add safe built-in functions for expression evaluation #206 (safe builtins) both had to solve "what does EQ mean for an enum-ranged slot" already, that precedent (str-coerce or use the enum's
permissible_valuetext) is worth reusing rather than re-deriving. - FIRST/LAST: lean toward a distinct concept from
AggregationType, per Implement multi-row aggregation, group-by, and collection membership (in) operator #188's own framing ("FIRST isn't really an aggregation") — and either way it needsobject_transformer.pyruntime support that doesn't exist yet, so scope that work explicitly rather than assuming the enum addition is sufficient. selectprojection: reusingSlotReference-shaped projection (asPivotOperation.variable_slot/value_slotalready do elsewhere intransformer_model.yaml) would keep this consistent with the existing operation vocabulary instead of introducing a new shape.
Recommend cross-linking #187 and #188 from here (and vice versa) so the operator-enum design happens once.
- Add filter_rows expression to ClassDerivation for row-level filtering #187 "Add
The flagship
personinfo_basicexample can't run under default flags. Three of its Agent derivations need to filter a multivalued slot and pluck a field off the match:That's a list comprehension plus multi-statement assignment, so it requires
--unrestricted-eval. #270 documented the requirement and pinned it with tests; this issue is about closing the capability gap so the shape doesn't need unsafe eval at all.What's actually missing
Filtering is only one of three capabilities the derivation needs, and none of them have a declarative shape today:
expr: "[x.id for x in {friends}]")AggregationTypehas SUM/AVERAGE/COUNT/MIN/MAX/STD_DEV/VARIANCE/MEDIAN, but no FIRST/LASTAdding any one alone still leaves the example needing the flag.
Preferred shape: declarative, not new safe functions
transformer_model.yamlalready has the extension point — an abstractTransformationOperationwithAggregationOperation,GroupingOperation, andPivotOperationas siblings. AFilterOperationis the missing fourth. Sketch:The function route looks cheaper and isn't. Without lambdas you can't pass a predicate, so you land on
filter_eq(list, "event_name", "DEATH")— stringly-typed, equality-only — and then the pressure is unbounded:ne,gt/lt,in,contains,startswith,matches,is_null, thenpluck(you can't finish without it), thensort_by,distinct,flatten,group_by. That's hand-rolling JMESPath one allowlist entry at a time with no spec for the semantics. A declarative op makes the operator set an explicit, reviewable enum instead of an open backlog, and is safe by construction rather than safe by allowlist.The WHERE / DuckDB facet
This is the part worth settling before designing anything: a filter predicate is a WHERE clause. The set-based DuckDB join engine landed in 510b0a5, and the joins rework is ongoing. If a query engine is emerging under that work, filter semantics should be defined in terms both engines can execute rather than invented separately in
ObjectTransformer— otherwise we get two dialects with subtly different null and type-coercion behavior, which is exactly the class of bug that's expensive to find later.Concretely, a
FilterOperationshould lower to a DuckDBWHEREon the set-based path and to a per-object predicate on the object path, with one definition of what EQ means for nulls, enums (noteevent_name's range is an enum, and the current expr works around this withstr(x.event_name)), and numeric/string coercion. Aligning the operator enum with what the DuckDB path can express for free seems like the right forcing function on scope.Open questions
selectthe right projection shape, or should projection be a separate operation?AggregationTypevalues, or a distinct selection operation? FIRST isn't really an aggregation.Related: #270 (documented the requirement), #292 (hidden slots skip reshaping — blocks expressing this via the existing multi-step
hide/slot()mechanism), #206 (added safe built-ins to reduce reliance on unsafe eval), #241 (transform-time validation).