Store terms in vectors to reduce compilation latency

NEWS.md

@@ -1,3 +1,61 @@
+# v0.8.0
+
+- Term collections are now `Vector{AbstractTerm}` instead of tuples (#354).
+  This is a breaking change for package developers; user-facing `@formula`
+  syntax and `modelcols`/`modelmatrix` are unchanged.  The motivation is
+  compile latency: with tuples, every distinct number and order of terms in a
+  formula triggered a fresh specialization of the whole `apply_schema`,
+  `modelcols`, `coefnames`, etc. pipeline, so fitting a model with a
+  never-seen formula cost ~0.1-0.8s of compilation.  With vectors, this drops
+  to milliseconds.
+
+  - `+` on terms returns a `Vector{AbstractTerm}`, and `&` distributes over
+    vectors of terms instead of tuples.
+
+  - `InteractionTerm` and `MatrixTerm` are no longer parametric: both store
+    their terms in a `terms::Vector{AbstractTerm}` field.  The constructors
+    still accept any iterable of terms (including tuples), so
+    `InteractionTerm((a, b))` keeps working.  Code that dispatched on the
+    element types, e.g. `InteractionTerm{<:NTuple{N,CategoricalTerm}}`, must
+    switch to a run-time check such as
+    `all(t -> t isa CategoricalTerm, it.terms)`.
+
+  - `TupleTerm` is removed.  Methods that took a `TupleTerm` should take an
+    `AbstractVector{<:AbstractTerm}` instead.  `TermOrTerms` is now
+    `Union{AbstractTerm, AbstractVector{<:AbstractTerm}}`.
+
+  - `collect_matrix_terms` returns a `Vector{AbstractTerm}` (with the
+    `MatrixTerm` first) in the mixed matrix/non-matrix case, and `modelcols`
+    on a vector of terms returns a vector of the per-term columns.
+
+  - `apply_schema` on a vector of terms applies the schema term by term, from
+    left to right, and combines the results with `+`, so duplicates are
+    dropped and terms that expand to several terms are flattened.
+
+  - An empty vector of terms is valid everywhere a vector of terms is: it
+    yields an empty `MatrixTerm` of width 0 whose `modelcols` is a matrix with
+    no columns, like `InterceptTerm{false}`.  Constructing an
+    `InteractionTerm` with no terms throws an `ArgumentError`.
+
+  - `FormulaTerm`, `InteractionTerm`, and `MatrixTerm` now have content-based
+    `==` and `hash`, since the default field-identity comparison no longer
+    holds with vector fields, and `FunctionTerm` gains a `hash` consistent
+    with its existing `==`.  As a consequence, formulas compare equal after
+    `apply_schema` when their terms do, and hash-based deduplication
+    (`unique`, `Set`) works for formulas containing function calls.
+
+- The right-hand side of a formula built with `~` or `@formula` is always a
+  `Vector{AbstractTerm}`, even when it contains a single term (#354).
+  Previously `@formula(y ~ x).rhs` was the bare term `x`; it is now `[x]`.
+  This removes the lone-term special case, so code that handled both a term
+  and a tuple on the right-hand side can handle a vector only.  The left-hand
+  side is still a bare term, and after `apply_schema` the right-hand side is
+  still collected into a single `MatrixTerm` when every term is a matrix
+  term, so the `(y, X)` returned by `modelcols(f, data)` is unchanged.  The
+  same applies to `+` (`a + a` is now `[a]` rather than `a`) and to
+  `apply_schema` on a vector of terms, which returns a vector even when only
+  one term remains.
+
 # v0.7.0
 
 - `FunctionTerm` rework (#183)

Project.toml

@@ -1,6 +1,6 @@
 name = "StatsModels"
 uuid = "3eaba693-59b7-5ba5-a881-562e759f1c8d"
-version = "0.7.10"
+version = "0.8.0"
 
 [deps]
 DataAPI = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"

docs/src/formula.md

@@ -299,7 +299,7 @@ symbols or strings (`Term`) and numbers (`ConstantTerm`), which makes it easy to
 work with collections of mixed type:
 
 ```jldoctest 1
-julia> ts = term.((1, :a, "b"))
+julia> ts = term.([1, :a, "b"])
 1
 a(unknown)
 b(unknown)

docs/src/internals.md

@@ -57,26 +57,27 @@ expression returned by the `@formula` macro is evaluated.  At this point, the
 julia> using StatsModels;
 
 julia> dump(Term(:a) & Term(:b))
-InteractionTerm{Tuple{Term, Term}}
-  terms: Tuple{Term, Term}
+InteractionTerm
+  terms: Array{AbstractTerm}((2,))
     1: Term
       sym: Symbol a
     2: Term
       sym: Symbol b
 
 julia> dump(Term(:a) + Term(:b))
-Tuple{Term, Term}
+Array{AbstractTerm}((2,))
   1: Term
     sym: Symbol a
   2: Term
     sym: Symbol b
 
 julia> dump(Term(:y) ~ Term(:a))
-FormulaTerm{Term, Term}
+FormulaTerm{Term, Vector{AbstractTerm}}
   lhs: Term
     sym: Symbol y
-  rhs: Term
-    sym: Symbol a
+  rhs: Array{AbstractTerm}((1,))
+    1: Term
+      sym: Symbol a
 ```
 
 !!! note
@@ -254,7 +255,7 @@ terms:
 
 * `Term`s become `ContinuousTerm`s or `CategoricalTerm`s
 * `ConstantTerm`s become `InterceptTerm`s
-* Tuples of terms become [`MatrixTerm`](@ref)s where appropriate to explicitly indicate
+* Vectors of terms become [`MatrixTerm`](@ref)s where appropriate to explicitly indicate
   they should be concatenated into a single model matrix
 * Any model-specific (context-specific) interpretation of the terms is made, including
   transforming calls to functions that have special meaning in particular
@@ -274,7 +275,7 @@ Predictors:
   b(unknown) & c(unknown)
 
 julia> typeof(f)
-FormulaTerm{Term, Tuple{ConstantTerm{Int64}, Term, Term, Term, InteractionTerm{Tuple{Term, Term}}}}
+FormulaTerm{Term, Vector{AbstractTerm}}
 
 julia> f = apply_schema(f, schema(f, df))
 FormulaTerm
@@ -288,7 +289,7 @@ Predictors:
   b(continuous) & c(DummyCoding:3→2)
 
 julia> typeof(f)
-FormulaTerm{ContinuousTerm{Float64}, MatrixTerm{Tuple{InterceptTerm{true}, ContinuousTerm{Float64}, ContinuousTerm{Float64}, CategoricalTerm{DummyCoding, Matrix{Float64}, 2}, InteractionTerm{Tuple{ContinuousTerm{Float64}, CategoricalTerm{DummyCoding, Matrix{Float64}, 2}}}}}}
+FormulaTerm{ContinuousTerm{Float64}, MatrixTerm}
 ```
 
 This transformation is done by calling `apply_schema(term, schema, modeltype)`
@@ -575,9 +576,8 @@ julia> poly(my_col, my_degree)
 poly(a, 3)
 
 julia> poly.([:a, :b], my_degree)
-2-element Vector{PolyTerm{Term, ConstantTerm{Int64}}}:
- poly(a, 3)
- poly(b, 3)
+poly(a, 3)
+poly(b, 3)
 ```
 
 These run-time `PolyTerm`s are "schema-less" though, and to be able to construct
@@ -610,7 +610,7 @@ regression as above (which used `@formula(y ~ 1 + poly(a, 2) + poly(b, 2)`), but
 with the predictor names and the polynomial degree stored in variables:
 
 ```jldoctest 1
-julia> poly_vars = (:a, :b); poly_deg = 2;
+julia> poly_vars = [:a, :b]; poly_deg = 2;
 
 julia> poly_formula = term(:y) ~ term(1) + poly.(poly_vars, poly_deg)
 FormulaTerm

docs/src/temporal_terms.md

@@ -50,7 +50,7 @@ Predictors:
   lead(x, 2)
 
 julia> modelmatrix(f, df)
-5×3 reshape(::Matrix{Union{Missing, Int64}}, 5, 3) with eltype Union{Missing, Int64}:
+5×3 Matrix{Union{Missing, Int64}}:
   2   missing   6
   4   missing   8
   6  2         10
@@ -88,7 +88,7 @@ Predictors:
   lead(x, 2)
 
 julia> modelmatrix(f2, df)
-5×3 reshape(::Matrix{Union{Missing, Int64}}, 5, 3) with eltype Union{Missing, Int64}:
+5×3 Matrix{Union{Missing, Int64}}:
   2   missing   6
   4   missing   8
   6  2         10

src/modelframe.jl

@@ -113,7 +113,7 @@ keyword arguments are passed to [`apply_schema`](@ref).
     [`modelcols`](@ref) pipeline directly
 
 """
-function StatsAPI.modelmatrix(t::Union{AbstractTerm, TupleTerm}, data;
+function StatsAPI.modelmatrix(t::TermOrTerms, data;
                                hints=Dict{Symbol,Any}(), mod::Type{M}=StatisticalModel) where M
     Tables.istable(data) ||
         throw(ArgumentError("expected data in a Table, got $(typeof(data))"))

src/schema.jl

@@ -11,9 +11,9 @@
 terms(t::FormulaTerm) = union(terms(t.lhs), terms(t.rhs))
 terms(t::InteractionTerm) = terms(t.terms)
 terms(t::FunctionTerm) = mapreduce(terms, union, t.args)
-terms(t::AbstractTerm) = [t]
+terms(t::AbstractTerm) = AbstractTerm[t]
 terms(t::MatrixTerm) = terms(t.terms)
-terms(t::TupleTerm) = mapreduce(terms, union, t)
+terms(ts::AbstractVector{<:AbstractTerm}) = mapreduce(terms, union, ts, init=AbstractTerm[])
 
 needs_schema(::AbstractTerm) = true
 needs_schema(::ConstantTerm) = false
@@ -82,13 +82,13 @@ julia> ts = [Term(:x), Term(:y)];
 
 julia> schema(ts, d)
 StatsModels.Schema with 2 entries:
-  x => x
   y => y
+  x => x
 
 julia> schema(ts, d, Dict(:x => HelmertCoding()))
 StatsModels.Schema with 2 entries:
-  x => x
   y => y
+  x => x
 
 julia> schema(term(:y), d, Dict(:y => CategoricalTerm))
 StatsModels.Schema with 1 entry:
@@ -101,8 +101,8 @@ same in a container, but when printed alone are different:
 ```jldoctest 1
 julia> sch = schema(ts, d)
 StatsModels.Schema with 2 entries:
-  x => x
   y => y
+  x => x
 
 julia> term(:x)
 x(unknown)
@@ -117,16 +117,13 @@ y(continuous)
 schema(data, hints=Dict{Symbol,Any}()) = schema(columntable(data), hints)
 schema(dt::D, hints=Dict{Symbol,Any}()) where {D<:ColumnTable} =
     schema(Term.(collect(fieldnames(D))), dt, hints)
-schema(ts::AbstractVector{<:AbstractTerm}, data, hints::Dict{Symbol}) =
-    schema(ts, columntable(data), hints)
-
-# handle hints:
-schema(ts::AbstractVector{<:AbstractTerm}, dt::ColumnTable,
-       hints::Dict{Symbol}=Dict{Symbol,Any}()) =
-    sch = Schema(t=>concrete_term(t, dt, hints) for t in ts)
 
 schema(f::TermOrTerms, data, hints::Dict{Symbol}) =
-    schema(filter(needs_schema, terms(f)), data, hints)
+    schema(f, columntable(data), hints)
+
+# handle hints:
+schema(f::TermOrTerms, dt::ColumnTable, hints::Dict{Symbol}=Dict{Symbol,Any}()) =
+    Schema(t => concrete_term(t, dt, hints) for t in filter(needs_schema, terms(f)))
 
 schema(f::TermOrTerms, data) = schema(f, data, Dict{Symbol,Any}())
 
@@ -237,7 +234,16 @@ in _most_ cases, but cause method ambiguity in some.
 """
 apply_schema(t, schema) = apply_schema(t, schema, Nothing)
 apply_schema(t, schema, Mod::Type) = t
-apply_schema(terms::TupleTerm, schema, Mod::Type) = reduce(+, apply_schema.(terms, Ref(schema), Mod))
+# sequential (left-to-right) application over a vector of terms, combined with
+# `+` so that duplicates are dropped and nested vectors are flattened; always
+# returns a Vector{AbstractTerm}
+function apply_schema(terms::AbstractVector{<:AbstractTerm}, schema, Mod::Type)
+    out = AbstractTerm[]
+    for t in terms
+        out = out + apply_schema(t, schema, Mod)
+    end
+    return out
+end
 
 apply_schema(t::Term, schema::Schema, Mod::Type) = schema[t]
 apply_schema(ft::FormulaTerm, schema::Schema, Mod::Type) =
@@ -250,7 +256,7 @@ apply_schema(it::InteractionTerm, schema::Schema, Mod::Type) =
 apply_schema(t::Union{ContinuousTerm, CategoricalTerm}, schema::Schema, Mod::Type) =
     get(schema, term(t.sym), t)
 apply_schema(t::MatrixTerm, sch::Schema, Mod::Type) =
-    MatrixTerm(apply_schema.(t.terms, Ref(sch), Mod))
+    MatrixTerm(apply_schema(t.terms, sch, Mod))
 
 # TODO: special case this for <:RegressionModel ?
 function apply_schema(t::ConstantTerm, schema::Schema, Mod::Type)
@@ -390,7 +396,7 @@ has_schema(t::ConstantTerm) = false
 has_schema(t::Term) = false
 has_schema(t::Union{ContinuousTerm,CategoricalTerm}) = true
 has_schema(t::InteractionTerm) = all(has_schema(tt) for tt in t.terms)
-has_schema(t::TupleTerm) = all(has_schema(tt) for tt in t)
+has_schema(t::AbstractVector{<:AbstractTerm}) = all(has_schema(tt) for tt in t)
 has_schema(t::MatrixTerm) = has_schema(t.terms)
 has_schema(t::FormulaTerm) = has_schema(t.lhs) && has_schema(t.rhs)
 # FunctionTerms may always be transformed by apply_schema
@@ -503,8 +509,8 @@ end
 drop_term(from, to) = symequal(from, to) ? ConstantTerm(1) : from
 drop_term(from::FormulaTerm, to) = FormulaTerm(from.lhs, drop_term(from.rhs, to))
 drop_term(from::MatrixTerm, to) = MatrixTerm(drop_term(from.terms, to))
-drop_term(from::TupleTerm, to) =
-    tuple((t for t = from if !symequal(t, to))...)
+drop_term(from::AbstractVector{<:AbstractTerm}, to) =
+    AbstractTerm[t for t in from if !symequal(t, to)]
 function drop_term(from::InteractionTerm, t)
     terms = drop_term(from.terms, t)
     length(terms) > 1 ? InteractionTerm(terms) : terms[1]
@@ -537,7 +543,7 @@ The data variables that this term refers to.
 termvars(::AbstractTerm) = Symbol[]
 termvars(t::Union{Term, CategoricalTerm, ContinuousTerm}) = [t.sym]
 termvars(t::InteractionTerm) = mapreduce(termvars, union, t.terms)
-termvars(t::TupleTerm) = mapreduce(termvars, union, t, init=Symbol[])
+termvars(ts::AbstractVector{<:AbstractTerm}) = mapreduce(termvars, union, ts, init=Symbol[])
 termvars(t::MatrixTerm) = termvars(t.terms)
 termvars(t::FormulaTerm) = union(termvars(t.lhs), termvars(t.rhs))
 termvars(t::FunctionTerm) = mapreduce(termvars, union, t.args, init=Symbol[])

src/temporal_terms.jl

@@ -26,7 +26,7 @@ struct LeadLagTerm{T<:AbstractTerm, F<:Union{typeof(lead), typeof(lag)}} <: Abst
     nsteps::Int
 end
 
-terms(t::LeadLagTerm) = (t.term, )
+terms(t::LeadLagTerm) = AbstractTerm[t.term]
 
 function apply_schema(t::FunctionTerm{F}, sch::Schema, ctx::Type) where F<:Union{typeof(lead), typeof(lag)}
     opname = string(nameof(F.instance))