feat(semantic-layer): sort NULL dates last in the default order

UPDATING.md

@@ -75,6 +75,19 @@ assists people when migrating to a new version.
   up. Clients adding more views should split them across requests and retry
   timed-out views.
 
+- **Semantic-layer default order puts NULL dates last.** When a semantic query
+  has no explicit order, Superset sorts by its first date, timestamp or time
+  dimension ascending with NULL values last, and sends that order to the
+  provider as `OrderTerm(dimension, OrderDirection.ASC, NullPlacement.LAST)`.
+  `OrderTerm` still unpacks as a `(member, direction)` pair, so existing
+  providers keep running; to return the same rows as Superset's cache, a
+  provider must apply the placement (read it with `null_placement(term)`) before
+  `offset` and `limit`, and keep it when it rebuilds order terms. Explicit orders
+  are sent unchanged and keep the provider's own NULL placement. The provider
+  conformance kit adds a required `null_placement` query case. The semantic
+  result cache key format changes, so cached semantic results are recomputed
+  once after upgrading.
+
 - `superset-frontend` type-checks with TypeScript 7, whose native compiler no
   longer ships the JavaScript compiler API. The webpack dev server therefore
   no longer runs `fork-ts-checker-webpack-plugin`, and the `DISABLE_TS_CHECKER`

docs/developer_docs/semantic-layer-provider-testing.md

@@ -59,8 +59,18 @@ separate integration tests.
 | `semantic_change_catalog` | No-argument callable changing the controlled source catalog: add a visible member on every call, without clearing provider caches. |
 | `semantic_provider_class` | `(importable_module_name, class_name)` of the layer for an isolated import and static-schema check. Dependencies must already be installed. |
 
-Required query cases are `identity`, `grain`, `null`, `not_null`, `or`, and
-`order_limit`. Use member names distinct from private IDs and display labels.
+Required query cases are `identity`, `grain`, `null`, `not_null`, `or`,
+`order_limit` and `null_placement`. Use member names distinct from private IDs
+and display labels.
+For `null_placement`, order first by a selected dimension with an
+`OrderTerm(member, direction, NullPlacement.LAST)` (or `FIRST`) and a limit, and
+seed NULL and non-NULL values of that member so the limited page keeps different
+rows under the opposite placement. The SQLite example sorts `category` ascending
+with NULL values last and a limit of 2, keeping A and B where SQLite's own
+default would keep NULL and A. The suite also checks that the unlimited result
+puts every NULL value of that member at the requested end, and compares the
+limited page with the same rows of the unlimited result, so give the order key
+distinct values and a single NULL group: tied rows may come back in either order.
 For `grain`, use a grain-bearing dimension with data whose aggregation distinguishes the selected grain from raw values. Multiple grain variants for one name are allowed, not required. When the catalog declares no grain-bearing dimension, the case skips with an explicit reason before looking up its fixture entry; such providers may omit the `grain` entry. This reports unexercised grain coverage, not a passing grain check.
 NULL and grouped-OR fixtures must select NULL and non-NULL branches observably;
 the `or` case must use the host-normalized shape: one `OrFilter` containing
@@ -108,9 +118,10 @@ unverified release-gate coverage, not passing conformance; do not fabricate an
 SDK exception just to fill the matrix. The count lane only
 checks failure propagation; it does not settle pending count or value-shape
 rules. Construction fixtures avoid copying a factory signature or context type.
-The host supplies default chronological ordering before calling the provider.
-This suite checks the provider obligation to honor explicit ordering before limit
-and offset, including the ordered grain case.
+The host supplies default chronological ordering before calling the provider,
+as an ascending `OrderTerm` with NULL values last. This suite checks the provider
+obligation to honor explicit ordering and NULL placement before limit and offset,
+including the ordered grain case.
 
 The import/schema check starts a fresh Python interpreter with the test import
 path. It rejects imports of `superset`, `flask` and `flask_appbuilder`, including

superset-core/src/superset_core/semantic_layers/testing.py

@@ -42,6 +42,8 @@
     Dimension,
     Filter,
     Metric,
+    null_placement,
+    NullPlacement,
     Operator,
     OrFilter,
     SemanticQuery,
@@ -87,7 +89,8 @@ class SemanticProviderConformance:
     """Subclass in a ``Test...`` class and supply the documented provider fixtures.
 
     Query fixtures seed discriminating data and provide independent expected
-    Arrow tables for identity, grain, NULL, grouped OR, and order-before-limit.
+    Arrow tables for identity, grain, NULL, grouped OR, order-before-limit and
+    NULL placement.
     Error fixtures call real provider methods against controlled backend outcomes.
     See the provider author guide for every fixture and the limits of this suite.
     """
@@ -101,7 +104,16 @@ def test_catalog_identity(
         assert _catalog(semantic_view) == _catalog(semantic_reloaded_view)
 
     @pytest.mark.parametrize(
-        "case", ["identity", "grain", "null", "not_null", "or", "order_limit"]
+        "case",
+        [
+            "identity",
+            "grain",
+            "null",
+            "not_null",
+            "or",
+            "order_limit",
+            "null_placement",
+        ],
     )
     def test_query_outputs(
         self,
@@ -116,6 +128,10 @@ def test_query_outputs(
             pytest.skip("grain: catalog declares no grain-bearing dimension")
         query: SemanticQuery
         expected: pa.Table
+        assert case in semantic_query_cases, (
+            f"semantic_query_cases has no {case!r} case; see the provider "
+            "testing guide for the required cases"
+        )
         query, expected = semantic_query_cases[case]
         assert expected.num_rows > 0, (
             f"{case} must distinguish a correct result from no data"
@@ -148,6 +164,8 @@ def test_query_outputs(
             _assert_result_equal(
                 full.results.slice(query.offset or 0, query.limit), expected
             )
+        elif case == "null_placement":
+            _assert_null_placement_witness(query, semantic_view, expected)
         result: SemanticResult = semantic_view.get_table(query)
         _assert_result_equal(result.results, expected)
         assert set(result.results.column_names) == {
@@ -272,6 +290,56 @@ def test_imports_and_static_schema(
         assert completed.returncode == 0, completed.stdout + completed.stderr
 
 
+def _assert_null_placement_witness(
+    query: SemanticQuery, view: SemanticView, expected: pa.Table
+) -> None:
+    """Require NULL values of the first order key where its placement puts them.
+
+    The host's default chronological order asks for NULL values last; a
+    provider must honor a placement on any order key, before offset and limit.
+    The page must keep different rows from the ones the opposite placement would.
+    """
+    assert query.order, "null_placement fixture requires an order"
+    first: tuple[object, ...] = query.order[0]
+    placement: NullPlacement | None = null_placement(first)
+    assert placement is not None, (
+        "null_placement fixture's first order key requires a NullPlacement"
+    )
+    member: object = first[0]
+    assert isinstance(member, Dimension), (
+        "null_placement fixture must order by a selected dimension"
+    )
+    assert member in query.dimensions, (
+        "null_placement fixture must order by a selected dimension"
+    )
+    assert query.limit is not None
+    assert query.limit > 0
+    full: pa.Table = view.get_table(replace(query, limit=None, offset=None)).results
+    values: list[object] = full.column(member.name).to_pylist()
+    nulls: list[object] = [value for value in values if value is None]
+    others: list[object] = [value for value in values if value is not None]
+    assert nulls, "null_placement fixture requires NULL values of the order key"
+    assert others, "null_placement fixture requires non-NULL values of the order key"
+    placed: list[object] = (
+        others + nulls if placement is NullPlacement.LAST else nulls + others
+    )
+    opposite: list[object] = (
+        nulls + others if placement is NullPlacement.LAST else others + nulls
+    )
+    assert values == placed, (
+        f"{member.name}: NULL values must sort {placement.value.lower()}, got {values}"
+    )
+    start: int = query.offset or 0
+    stop: int = start + query.limit
+    assert sorted(map(repr, placed[start:stop])) != sorted(
+        map(repr, opposite[start:stop])
+    ), (
+        "null_placement fixture's page must keep different rows "
+        "under the opposite placement"
+    )
+    _assert_result_equal(full.slice(start, query.limit), expected)
+
+
 def _assert_normalized_or_witness(query: SemanticQuery) -> None:
     """Require the host's mixed-IN/NULL group and an outer comparison."""
     groups: list[OrFilter] = [

superset-core/src/superset_core/semantic_layers/types.py

@@ -315,9 +315,91 @@ class OrderDirection(enum.Enum):
     DESC = "DESC"
 
 
+class NullPlacement(enum.Enum):
+    """Where NULL values of an order key sort, relative to all other values."""
+
+    FIRST = "FIRST"
+    LAST = "LAST"
+
+
 OrderTuple = tuple[Metric | Dimension | AdhocExpression, OrderDirection]
 
 
+class OrderTerm(tuple[Metric | Dimension | AdhocExpression, OrderDirection]):
+    """
+    An ``(member, direction)`` order pair with an optional NULL placement.
+
+    It unpacks, indexes and has the length of a two-element ``OrderTuple``, so a
+    provider reading ``member, direction`` keeps working. Read the placement with
+    :func:`null_placement`, which also accepts a plain pair.
+
+    ``nulls`` of ``None`` means the provider's default placement, the same as a
+    plain pair, and such a term equals the plain pair. A term with a placement
+    differs from the pair without one: the provider must sort NULL values of
+    that key first or last, before ``offset`` and ``limit``.
+
+    ``dataclasses.asdict`` and ``astuple`` rebuild a term from its two
+    elements, so they keep the member and direction but not the placement;
+    serialize a term with :func:`null_placement` to keep it.
+    """
+
+    # Lets ``dataclasses.asdict``/``astuple`` rebuild a term from its elements.
+    _fields: tuple[str, str] = ("member", "direction")
+    nulls: NullPlacement | None
+
+    def __new__(
+        cls,
+        member: Metric | Dimension | AdhocExpression,
+        direction: OrderDirection,
+        nulls: NullPlacement | None = None,
+    ) -> OrderTerm:
+        if not isinstance(direction, OrderDirection):
+            raise TypeError("OrderTerm direction must be an OrderDirection")
+        if nulls is not None and not isinstance(nulls, NullPlacement):
+            raise TypeError("OrderTerm nulls must be a NullPlacement or None")
+        term: OrderTerm = super().__new__(cls, (member, direction))
+        object.__setattr__(term, "nulls", nulls)
+        return term
+
+    def __setattr__(self, name: str, value: object) -> None:
+        raise AttributeError("OrderTerm is immutable")
+
+    def __delattr__(self, name: str) -> None:
+        raise AttributeError("OrderTerm is immutable")
+
+    def __getnewargs__(
+        self,
+    ) -> tuple[
+        Metric | Dimension | AdhocExpression, OrderDirection, NullPlacement | None
+    ]:
+        return (self[0], self[1], self.nulls)
+
+    def __eq__(self, other: object) -> bool:
+        if not isinstance(other, tuple):
+            return NotImplemented
+        return tuple.__eq__(self, other) is True and self.nulls == null_placement(other)
+
+    def __ne__(self, other: object) -> bool:
+        equal: bool = self.__eq__(other)
+        return equal if equal is NotImplemented else not equal
+
+    def __hash__(self) -> int:
+        if self.nulls is None:
+            return tuple.__hash__(self)
+        return hash((tuple(self), self.nulls))
+
+    def __repr__(self) -> str:
+        return f"OrderTerm({self[0]!r}, {self[1]!r}, nulls={self.nulls!r})"
+
+
+def null_placement(term: tuple[object, ...]) -> NullPlacement | None:
+    """Return an order term's NULL placement, or ``None`` for the provider default.
+
+    Accepts a plain ``(member, direction)`` pair, which has no placement.
+    """
+    return getattr(term, "nulls", None)
+
+
 @dataclass(frozen=True)
 class GroupLimit:
     """
@@ -374,20 +456,25 @@ class SemanticQuery:
     Truncating first and sorting the truncated page returns a different, and
     wrong, answer. ``order`` entries are ``(member, direction)`` pairs whose
     member must be defined by the view; an adhoc expression is only a valid
-    order key for a view declaring ``ADHOC_EXPRESSIONS_IN_ORDERBY``.
+    order key for a view declaring ``ADHOC_EXPRESSIONS_IN_ORDERBY``. An entry
+    may be an :class:`OrderTerm` carrying a :class:`NullPlacement`; the
+    provider must then sort that key's NULL values first or last before
+    ``offset`` and ``limit``. A plain pair, or a term without a placement,
+    leaves NULL placement to the provider.
 
     **Default chronological order.** When ``order`` is empty (``None`` or
     ``[]``) and ``dimensions`` includes a date, timestamp or time dimension, the
     host sorts ascending by the first such dimension in ``dimensions`` (by list
-    position, not ``SemanticView.preferred_temporal_dimension``), before
-    ``offset`` and ``limit``. Rows that tie on it come back in no particular
-    order, and where its NULL values sort is not specified. The host applies
-    this both to the ``get_table`` query it sends the provider and to results it
-    serves from its cache. It is a default, never a prefix: a non-empty
-    ``order`` is used exactly as given. A ``get_table`` call therefore receives
-    an empty ``order`` only when no temporal dimension is selected, and may then
-    return rows in any order. ``get_row_count`` gets no default, since order
-    cannot change a count.
+    position, not ``SemanticView.preferred_temporal_dimension``), with its NULL
+    values last, before ``offset`` and ``limit``. Rows that tie on it come back
+    in no particular order. The host applies this both to the ``get_table``
+    query it sends the provider, as
+    ``OrderTerm(dimension, OrderDirection.ASC, NullPlacement.LAST)``, and to
+    results it serves from its cache. It is a default, never a prefix: a
+    non-empty ``order`` is used exactly as given. A ``get_table`` call therefore
+    receives an empty ``order`` only when no temporal dimension is selected, and
+    may then return rows in any order. ``get_row_count`` gets no default, since
+    order cannot change a count.
 
     ``group_limit`` is a separate ranking, with its own metric, direction and
     optional filters.

superset-core/src/superset_core/semantic_layers/view.py

@@ -172,8 +172,9 @@ def get_table(self, query: SemanticQuery) -> SemanticResult:
         Raise SemanticQueryRejectedError for deliberate input rejection.
         Leave operational and unexpected failures unclassified.
 
-        Sort by ``query.order`` before applying ``query.offset`` and
-        ``query.limit``; truncating before sorting returns the wrong rows. Name
+        Sort by ``query.order``, including any term's NULL placement, before
+        applying ``query.offset`` and ``query.limit``; truncating before sorting
+        returns the wrong rows. Name
         each result column after the member that produced it.
 
         Raise ``superset_core.semantic_layers.errors.SemanticResultCompletenessError``

superset-core/tests/semantic_layers/conformance_provider.py

@@ -37,6 +37,8 @@
     FilterExpression,
     Grains,
     Metric,
+    null_placement,
+    NullPlacement,
     Operator,
     OrFilter,
     SemanticQuery,
@@ -151,9 +153,9 @@ def get_table(self, query: SemanticQuery) -> SemanticResult:
             if any(member not in columns for member, _ in query.order):
                 raise SemanticQueryRejectedError(SemanticQueryErrorCode.INVALID_QUERY)
             sql += " ORDER BY " + ", ".join(
-                f'"{member.name}" {direction.value}'
-                for member, direction in query.order
-                if not isinstance(member, AdhocExpression)
+                f'"{term[0].name}" {term[1].value}{_nulls(null_placement(term))}'
+                for term in query.order
+                if not isinstance(term[0], AdhocExpression)
             )
         if query.limit is not None:
             sql += " LIMIT ? OFFSET ?"
@@ -181,6 +183,11 @@ def get_row_count(self, query: SemanticQuery) -> SemanticResult:
         raise SemanticQueryRejectedError(SemanticQueryErrorCode.UNSUPPORTED_QUERY)
 
 
+def _nulls(placement: NullPlacement | None) -> str:
+    """Render a requested NULL placement; without one, SQLite's default applies."""
+    return f" NULLS {placement.value}" if placement is not None else ""
+
+
 def _predicate(expression: FilterExpression, parameters: list[object]) -> str:
     """Render only the normalized predicates used by this example."""
     if isinstance(expression, OrFilter):