fix(partition-mapping): stop epoch partition mirroring from dropping rows on hourly/daily keys

UPDATING.md

@@ -24,6 +24,8 @@ assists people when migrating to a new version.
 
 ## Next
 
+- Partition filter mapping no longer suggests bare `unix_timestamp(:value)` on Hive/Impala-family engines. Existing temporal-to-numeric mappings using that expression, or any transform that answers raw epoch instants in seconds through nanoseconds (for example `CAST(unix_timestamp(:value) AS BIGINT)` or `utc_to_unix_micros(:value)`, from a timestamp column or a text one in any common layout), stop pruning to prevent incorrect counts on hour/day epoch keys; this includes keys that really store raw epoch instants. Replace it with the expression that populates the partition key, including rounding and timezone. The dataset editor can explicitly compare a candidate against sampled rows from selected numeric partitions; a sample match is not a guarantee for the entire dataset.
+
 - With a Redis Sentinel coordination backend (`DISTRIBUTED_COORDINATION_CONFIG`
   using `RedisSentinelCache`), coordination commands now honor `CACHE_REDIS_DB` and
   `CACHE_REDIS_USER`. Previously they always ran on DB 0 and without the ACL user.

docs/admin_docs/configuration/partition-filter-mapping.mdx

@@ -66,21 +66,86 @@ Expand the mapped column's row in *Column Settings* and set the **value transfor
 expression containing a `:value` placeholder, which stands for the filter value being
 mirrored. The **Transform preserves ordering** checkbox sits directly beneath it.
 
-| Mapped column | Partition column | Transform |
-|---|---|---|
-| `event_time` (`TIMESTAMP`) | `dt_epoch` (`BIGINT`) | `unix_timestamp(:value)` |
-| `country` (`VARCHAR`) | `region_key` (`VARCHAR`) | `lower(:value)` |
+No engine default can infer how your pipeline buckets a numeric partition key.
+Enter the expression that actually populates it, including units and timezone.
 
-A filter of `event_time >= '2026-01-01'` then adds `dt_epoch >= 1767225600` to the query.
-The added predicate is an ordinary `WHERE` clause and shows up in **View query**.
+| Mapped column | Stored partition key | Transform |
+|---|---|---|
+| Epoch seconds (`BIGINT`, datetime format `epoch_s`) | Hour-start epoch seconds | `CAST(floor(:value / 3600) * 3600 AS BIGINT)` |
+| Epoch seconds (`BIGINT`, datetime format `epoch_s`) | UTC day-start epoch seconds | `CAST(floor(:value / 86400) * 86400 AS BIGINT)` |
+| UTC `TIMESTAMP`, with UTC epoch conversion in the database session | Hour-start epoch seconds | `CAST(floor(unix_timestamp(:value) / 3600) * 3600 AS BIGINT)` |
+| UTC `TIMESTAMP`, with UTC epoch conversion in the database session | UTC day-start epoch seconds | `CAST(floor(unix_timestamp(:value) / 86400) * 86400 AS BIGINT)` |
+| `country` (`VARCHAR`) | Lowercase `region_key` | `lower(:value)` |
+
+The timestamp examples use Hive/Impala syntax. Confirm conversion settings in both
+query and transform-evaluation sessions; `unix_timestamp` can depend on session timezone.
+The numeric epoch examples avoid that conversion. Use mathematical floor, including for
+negative epochs. Millisecond inputs require conversion to seconds before these expressions;
+local-calendar-day keys are not interchangeable with fixed UTC 86,400-second buckets.
+A session in a non-UTC zone shifts these boundaries: under IST (UTC+05:30) the hourly
+timestamp recipe lands on the wrong hour key, five or six hours earlier, so mirrored filters
+miss the matching rows. The daily recipe puts early-morning rows on the previous day.
+Set the evaluating session to UTC, or use the
+numeric epoch recipes.
+
+Bare `unix_timestamp(:value)` is not suggested or mirrored for temporal-to-numeric
+mappings on Hive/Impala and their engine-spec subclasses. It produces an instant,
+not an hour/day boundary, and can silently exclude matching rows. Existing expressions
+remain editable; replace them with the pipeline's actual partition expression.
+On the same engines, a transform from a timestamp or text column onto a numeric key is
+also evaluated at two instants 2,843 seconds apart within one hour. A text column is asked
+in each common layout (`2001-01-01 10:00:00`, `2001-01-01T10:00:00`,
+`2001-01-01T10:00:00Z`, and its own datetime format, if set), whether or not it is flagged
+as a datetime. A datetime format whose output contains a quote or backslash the engine
+would read differently is skipped rather than sent. If any answers are exactly that far apart in seconds, milliseconds,
+microseconds or nanoseconds, the transform returns raw instants, however it is spelled:
+`CAST(unix_timestamp(:value) AS BIGINT)`, `unix_timestamp(:value) + 0`,
+`floor(unix_timestamp(:value))` and `utc_to_unix_micros(:value)` are all declined. A key
+that really stores raw epoch instants is declined too; queries then scan every partition
+but return correct rows. This check cannot catch a wrong bucket size or timezone, so still
+check every transform with **Validate against data**.
+
+Dataset time-range bounds already pass through Superset's timezone/hour-offset adjustment
+before mirroring. Do not convert those bounds from the dataset timezone a second time.
+The added partition predicate remains visible in **View query**.
+
+### Validate against stored data
+
+The automatic preview reports **Expression evaluated — data not checked**. This means
+Superset could evaluate the expression and build a predicate, not that it matches your data.
+
+For a physical dataset with a numeric epoch key, enter one to three **existing partition
+keys** (whole numbers within the 64-bit range), then choose **Validate against data**. The
+section is not shown for a text partition key, and a disabled button says what is missing. Superset reads at most 50 rows from each
+selected partition and compares each transformed business value with that same row's
+stored key. It applies your data-access permissions and row-level security. It does not
+run a total-count query or infer sample partitions from the candidate expression.
+
+- **Matched sampled rows** reports the checked rows and covered partitions. This is
+  evidence about that sample, not proof for all rows, backfills, or future ingestion.
+- **Partition mismatch found** shows the business value, stored key, and transform output.
+  An output found in another partition still fails the same-row comparison.
+- **Validation inconclusive** names each selected key that had no checkable rows or whose
+  evaluation/query failed. NULL business values cannot establish a match.
+- **Filters will not be mirrored** means the mapping itself refuses the transform, for
+  example one that answers raw epoch instants. No rows are sampled: rows on the hour
+  boundary would match it exactly and report a transform the mirror declines as matching.
+
+Choose partitions containing off-boundary business timestamps, such as 10:47:23, and repeat
+across relevant dates and timezone transitions. The read is restricted to the selected keys;
+a row limit does not guarantee a cheap scan within a large partition. Existing warehouse
+query limits still apply. Changing the candidate clears the displayed validation.
+Validation does not save a global verdict, change chart behavior, or override the bare-epoch
+safety refusal. Before enabling a production mapping, compare chart row IDs and counts with
+and without its partition predicates, including equality, IN, and partial-bucket ranges.
 
 ### Transform preserves ordering
 
 Range filters — including the Explore time range, the most important case — are only
 mirrored when you check **Transform preserves ordering**.
 
 Monotonicity is a property of the *transform*, not of the column's data type.
-`unix_timestamp(:value)` preserves ordering. `hour(:value)`,
+`CAST(floor(unix_timestamp(:value) / 3600) * 3600 AS BIGINT)` preserves ordering. `hour(:value)`,
 `date_format(:value, 'dd')` and `dayofweek(:value)` are all perfectly reasonable
 partition transforms on a `TIMESTAMP` column and none of them do: `hour('2026-01-01 23:00')`
 is greater than `hour('2026-01-02 01:00')` even though the first instant is earlier. Mirroring
@@ -89,7 +154,7 @@ declare it rather than guessing.
 
 A *bucketing* transform preserves ordering and is the common case rather than
 an exception: `to_char(:value, 'YYYYMMDD')`, `date(:value)` and
-`cast(unix_timestamp(:value) / 86400 as bigint)` each map a whole day onto one
+`CAST(floor(unix_timestamp(:value) / 86400) * 86400 AS BIGINT)` each map a whole day onto one
 partition key, which is exactly what a daily-partitioned table wants. Check the
 box for these.
 
@@ -105,7 +170,7 @@ says why. Any other engine declines too, until it is confirmed and declared.
 If your partition column is temporal and your transform returns text, the robust answer is
 to make the transform's return type match the column: `date(:value)` or
 `cast(... as date)` rather than `to_char(...)`. A *numeric* partition column — the
-`unix_timestamp(:value)` / epoch shape — is unaffected by any of this.
+epoch shape, such as `CAST(floor(unix_timestamp(:value) / 3600) * 3600 AS BIGINT)` — is unaffected by any of this.
 
 When the box is unchecked, `=` and `IN` filters still mirror; ranges do not.
 
@@ -239,7 +304,7 @@ column. That substitution is only valid if, for every row in the table:
 partition_column = <transform>(mapped_column)
 ```
 
-**Superset cannot verify this.** It is a property of whatever ETL populates the partition
+**Sample validation cannot prove this for the entire dataset.** It is a property of whatever ETL populates the partition
 column. If that job lags, backfills with different logic, or writes the partition key in a
 different timezone than the transform resolves, mirrored predicates silently drop real rows
 and charts show quietly wrong numbers. Confirm the invariant with whoever owns the pipeline
@@ -280,12 +345,12 @@ accepts a comparison across those types: Postgres, Trino and BigQuery refuse it
 SQLite compares it as never equal rather than erroring, which would drop every row the filter
 keeps. So Superset declines the mirror instead. The preview panel names the column and the
 value it got, and the fix is to cast the result to the key's own type —
-`cast(unix_timestamp(:value) as string)` for a text partition key.
+`date_format(:value, 'yyyyMMddHH')` for a text partition key holding hours.
 
 Because the evaluation happens in a **different session** from the chart query, transforms
 that call non-deterministic functions are rejected when you save. That includes `now()`,
 `current_date`, `current_timestamp`, `rand()` and the zero-argument `unix_timestamp()`, which
-means "now" on Hive and Impala. The one-argument `unix_timestamp(:value)` is fine.
+means "now" on Hive and Impala. The one-argument function is deterministic only under consistent session settings; the bare expression is declined for temporal-to-numeric Hive/Impala-family mappings because it does not specify a bucket.
 
 Session-dependent behaviour that Superset cannot detect is still your responsibility:
 `unix_timestamp(:value)` is timezone-dependent on Hive and Impala, so if the evaluating
@@ -314,6 +379,10 @@ refused when you save:
   PostgreSQL large-object writer. The transform is evaluated against the engine to check
   that it mirrors, and that evaluation is cached and repeated, so it has to be a read.
 
+An ordinary comment that does not swallow `:value`, such as
+`floor(unix_timestamp(:value) / 3600) * 3600 -- hourly`, is allowed. Superset removes it
+before evaluating the transform.
+
 Only the mapped column may carry a transform. Setting one on any other column through the
 API, or in an import bundle, has no effect: Superset drops it on write, because a value parked on a column the
 mapping does not mirror is invisible in the editor and would go live the moment the mapped

docs/static/resources/openapi.json

@@ -8537,6 +8537,18 @@
             "minItems": 1,
             "type": "array"
           },
+          "validation_partition_values": {
+            "default": null,
+            "description": "One to three existing integer partition keys for explicit data validation",
+            "items": {
+              "pattern": "^-?[0-9]{1,19}$",
+              "type": "string"
+            },
+            "maxItems": 3,
+            "minItems": 1,
+            "nullable": true,
+            "type": "array"
+          },
           "value_transform": {
             "description": "SQL expression containing a :value placeholder",
             "maxLength": 1024,
@@ -25816,15 +25828,74 @@
                   "properties": {
                     "result": {
                       "properties": {
+                        "data_validation": {
+                          "properties": {
+                            "checked_rows": {
+                              "type": "integer"
+                            },
+                            "examples": {
+                              "items": {
+                                "properties": {
+                                  "source_value": {
+                                    "type": "string"
+                                  },
+                                  "stored_key": {
+                                    "type": "string"
+                                  },
+                                  "transformed_value": {
+                                    "nullable": true,
+                                    "type": "string"
+                                  }
+                                },
+                                "type": "object"
+                              },
+                              "type": "array"
+                            },
+                            "inconclusive_keys": {
+                              "description": "Partition keys whose rows could not be read, present when the status is inconclusive.",
+                              "items": {
+                                "type": "string"
+                              },
+                              "type": "array"
+                            },
+                            "message": {
+                              "type": "string"
+                            },
+                            "mismatch_count": {
+                              "type": "integer"
+                            },
+                            "partitions_checked": {
+                              "type": "integer"
+                            },
+                            "partitions_requested": {
+                              "type": "integer"
+                            },
+                            "skipped_rows": {
+                              "type": "integer"
+                            },
+                            "status": {
+                              "enum": [
+                                "not_checked",
+                                "matched",
+                                "mismatch",
+                                "inconclusive",
+                                "refused"
+                              ],
+                              "type": "string"
+                            }
+                          },
+                          "type": "object"
+                        },
                         "emitted_predicate": {
                           "type": "string"
                         },
                         "error": {
                           "type": "string"
                         },
                         "reason": {
-                          "description": "Why the mapping is not valid. Absent when it is. The editor branches on this to decide how to present the failure.",
+                          "description": "Why predicate evaluation was declined. Absent on evaluation success. The editor branches on this to decide how to present the failure.",
                           "enum": [
+                            "unbucketed_epoch",
                             "unconfigured",
                             "validation",
                             "parse",
@@ -25842,6 +25913,7 @@
                           "type": "string"
                         },
                         "valid": {
+                          "description": "Expression/predicate evaluation succeeded; not a guarantee of partition correspondence.",
                           "type": "boolean"
                         }
                       },

superset-frontend/src/components/Datasource/components/DatasourceEditor/components/PartitionFilterMapping/PartitionColumnFields.tsx

@@ -133,7 +133,13 @@ export default function PartitionColumnFields({
   // mapped column's row already asks it. Deferring to that answer where there
   // is one is what keeps this banner and that panel from contradicting each
   // other.
+  const refusal = datasource.partition_filter_mapping?.safety_refusal;
+  const refusalApplies =
+    refusal?.mapped_column === mappedColumn &&
+    refusal?.partition_column === datasource.partition_column &&
+    refusal?.value_transform === liveTransform;
   const isActive =
+    !refusalApplies &&
     mappingIsActive(datasource, allColumns) &&
     previewMirrors !== false &&
     !(
@@ -254,9 +260,13 @@ export default function PartitionColumnFields({
                 </Typography.Text>
               )}
               <Typography.Text type="secondary">
-                {t(
-                  'Filters on this column are mirrored onto the partition column.',
-                )}
+                {isActive
+                  ? t(
+                      'Filters on this column are mirrored onto the partition column.',
+                    )
+                  : t(
+                      'Filters on this column are mapped onto the partition column, but are not mirrored.',
+                    )}
                 {differentColumnTarget && (
                   <>
                     {' '}
@@ -306,15 +316,17 @@ export default function PartitionColumnFields({
                   type="warning"
                   showIcon
                   message={
-                    liveTransform
-                      ? t(
-                          'The value transform on %(mapped)s is not mirroring filters, so queries will scan every partition.',
-                          { mapped: mappedColumn },
-                        )
-                      : t(
-                          'No value transform is set on %(mapped)s, so nothing is mirrored yet and queries will scan every partition.',
-                          { mapped: mappedColumn },
-                        )
+                    refusalApplies
+                      ? refusal?.message
+                      : liveTransform
+                        ? t(
+                            'The value transform on %(mapped)s is not mirroring filters, so queries will scan every partition.',
+                            { mapped: mappedColumn },
+                          )
+                        : t(
+                            'No value transform is set on %(mapped)s, so nothing is mirrored yet and queries will scan every partition.',
+                            { mapped: mappedColumn },
+                          )
                   }
                 />
               )}