Embucket vs Snowflake parity harness for dbt-snowplow-web - #6
Open
rampage644 wants to merge 30 commits into
Open
Embucket vs Snowflake parity harness for dbt-snowplow-web#6rampage644 wants to merge 30 commits into
rampage644 wants to merge 30 commits into
Conversation
… Tables Keeps target schema identical to the synthetic loader so the existing dbt-snowplow-web wiring works unchanged; only sources.yml gets an identifier override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds scripts/ctas_from_glue.sql + scripts/load_from_glue.py to materialise the Hooli Iceberg-in-Glue source into the Embucket S3 Tables bucket as atomic.events_0416 (filtered to the post-realism-fix cutover at 2026-04-17 19:47 UTC). Points dbt-snowplow-web at the new table via snowplow__events_table, removes synthetic-specific var overrides, and makes the package's five context flags explicit-off. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both engines read the same S3 Tables Iceberg source; Athena loads it in two 30-minute batches; parity script rowcount- and MD5-diffs the three headline derived tables after each dbt run to surface semantic drift.
10 tasks: deps + schema split + load refactor + profile + parity script + snowflake setup/refresh + README + end-to-end smoke. Unit-tested hash and diff logic; integration steps exercise real Athena and Snowflake.
Adds pytest as a dev dependency. Pure hash/diff logic is unit tested; DB wrappers are exercised by the end-to-end smoke run.
Snowflake side: reuses the existing snowflake-table-bucket-access IAM role and creates a new SNOWPLOW_S3T Glue federated Iceberg REST catalog integration pointing at the snowplow bucket. The IAM policy was extended out-of-band to include the snowplow catalog ARNs, and Lake Formation DESCRIBE+SELECT was granted to the role on atomic.events_0416.
…ling Snowflake needs VARIANT columns (not VARCHAR) to index JSON contexts; a new sturukin_db.atomic.events_0416_v view TRY_PARSE_JSONs the 7 context columns and the snowplow__events_table var now dispatches on target.type. parity.py no longer attempts to hash-diff when rowcounts already differ (the first-order finding), which also avoids hitting the 6 MB Lambda response limit on 1M+ row tables.
…up source Clean batch-1-only investigation: source has 4.7% duplicate event_ids (Glue DQ issue). Scratch tables are essentially identical between engines. The divergence is in the final incremental MERGE: Snowflake errors loudly (ANSI-correct), Embucket silently lets duplicate unique_key rows into the derived tables.
…bucket MERGE First dbt run on either engine does CTAS (no MERGE) and both engines preserve the source's 4.7% duplicate event_ids into the derived tables identically. The MERGE divergence (Snowflake errors / Embucket appends) only manifests on a second run against the same scratch state. The missing dedup is in snowplow-web's _this_run -> derived step, not in the Embucket adapter.
…dings parity_deep.py returns one row per table (count, distinct key count, micros-epoch min/max start, a few SUM metrics, 16 MD5 position checksums) -- avoids Lambda payload limits and makes engine-vs-engine divergence directly comparable. Run against a clean first-dbt-run on both engines: distinct natural keys match exactly on every headline table. Divergences are (1) different duplicate-row multiplicity in the scratch-to-derived CTAS path and (2) occasional 1-second rounding on absolute_time_in_s from different timestamp-difference arithmetic between engines.
After fresh dbt deps: unpatched package errors at compile (Unexpected target type embucket). Patched package hits DataFusion memory-pool exhaustion on the 2.83M-row base-events repartition, even at 10 GB function memory with 10 GB disk spill. Snowflake on the same source runs to completion (20 PASS).
…er tie-break At 323,559 events, Embucket no longer OOMs and both engines produce matching rowcounts / distinct-key counts / user-table SUMs. Divergence shrinks to (1) 1-second rounding on page_view/session absolute_time_in_s and (2) 239 users where MAX(domain_sessionid) tie-breaks to a different first session whose referrer NULL-state differs.
9% of users (3,164 of 35,094) end up with smaller aggregates on Embucket than Snowflake after the second dbt run. Direction is strictly one-sided (sf >= emb, never the reverse). Consistent with Embucket's snowplow_web_users_this_run producing batch-2-only aggregates that then replace the batch-1 row via MERGE UPDATE, while Snowflake's producing cumulative aggregates.
Every upstream scratch stage has identical aggregates on both engines (users_aggs SUM(page_views) = 181,900 on both). The divergence appears only in the final snowplow_web_users derived table after MERGE: Embucket's target still carries batch-1 values while its own scratch source has the correct batch-1+batch-2 cumulative values and the compiled MERGE SQL is identical to Snowflake's. Snowflake applies the WHEN MATCHED UPDATE; Embucket does not -- 9% of users affected, all one-sided (sf >= emb). Likely bug in Embucket server-side MERGE UPDATE execution when the ON clause combines a BETWEEN predicate with an equality predicate.
Single standalone report covering setup, shared-source mechanics, tooling, checks performed at two data volumes and two dbt run paths (CTAS and MERGE), findings, and conclusions. Puts the localized Embucket MERGE UPDATE correctness issue (9% of users unchanged despite identical source and ON clause satisfied) alongside the upstream findings (source DQ, snowplow-web dedup assumptions, 1-second rounding) for context.
Hypothesis that MAX(uuid) tie-break caused the users.referrer drift is rejected (zero users have tied start_tstamp). The real bug: Embucket's planner resolves column references inside aggregate CASE expressions against SELECT-list aliases instead of FROM-clause columns. When snowplow_web_users_aggs aliases user_start_tstamp AS start_tstamp in the SELECT list and then references start_tstamp inside a MAX(CASE ...), Embucket rebinds it to user_start_tstamp, making the condition always-true and degenerating MAX to MAX(domain_sessionid). 10-line standalone reproducer confirmed. Silently wrong result, no error. Promoted to Embucket-specific engine issue in the conclusions.
…mula
Snowflake's DATEDIFF('second', a, b) is boundary-count
(floor(epoch_seconds(b)) - floor(epoch_seconds(a))). Embucket's is
CEIL((b - a) / 1 second). They agree only when both endpoints are on
exact second boundaries or the ceil value coincides with boundary
count. Seven-case standalone reproducer added; impact is strictly
one-sided inflation (+0 to +1 sec per row) on any DATEDIFF('second')
output, surfacing as +1.25% on page_views.sum_absolute_time_in_s and
+0.85% on sessions.sum_absolute_time_in_s at 609K events. Finding 3
re-attributed from 'both engines / truncation difference' to
Embucket-specific non-compliance with documented Snowflake semantics.
This was referenced Apr 23, 2026
…equivalent Add a Snowflake-only parity harness that runs the Snowplow dbt pipeline twice on identical input -- once with upstream models, once with the Apr-10 decomposed stash -- and diffs golden tables row-hash by row-hash. Result: all three golden tables inflate under the patch (users +71%, sessions +13.8%, page_views +5.0%). Root cause is the removed event_id QUALIFY in base_create_snowplow_events_this_run.sql; ~134k duplicate event_ids in the source propagate into every downstream count-based aggregate. See specs/2026-04-24-*-results.md for the fix sketch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes the base_create_snowplow_events_this_run.sql macro patch (which had silently dropped event_id dedup), keeping upstream's QUALIFY active. Adds manifest reset between variants so incremental comparisons aren't poisoned by shared Snowplow state; parity_self now excludes event_id from the page_view hash because row_number()=1 tiebreak over identical (derived_tstamp, dvce_created_tstamp) is non-deterministic in both pipelines (every non-event_id column matches). All three golden tables match exactly: 775,543 / 317,162 / 71,352 on full-refresh, 1,731,722 / 704,107 / 148,357 on incremental, 0 diffs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…+ harness Ran the same verification flow on Embucket Lambda. Both the upstream baseline and the iteration-2 patched build fail at snowplow_web_base_events_this_run with DataFusion RepartitionExec resource exhaustion: the Snowflake-equivalence fix kept upstream's wide-column QUALIFY in the base-events macro, which is exactly the OOM site. Adds Embucket-side harness (manifest reset, CTAS snapshot, row-hash parity) so the verification is ready to re-run once the base-events dedup is decomposed into the three-model narrow-scratch pattern sketched in the results doc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a narrow-scratch dedup chain for snowplow_web_base_events_this_run:
_raw_this_run (wide, dups allowed)
-> _winners_this_run (narrow: event_id + md5 of order-by tuple)
-> _this_run (INNER JOIN on event_id+winner_hash)
Each hop materialises so DataFusion's memory pool resets, keeping the
QUALIFY on a 3-column projection instead of 137. MD5 replaces Snowflake
HASH() for portability (DataFusion does not implement HASH yet).
Embucket: full-refresh + incremental both complete, 33/33 pass, no
OOM. Snowflake self-parity (baseline vs patched): 0 diffs across all
three golden tables in both fr and inc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a harness that loads the Glue Iceberg source into a single S3 Tables Iceberg table via Athena, runs dbt-snowplow-web on both Embucket and Snowflake against it, and diffs the three headline derived tables. Source parity is byte-for-byte; initial run surfaces the intended divergences between engines (recorded in
specs/2026-04-22-parity-results.md) rather than being silenced.🤖 Generated with Claude Code