feat(tools): ensure-sort-keys — canonical DuckLake sort keys for the tenant column

[fuziontech]

Summary

Sets SET SORTED BY (team_id[, timestamp]) on the megaduck tenant tables — idempotently, from a new ensure-sort-keys subcommand + just ensure-sort-keys recipe for the maintenance CronJob chain.

Why

team_id is the universal filter column, but every DuckLake file spans the full team range (verified live on megaduck: avg per-file stats span = 563,336 of ~566k teams), so min/max pruning eliminates nothing and a team_id = X query is a full table scan — one of these OOM-killed the 26Gi reader pod.

With sort keys set, the DuckLake extension sorts new writes per file and honors the sort in compaction merges, so parquet row-group min/max stats become tight (verified locally: 4/4 row groups reduce to 1 relevant group for a single-team filter). The ClickHouse DuckLake reader already prunes row groups from embedded parquet stats — no reader change needed. Existing unsorted files improve progressively as compaction churns them.

Notes

• duckdb-side DDL (ALTER TABLE … SET SORTED BY), safe inside the extension's transaction semantics — unlike the CONCURRENTLY index recipes, which correctly stay on psql in the justfile.
• Catalog version stays 1.0 (verified): no reader-compat impact. duckdb 1.5.5 — the version viaduck and the maintenance image run.
• RESET+SET on changed keys: SET SORTED BY appends a sort spec rather than replacing, and the live spec is the end_snapshot IS NULL row of ducklake_sort_info (both covered by tests).
• Already applied to megaduck imperatively (2026-08-19); this codifies it so the operation is reproducible and kept in place by the cron.

Testing

New integration tests on a real local DuckLake catalog: apply, idempotence, change-of-key RESET, dry-run, and a pin of the canonical set. Full suite: 744 passed.

Agent-authored (Shelley on exe.dev).

[jghoman]

@fuziontech What about the row_id bug we saw?

[fuziontech]

@jghoman good question — I traced the row-lineage interaction before calling this safe, and I think sort keys are clean here. Analysis (happy to be corrected if you saw something different):

Deletes are physical-position based, so write-time sorting cannot invalidate them. DuckDB's delete machinery records FILE_ROW_NUMBER (GetRowIdColumns in ducklake_table_entry.cpp → inlined deletes and delete files both store the file-local position). A delete recorded later is resolved against the file's layout as it exists at that moment — sort order at write time doesn't enter into it.

Row lineage is stamped after the sort. PlanInsertSort puts the sort operator upstream of the copy-to-file operator in the insert plan (ducklake_insert.cpp:795), and _ducklake_internal_row_id is generated by the writer over the already-sorted rows, so row_id = row_id_start + position holds for sorted inserts.

Compaction is lineage-safe by construction: merge-adjacent skips delete-bearing files outright, tracks row-id adjacency across merge inputs (concatenates only adjacent ranges), writes explicit _ducklake_internal_row_id columns when a merge isn't adjacent, and rewrite_data_files always writes explicit row-ids after applying deletes (ducklake_compaction_functions.cpp:425-530). No path scrambles a delete reference.

Verified empirically before the prod ALTER: post-ALTER inserts produce tight, disjoint row-group team_id spans ([0..31],[31..61],[61..92],[92..100]); plain sorted inserts carry no lineage columns; catalog version stays 1.0.

If the bug you two saw was the reserved-field-id crash on millpond-written lineage columns (the one the reader hit pre-v4): orthogonal — that was reader strictness about Iceberg reserved-range field ids, already exempted on our port and upstream master. Worth noting sorted compaction will write explicit row_id columns more often (non-adjacent merges), so that exemption matters — and it's covered by tests.

If it was a different row_id bug (e.g. something in the ducklake_add_data_files corruption class), can you point me at it? I don't see a sort-key interaction with any of the candidates, but I'd rather check against the actual incident than reason from analogy.

[bill-ph]

Reviewed current head 55b8261. No P0 blockers found.\n\nNon-blocking finding:\n- [P1] The change adds the ensure-sort-keys command and just recipe, but the repository documents that the maintenance sequence is assembled in the external composition. Please confirm the production CronJob composition invokes this new recipe; otherwise the sort keys remain a manual operation and the stated runtime benefit will not be applied automatically.\n\nThe index/sort-key implementation is idempotent and tests are green.\n\n— Robo Bill