Skip to content

feat(bigquery): select only those columns that were configured via table mappings - #4740

Open
dtunikov wants to merge 1 commit into
bq/watermark-column-replicationfrom
bq/select-only-mirror-columns
Open

feat(bigquery): select only those columns that were configured via table mappings#4740
dtunikov wants to merge 1 commit into
bq/watermark-column-replicationfrom
bq/select-only-mirror-columns

Conversation

@dtunikov

@dtunikov dtunikov commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Select columns configured for the table explicitly instead of relying on SELECT * EXCEPT a, b, c FROM ....
If a column was removed on the source, but wasn't removed in PeerDB:

  • query all columns configured on the TableMapping lvl
  • get an error from BQ
  • exclude removed column from the selection set
  • store it in in-memory connector map with TTL
    This means that there could be gaps in the data for such column if it was removed and re-added later on the source (up to configured TTL for our in memory map)

@dtunikov
dtunikov requested review from a team as code owners August 27, 2026 14:16
@dtunikov dtunikov changed the title feat(bigquery): select only those columns that were configured via table mappings for… feat(bigquery): select only those columns that were configured via table mappings Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant