feat: Add SQL schema discovery library for PostgreSQL - #6812
Open
patelchaitany wants to merge 2 commits into
Open
feat: Add SQL schema discovery library for PostgreSQL#6812patelchaitany wants to merge 2 commits into
patelchaitany wants to merge 2 commits into
Conversation
Introduces feast.schema_discovery, which introspects a PostgreSQL database from an ad-hoc connection URL. It requires neither a feature_store.yaml nor a registered data source, so a user can explore a database before defining anything in Feast. SchemaDiscoveryService exposes list_schemas(), list_tables() and describe_table(). Column discovery deliberately combines two sources: - Feast's own DataSource supplies column types. Its get_table_column_names_and_types() reports canonical catalog spellings such as "character varying", which is what pg_type_to_feast_value_type keys on. SQLAlchemy reports "VARCHAR(255)" for the same column, which maps to UNKNOWN. - SQLAlchemy's Inspector supplies schema and table navigation, plus the nullability and primary-key metadata that a cursor description omits. Table and schema names reach the database through string interpolation in DataSource.get_table_query_string(), so identifiers are validated against a strict unquoted-identifier pattern before any connection is opened. That pattern is anchored with \Z rather than $, because $ also matches immediately before a trailing newline and would admit "users\n". Errors subclass FeastError and carry http_status_code(), so the REST layer added later can map them without a translation table. v1 accepts PostgreSQL only; every other dialect is refused by allowlist. Signed-off-by: Chaitany Patel <patelchaitany93@gmail.com>
Exercises SchemaDiscoveryService against a real PostgreSQL 16 instance
via testcontainers, following the PostgresContainer pattern already used
elsewhere in the integration suite.
Covers schema listing with system namespaces excluded, table and view
enumeration, column type mapping across the common PostgreSQL types,
primary-key and nullability reporting, describing a view, and the
not-found and connection-failure paths.
Includes a regression guard asserting that no column in the fixture
table maps to UNKNOWN. That assertion fails if column types ever start
coming from SQLAlchemy's type spellings ("VARCHAR(255)") rather than the
catalog spellings the Feast type mappers expect.
These tests require Docker and are marked with pytest.mark.integration.
Signed-off-by: Chaitany Patel <patelchaitany93@gmail.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6812 +/- ##
==========================================
+ Coverage 47.08% 47.26% +0.17%
==========================================
Files 419 425 +6
Lines 51878 52155 +277
Branches 7525 7549 +24
==========================================
+ Hits 24429 24649 +220
- Misses 25700 25754 +54
- Partials 1749 1752 +3
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
patelchaitany
force-pushed
the
feat/sql-schema-discovery-upstream
branch
2 times, most recently
from
September 8, 2026 10:08
3cf819e to
3e40a54
Compare
patelchaitany
marked this pull request as ready for review
September 8, 2026 11:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds
feast.schema_discovery, a library for introspecting a PostgreSQL database from an ad-hoc connection URL. It needs neither afeature_store.yamlnor a registered data source, so a user can explore a database before defining anything in Feast.SchemaDiscoveryServiceexposes three operations:list_schemas(url)list_tables(url, schema)describe_table(url, table, schema?)ValueTypeThis is the library phase only — no HTTP endpoint yet.
Design decisions
The choices worth arguing about, and what was rejected:
DataSourceInspectorfor everythingget_table_column_names_and_types()reports canonical catalog spellings (character varying) — exactly whatpg_type_to_feast_value_typekeys on. SQLAlchemy reportsVARCHAR(255), which maps toUNKNOWN. A SQLAlchemy-only build silently mistypes most columns.information_schemaqueriesDataSourcecannot list schemas or tables at all. SQLAlchemy already does this well per-dialect.DataSourcefrom the registryRepoConfigis built in memory; nothing is written to a registry.DataSource+ offline-store config pair to reuse decision 1. Unsupported dialects fail fast with a400rather than half-working.get_table_query_string(). Validation runs first so a malicious name never reaches a connection. Anchored with\Z, not$—$also matches before a trailing newline and would admit"users\n".http_status_code(), but no route ships{name, type}objectssslmodedefaults torequiredisablefor conveniencePostgreSQLConfigdefault. Override per-connection with?sslmode=disablein the URL.publicsearch_pathsearch_pathresolution can be revisited if anyone asks for it.describe_tableScope
Worth flagging for reviewers: reusing
DataSourcemeans Snowflake, BigQuery, Redshift and Trino are comparatively cheap to add later, whereas MySQL and SQLite have no offlineDataSourcein Feast and would each need new type mappers written first.Testing
PostgresContainerpattern.ruff check,ruff format --checkandmypy feast/schema_discovery/are clean.Important
The integration tests have not been executed. Docker was unavailable in the environment where this was written, so every assertion in
tests/integration/schema_discovery/is unverified. They collect cleanly and the file compiles, but CI is the first place they will actually run. Please treat that suite as unproven until CI reports on it.Still open
The REST endpoint is deliberately not in this PR. It carries a security question that deserves its own review: accepting arbitrary credentialed connection URLs lets an authenticated caller open outbound connections from the server (SSRF), and puts user database passwords through the API surface. That belongs in review of the endpoint, not the library.
Draft while the integration suite gets its first real CI run.