diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 35ce371..35ea6a8 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -45,7 +45,7 @@ env: # of that layer, so the imports this backend now makes do not resolve against # it. CORE_PKG: "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@release/v1.0.0.dev30" - TESTSUITE_PKG: "rhosocial-activerecord-testsuite @ git+https://github.com/rhosocial/python-activerecord-testsuite.git@feature/extract-epoch" + TESTSUITE_PKG: "rhosocial-activerecord-testsuite @ git+https://github.com/rhosocial/python-activerecord-testsuite.git@release/v1.0.0.dev17" jobs: test: diff --git a/README.md b/README.md index f2d32af..d597c6e 100644 --- a/README.md +++ b/README.md @@ -188,6 +188,31 @@ CI shards the emulator across several containers on distinct ports; see VIEW`. - **No connection pooling** in the backend itself. +## Introspection + +`backend.introspector` answers metadata questions from the project's +`INFORMATION_SCHEMA` views: datasets, tables, columns, views, primary keys, +partitions, and persistent routines (`INFORMATION_SCHEMA.ROUTINES` — UDFs, +aggregate and table functions, and stored procedures). The `introspect` +CLI exposes the same families. + +Some introspection families from other engines have no BigQuery equivalent, +and none is invented here: + +- **Types** — no standalone user-defined types; `COLUMNS.data_type` is the + only type surface. +- **Sequences** — no sequences; identity columns are reported by the + `columns` family. +- **Operators / operator classes / operator families** — no user-defined + operators. +- **Reservations and capacity commitments** — Admin API capacity objects, + not `INFORMATION_SCHEMA` schema metadata. +- **Jobs** — project-level job history (`INFORMATION_SCHEMA.JOBS*`), not a + describable schema object. + +See [Database Introspection](docs/en_US/bigquery_specific_features/introspection.md) +for the full contract. + ## Testing ```bash diff --git a/changelog.d/8.added.md b/changelog.d/8.added.md new file mode 100644 index 0000000..f5f70f3 --- /dev/null +++ b/changelog.d/8.added.md @@ -0,0 +1 @@ +Added BigQuery routines introspection. diff --git a/docs/en_US/README.md b/docs/en_US/README.md index e189f29..71e5ce4 100644 --- a/docs/en_US/README.md +++ b/docs/en_US/README.md @@ -12,8 +12,8 @@ a **dataset**, and unlike every other backend, column references never carry it. session-level current dataset, and the limits of single-level qualification - **[Database Introspection](bigquery_specific_features/introspection.md)**: what this backend answers from `INFORMATION_SCHEMA`, the billed scans behind the three data - families, the two `COLUMNS` projections and the emulator fallback, and the fifteen - CLI types + families, routines, the two `COLUMNS` projections and the emulator fallback, and the + fifteen CLI types - **[Temporal Expressions](bigquery_specific_features/datetime_expressions.md)**: the `seconds_between` spelling — `TIMESTAMP_DIFF(end, start, MILLISECOND) / 1000.0`, argument order, bind order, sign, and the one-millisecond resolution limit diff --git a/docs/en_US/bigquery_specific_features/introspection.md b/docs/en_US/bigquery_specific_features/introspection.md index c89070c..2c88703 100644 --- a/docs/en_US/bigquery_specific_features/introspection.md +++ b/docs/en_US/bigquery_specific_features/introspection.md @@ -10,6 +10,7 @@ The BigQuery backend answers three layers of questions: what the project and its - **Project and datasets**: `@@project_id`, and `INFORMATION_SCHEMA.SCHEMATA` (the one view that answers at project level) - **Tables and views**: `INFORMATION_SCHEMA.TABLES` / `VIEWS`, with the dataset-scoped partitions read reduced onto each table's stored bytes +- **Routines**: `INFORMATION_SCHEMA.ROUTINES`, for persistent UDFs (scalar, aggregate, table functions) and stored procedures - **Columns**: `INFORMATION_SCHEMA.COLUMNS`, whose projection is the only question that says which of its columns an endpoint carries - **Primary keys**: `TABLE_CONSTRAINTS` ⋈ `KEY_COLUMN_USAGE`; there is no index catalogue - **Table options and partitions**: `TABLE_OPTIONS` and `PARTITIONS` @@ -47,6 +48,7 @@ Read the `False` rows first: a probe that answers `False` says the question cann | `supports_partition_info()` | `True` | One table's partitions | | `supports_extensions()` | `False` (inherited) | Installed add-on components | | `supports_dataset_introspection()` | `True` | BigQuery's own dataset family | +| `supports_routine_introspection()` | `True` | BigQuery's own routines family | Three of the `False` rows are not "the engine cannot": they are "the catalogue does not carry it", and each says which part is missing. @@ -68,6 +70,7 @@ A `True` promises only that the statement renders and the project accepts it. It | Columns | `list_columns()` | `INFORMATION_SCHEMA.COLUMNS`, asked with a projection | | Primary key | `get_primary_key()`, `list_indexes()` | `TABLE_CONSTRAINTS` ⋈ `KEY_COLUMN_USAGE` | | Views | `list_views()`, `get_view_info()` | `INFORMATION_SCHEMA.VIEWS` | +| Routines | `list_routines()`, `get_routine_info()` | `INFORMATION_SCHEMA.ROUTINES` | | Table options | `list_table_options()` | `INFORMATION_SCHEMA.TABLE_OPTIONS` | | Partitions | `list_partitions()` | `INFORMATION_SCHEMA.PARTITIONS` | | Storage | `get_table_storage_info()`, `list_tables()` | `TABLE_OPTIONS`, `PARTITIONS` and `COLUMNS`, composed | @@ -78,6 +81,7 @@ A few consequences of each statement, in the order a reader will meet them: - **Tables.** `TABLE_TYPE` is BigQuery's own spelling — `BASE TABLE`, `CLONE`, `SNAPSHOT`, `EXTERNAL`, `VIEW`, `MATERIALIZED VIEW` — mapped onto core's enum with `CLONE` and `SNAPSHOT` counted as base tables, and the original spelling kept in `extra["bigquery_table_type"]`. `include_system` is ignored: the `TABLES` view lists no system tables. The table row also carries the catalog's `ddl` text and the `is_typed` / `is_insertable_into` flags in `extra`. - **Datasets.** `SCHEMATA` is the one view that answers at project level, and the only one that accepts a region qualifier (`region-us`, `region-europe-north1`, ...). It reports the dataset's location, DDL text, default collation and creation/modification times. - **Views.** `check_option` is documented as always `NULL` for BigQuery; it is selected so the parsed object can say so rather than assume it. `use_standard_sql` rides in `extra`. +- **Routines.** `routine_type` is the catalog's own word — `FUNCTION`, `AGGREGATE FUNCTION`, `TABLE FUNCTION` or `PROCEDURE`. `data_type` is the declared return type and is `NULL` for a procedure; `routine_body` says `SQL` or `EXTERNAL`, with `external_language` naming JavaScript for the external case; `is_deterministic` is read from the catalog's `YES`/`NO` into a boolean and stays `None` when the catalog does not know. The record also keeps the catalog's `created` / `last_altered` times, the remote-function `connection`, and the `ddl` text when the service reports one. - **Table options.** Options arrive one row per option — `description`, `expiration_timestamp`, `labels`, `partition_expiration_days`, `require_partition_filter`, `refresh_interval_minutes`, and the external-table ones — so the same option name can appear more than once with a different `option_type`. They are kept as rows rather than folded into a dict for that reason. - **Partitions.** `partition_id` is `NULL` for an unpartitioned table and `__NULL__` for the partition holding rows whose partitioning key is `NULL`. Each row carries rows, logical bytes, billable bytes, last-modified time and storage tier. With the table name omitted the same question is asked of the whole dataset — one row per partition of every table. - **Stored bytes.** A table's `size_bytes` is the sum of its partitions' `total_billable_bytes`, reduced from **one** dataset-scoped read rather than one read per table. A table with no partition rows — a view — keeps `None`, which is "not an object this engine stores bytes for", a different answer from zero. `data_size_bytes` and `index_size_bytes` stay `None` because managed storage has no index side to split off; the partition rows themselves stay on the `PARTITIONS` family, which is finer than the common minimum. @@ -97,7 +101,21 @@ All six answer `False`, and they are inherited rather than declared here — the A `False` here is "the question cannot be asked here", not "nothing was used". It does not say that a table was never read or that the project has no dependencies; it says the catalog cannot be asked *through this layer*, and the CLI refuses the family rather than running a statement the service would reject or printing an empty row that reads like an answer of zero. -The `ddl` family is the one worth stating carefully, because a table record does carry `TABLES.ddl` in its `extra` dict: that text is what the catalog hands back on the listing, not a DDL-extraction family. The `ddl` probe answers `False` because the family has no statement here, and the CLI offers the subcommand only so that asking it is a capability refusal rather than an unknown command. +The `ddl` family is worth stating carefully, because a table record does carry `TABLES.ddl` in its `extra` dict: that text is what the catalog hands back on the listing, not a DDL-extraction family. The `ddl` probe answers `False` because the family has no statement here, and the CLI does not offer the subcommand at all -- a type whose capability is permanently false is not advertised. + +## Object Families BigQuery Does Not Have + +Some families in the common introspection vocabulary name objects this engine has no equivalent of. They are not implemented here, and no empty family is offered for them; adding a `types` or `sequences` type that always answers `[]` would claim a catalogue the service does not expose. + +| Family | Why there is no implementation | +|--------|----------------------------------| +| Types | BigQuery has no standalone user-defined types. `COLUMNS.data_type` names a column's declared type; there is no project- or dataset-level type object to list separately, and no enum, domain, composite or range *type* catalogue. | +| Sequences | BigQuery has no sequences and no `nextval`-style generator. Identity columns and their bounds are reported through `INFORMATION_SCHEMA.COLUMNS` (`is_identity`, `identity_start`, `identity_increment`, ...), which is the column family, not a sequence family. | +| Operators, operator classes, operator families | BigQuery exposes functions and routines, not user-defined operators. There is no `pg_operator`-style catalogue. | +| Reservations, capacity commitments | These are BigQuery **Admin API** resources, not `INFORMATION_SCHEMA` objects. They are also project/organization-level capacity configuration rather than schema metadata a model could be inferred from, so they are intentionally left out of introspection. Use the Admin API (`google-cloud-bigquery-reservation`) when capacity management is the question. | +| Jobs | `INFORMATION_SCHEMA.JOBS*` is project-level job history, not schema. It is operational telemetry rather than a describable structure, is retained for a limited window, and is not a family a model generator should read. The service's job history and monitoring APIs are the right entry point. | + +`ROUTINES` is the one family in this table's neighbourhood that *is* implemented, because BigQuery does have persistent routines and the catalog documents them. ## Cost: Every Data Family Is a Billed Read @@ -209,6 +227,7 @@ Two smaller facts about the same view: a column whose `is_hidden` is `YES` is a - A real BigQuery answers the full 24-column `COLUMNS` projection; the emulator answers 10. The code is written so a real server is asked the documented list and only an endpoint that fails it falls back — but the full-list path itself has not been exercised against a real service in this repository's CI. - The emulator is a shim, not the service: it does not bill, it does not implement everything, and the workflow restarts it between shards because it degrades as tables accumulate. A statement that parses and returns rows there is a statement BigQuery should accept; it is not proof that a real project answered it. - Anything whose behaviour differs between the emulator and the service — the column projection above is the visible one — is written to prefer the service and to degrade to the emulator, not the other way round. +- The `routines` family is newer than that verification: its statement and parser are covered by offline tests, but no live project or emulator run has exercised `INFORMATION_SCHEMA.ROUTINES` from this repository yet. ## Floors and Version Gates @@ -235,6 +254,10 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect datasets \ # One dataset's tables, views and partitions python -m rhosocial.activerecord.backend.impl.bigquery introspect tables \ --project my-project --dataset my_dataset +python -m rhosocial.activerecord.backend.impl.bigquery introspect routines \ + --project my-project --dataset my_dataset +python -m rhosocial.activerecord.backend.impl.bigquery introspect routines add_one \ + --project my-project --dataset my_dataset python -m rhosocial.activerecord.backend.impl.bigquery introspect partitions orders \ --project my-project --dataset my_dataset @@ -247,8 +270,6 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect value-samples # The families this backend declines -- refused, not answered with an empty row python -m rhosocial.activerecord.backend.impl.bigquery introspect unused-indexes \ --project my-project --dataset my_dataset -python -m rhosocial.activerecord.backend.impl.bigquery introspect ddl orders \ - --project my-project --dataset my_dataset ``` Connection settings come from flags or from the environment: `--project` (`BIGQUERY_PROJECT` or `GOOGLE_CLOUD_PROJECT`), `--dataset` (`BIGQUERY_DATASET`), `--location` (`BIGQUERY_LOCATION`), `--credentials-path` (`BIGQUERY_CREDENTIALS_PATH` or `GOOGLE_APPLICATION_CREDENTIALS`), `--credentials-json` (`BIGQUERY_CREDENTIALS_JSON`), `--api-endpoint` (`BIGQUERY_API_ENDPOINT`, e.g. `http://localhost:9050` for the emulator). `--use-async` runs the same commands through the async backend. @@ -263,6 +284,7 @@ No command fabricates an answer: without a dataset the introspector raises with |------|-------------|------| | `tables` | Tables and views in a dataset | not accepted | | `views` | Views in a dataset | not accepted | +| `routines` | The dataset's routines, or one named routine | optional (routine) | | `columns` | One table's columns | required (table) | | `datasets` | The project's datasets | not accepted | | `database` | The project this run is connected to | not accepted | @@ -275,11 +297,10 @@ No command fabricates an answer: without a dataset the introspector raises with | `runtime` | Live runtime measurements | not accepted | | `index-stats` | Per-index usage counters | not accepted | | `dependencies` | The object dependency graph | optional (object) | -| `ddl` | The DDL that recreates one table | required (table) | -Fifteen types, and the advertised list is derived from the same mapping that gates each type on its switch, so a family cannot be offered by the CLI and forgotten by the probe. +Fifteen types, and the advertised list is derived from the same mapping that gates each type on its switch, so a family cannot be offered by the CLI and forgotten by the probe. `ddl` is not among them: `supports_ddl_extraction()` is `False` and no extraction statement exists, so the type is not advertised. -`database` and `partitions` are here because the dialect answers them and the CLI could not ask: `database` is the project-level question, and `partitions` is the engine's own family — one row per partition with the rows and the bytes each holds, finer than the common minimum the table listing reduces onto. A CLI that could not ask what the dialect declares would be "answerable but unaskable". +`database`, `routines` and `partitions` are here because the dialect answers them and the CLI could not ask: `database` is the project-level question, `routines` is the persistent-function/procedure family, and `partitions` is the engine's own family — one row per partition with the rows and the bytes each holds, finer than the common minimum the table listing reduces onto. A CLI that could not ask what the dialect declares would be "answerable but unaskable". This command has **no `info` subcommand**, so there is no second output-format list: every `introspect` type takes `-o table` (the default), `json`, `csv` or `tsv`, and `--rich-ascii` switches the table borders to ASCII. There is also no `--limit`: `value-samples` is bounded by the introspector's own default (`VALUE_SAMPLE_LIMIT`, 100), so the number is not written in two places. diff --git a/docs/zh_CN/README.md b/docs/zh_CN/README.md index 90943ef..e2acd84 100644 --- a/docs/zh_CN/README.md +++ b/docs/zh_CN/README.md @@ -11,7 +11,7 @@ BigQuery 后端是 [rhosocial-activerecord](https://github.com/rhosocial/python- 以及单级限定的局限 - **[数据库内省](bigquery_specific_features/introspection.md)**：本后端从 `INFORMATION_SCHEMA` 回答什么、三个数据族背后的计费扫描、两套 `COLUMNS` 投影与 - 模拟器回退，以及十五个 CLI 类型 + 模拟器回退、例程（routines），以及十五个 CLI 类型 - **[时间表达式](bigquery_specific_features/datetime_expressions.md)**： `seconds_between` 的写法——`TIMESTAMP_DIFF(end, start, MILLISECOND) / 1000.0`、 参数顺序、绑定顺序、符号，以及一毫秒的精度上限 diff --git a/docs/zh_CN/bigquery_specific_features/introspection.md b/docs/zh_CN/bigquery_specific_features/introspection.md index 8389c1c..16c8169 100644 --- a/docs/zh_CN/bigquery_specific_features/introspection.md +++ b/docs/zh_CN/bigquery_specific_features/introspection.md @@ -10,6 +10,7 @@ BigQuery 后端回答三个层次的问题：项目与其 dataset **声明**了 - **项目与 dataset**：`@@project_id`，以及 `INFORMATION_SCHEMA.SCHEMATA`（唯一在项目级作答的视图） - **表与视图**：`INFORMATION_SCHEMA.TABLES` / `VIEWS`，并把 dataset 级的 partition 读取归约到每张表的已存字节 +- **例程**：`INFORMATION_SCHEMA.ROUTINES`，用于持久化 UDF（标量、聚合、表函数）与存储过程 - **列**：`INFORMATION_SCHEMA.COLUMNS`，其投影是唯一能说明某个端点到底携带了哪些列的问题 - **主键**：`TABLE_CONSTRAINTS` ⋈ `KEY_COLUMN_USAGE`；不存在索引目录 - **表选项与分区**：`TABLE_OPTIONS` 与 `PARTITIONS` @@ -47,6 +48,7 @@ BigQuery 后端回答三个层次的问题：项目与其 dataset **声明**了 | `supports_partition_info()` | `True` | 一张表的分区 | | `supports_extensions()` | `False`（继承） | 已安装的附加组件 | | `supports_dataset_introspection()` | `True` | BigQuery 自己的 dataset 族 | +| `supports_routine_introspection()` | `True` | BigQuery 自己的例程族 | 三个 `False` 行并不是"引擎做不到"，而是"目录里没有这个事实"，并且每一行都说明了缺的是哪一部分。 @@ -68,6 +70,7 @@ BigQuery 后端回答三个层次的问题：项目与其 dataset **声明**了 | 列 | `list_columns()` | `INFORMATION_SCHEMA.COLUMNS`，按投影提问 | | 主键 | `get_primary_key()`、`list_indexes()` | `TABLE_CONSTRAINTS` ⋈ `KEY_COLUMN_USAGE` | | 视图 | `list_views()`、`get_view_info()` | `INFORMATION_SCHEMA.VIEWS` | +| 例程 | `list_routines()`、`get_routine_info()` | `INFORMATION_SCHEMA.ROUTINES` | | 表选项 | `list_table_options()` | `INFORMATION_SCHEMA.TABLE_OPTIONS` | | 分区 | `list_partitions()` | `INFORMATION_SCHEMA.PARTITIONS` | | 存储 | `get_table_storage_info()`、`list_tables()` | 组合 `TABLE_OPTIONS`、`PARTITIONS` 与 `COLUMNS` | @@ -78,6 +81,7 @@ BigQuery 后端回答三个层次的问题：项目与其 dataset **声明**了 - **表。** `TABLE_TYPE` 是 BigQuery 自己的拼写——`BASE TABLE`、`CLONE`、`SNAPSHOT`、`EXTERNAL`、`VIEW`、`MATERIALIZED VIEW`——映射到核心枚举，其中 `CLONE` 与 `SNAPSHOT` 计为基础表，原始拼写保留在 `extra["bigquery_table_type"]`。`include_system` 被忽略：`TABLES` 视图不列出系统表。表记录还把目录的 `ddl` 文本与 `is_typed` / `is_insertable_into` 标志放在 `extra` 里。 - **Dataset。** `SCHEMATA` 是唯一在项目级作答的视图，也是唯一接受 region 限定符的视图（`region-us`、`region-europe-north1` 等）。它报告 dataset 的位置、DDL 文本、默认排序规则与创建/修改时间。 - **视图。** `check_option` 在 BigQuery 上被文档规定永远为 `NULL`；仍然查询它，是为了让解析出的对象能这样说，而不是假定如此。`use_standard_sql` 放在 `extra`。 +- **例程。** `routine_type` 是目录自己的词——`FUNCTION`、`AGGREGATE FUNCTION`、`TABLE FUNCTION` 或 `PROCEDURE`。`data_type` 是声明的返回类型，存储过程为 `NULL`；`routine_body` 为 `SQL` 或 `EXTERNAL`，外部情形下 `external_language` 指出 JavaScript；`is_deterministic` 从目录的 `YES`/`NO` 读成布尔值，目录未知时保持 `None`。记录还保留目录的 `created` / `last_altered` 时间、远程函数的 `connection`，以及服务报告时的 `ddl` 文本。 - **表选项。** 选项以每行一个选项返回——`description`、`expiration_timestamp`、`labels`、`partition_expiration_days`、`require_partition_filter`、`refresh_interval_minutes`，以及外部表的那几个——因此同一选项名可能以不同的 `option_type` 出现多次。正因为如此，它们保持为行，而不折叠成一个字典。 - **分区。** 未分区表的 `partition_id` 为 `NULL`；分区键为 `NULL` 的那一行所在分区的 `partition_id` 为 `__NULL__`。每行带行数、逻辑字节、计费字节、最后修改时间与存储层级。省略表名时，同一个问题会问整个 dataset——每张表每个分区一行。 - **已存字节。** 一张表的 `size_bytes` 是其各分区 `total_billable_bytes` 之和，来自**一次** dataset 级读取的归约，而不是每张表读一次。没有任何分区行的表——视图——保持 `None`，表示"引擎不为这个对象存字节"，这与零是不同的答案。`data_size_bytes` 与 `index_size_bytes` 保持 `None`，因为托管存储没有可拆分出去的索引部分；分区级行本身留在 `PARTITIONS` 族里，比共同最低要求更细。 @@ -97,7 +101,21 @@ BigQuery 后端回答三个层次的问题：项目与其 dataset **声明**了 这里的 `False` 是"无法在此提出这个问题"，不是"什么都没用过"。它并不表示某张表从未被读、或该项目没有依赖；它表示无法**通过这一层**向目录提问，CLI 会直接拒绝该族，而不是执行一条服务会拒绝的语句、或打印一行看起来像"答案为零"的空结果。 -`ddl` 族值得特别说明，因为表记录的 `extra` 字典里确实带着 `TABLES.ddl`：那段文本是目录在列表查询中返回的内容，不是一个 DDL 提取族。`ddl` 探针回答 `False`，因为这里没有该族的语句；CLI 仍提供该子命令，只是为了让"问它"成为一次能力拒绝，而不是未知命令。 +`ddl` 族值得特别说明，因为表记录的 `extra` 字典里确实带着 `TABLES.ddl`：那段文本是目录在列表查询中返回的内容，不是一个 DDL 提取族。`ddl` 探针回答 `False`，因为这里没有该族的语句；CLI 根本不提供该子命令——能力永远为 false 的类型不会被公布。 + +## BigQuery 不存在的对象族 + +共同内省词汇中的某些族，在本引擎里没有对应对象。它们在这里没有实现，也不会为它们提供一个永远返回空列表的空族；新增一个总是回答 `[]` 的 `types` 或 `sequences` 类型，等于宣称服务暴露了一个它并没有的目录。 + +| 族 | 为什么没有实现 | +|----|----------------| +| 类型 | BigQuery 没有独立的用户自定义类型。`COLUMNS.data_type` 说明一列声明的类型；项目级或 dataset 级没有可单独列出的类型对象，也没有 enum、domain、composite、range 的**类型**目录。 | +| 序列 | BigQuery 没有序列，也没有 `nextval` 式生成器。身份列及其边界由 `INFORMATION_SCHEMA.COLUMNS`（`is_identity`、`identity_start`、`identity_increment` 等）报告，那属于列族，而不是序列族。 | +| 运算符、运算符类、运算符族 | BigQuery 暴露函数与例程，不暴露用户自定义运算符。不存在 `pg_operator` 式的目录。 | +| 预留（reservations）、容量承诺（capacity commitments） | 这些是 BigQuery **Admin API** 资源，不是 `INFORMATION_SCHEMA` 对象。它们还是项目/组织级容量配置，而不是可据以推断模型的 schema 元数据，因此被刻意排除在内省之外。需要管理容量时请使用 Admin API（`google-cloud-bigquery-reservation`）。 | +| 作业（jobs） | `INFORMATION_SCHEMA.JOBS*` 是项目级作业历史，不是 schema。它是运维遥测而非可描述的结构，只保留有限时间窗口，也不是模型生成器应当读取的族。作业历史与监控 API 才是正确的入口。 | + +`ROUTINES` 是这张表附近唯一**被实现**的族，因为 BigQuery 确实有持久化例程，而且目录有文档说明它们。 ## 费用：每个数据族都是一次计费读取 @@ -209,6 +227,7 @@ print(coverage.left_coverage, coverage.right_coverage) - 真实 BigQuery 会回答完整的 24 列 `COLUMNS` 投影，而模拟器只回答 10 列。代码的写法是：向真实服务器请求有文档的清单，只有失败的端点才回退——但完整清单那条路径本身，并未在本仓库的 CI 中对真实服务执行过。 - 模拟器是层壳，不是服务本身：它不计费、并不实现全部功能，而且工作流会在各 shard 之间重启它，因为随着表和作业累积它会退化。在那里能解析并返回行的语句，是 BigQuery **应当**接受的语句；它不是"真实项目已回答过它"的证明。 - 凡是在模拟器与服务之间行为不同的地方——上面的列投影就是可见的一例——代码都写成优先真实服务、并向模拟器降级，而不是反过来。 +- `routines` 族比上述验证更新：它的语句与解析器有离线测试覆盖，但本仓库尚未在真实项目或模拟器上实际执行过 `INFORMATION_SCHEMA.ROUTINES`。 ## 门槛与版本检查 @@ -235,6 +254,10 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect datasets \ # 一个 dataset 的表、视图与分区 python -m rhosocial.activerecord.backend.impl.bigquery introspect tables \ --project my-project --dataset my_dataset +python -m rhosocial.activerecord.backend.impl.bigquery introspect routines \ + --project my-project --dataset my_dataset +python -m rhosocial.activerecord.backend.impl.bigquery introspect routines add_one \ + --project my-project --dataset my_dataset python -m rhosocial.activerecord.backend.impl.bigquery introspect partitions orders \ --project my-project --dataset my_dataset @@ -247,8 +270,6 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect value-samples # 本后端拒绝的族——被拒绝，而不是回答一行空结果 python -m rhosocial.activerecord.backend.impl.bigquery introspect unused-indexes \ --project my-project --dataset my_dataset -python -m rhosocial.activerecord.backend.impl.bigquery introspect ddl orders \ - --project my-project --dataset my_dataset ``` 连接设置来自命令行参数或环境变量：`--project`（`BIGQUERY_PROJECT` 或 `GOOGLE_CLOUD_PROJECT`）、`--dataset`（`BIGQUERY_DATASET`）、`--location`（`BIGQUERY_LOCATION`）、`--credentials-path`（`BIGQUERY_CREDENTIALS_PATH` 或 `GOOGLE_APPLICATION_CREDENTIALS`）、`--credentials-json`（`BIGQUERY_CREDENTIALS_JSON`）、`--api-endpoint`（`BIGQUERY_API_ENDPOINT`，例如模拟器的 `http://localhost:9050`）。`--use-async` 让同样的命令走异步后端。 @@ -263,6 +284,7 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect ddl orders \ |------|------|------| | `tables` | dataset 中的表与视图 | 不接受 | | `views` | dataset 中的视图 | 不接受 | +| `routines` | dataset 的例程，或一个具名例程 | 可选（例程） | | `columns` | 一张表的列 | 必填（表） | | `datasets` | 项目的 dataset | 不接受 | | `database` | 本次运行所连接的项目 | 不接受 | @@ -275,11 +297,10 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect ddl orders \ | `runtime` | 实时运行时测量 | 不接受 | | `index-stats` | 每索引使用计数 | 不接受 | | `dependencies` | 对象依赖图 | 可选（对象） | -| `ddl` | 重建一张表的 DDL | 必填（表） | -十五个类型，公布的清单衍生自同一份把每个类型绑定到其开关的映射，因此不可能出现"CLI 提供了某一族、但探针忘了它"的情况。 +十五个类型，公布的清单衍生自同一份把每个类型绑定到其开关的映射，因此不可能出现"CLI 提供了某一族、但探针忘了它"的情况。`ddl` 不在其中：`supports_ddl_extraction()` 为 `False`，且这里没有提取语句，因此该类型不被公布。 -`database` 与 `partitions` 之所以在这里，是因为方言回答了它们、CLI 却问不到：`database` 是项目级的问题，`partitions` 是引擎自己的族——每分区一行，带行数与所占字节，比表列表归约到的共同最低要求更细。一个问不到方言所声明内容的 CLI，就是"能回答却无法提问"。 +`database`、`routines` 与 `partitions` 之所以在这里，是因为方言回答了它们、CLI 却问不到：`database` 是项目级的问题，`routines` 是持久化函数/存储过程族，`partitions` 是引擎自己的族——每分区一行，带行数与所占字节，比表列表归约到的共同最低要求更细。一个问不到方言所声明内容的 CLI，就是"能回答却无法提问"。 该命令**没有 `info` 子命令**，因此没有第二份输出格式清单：每个 `introspect` 类型都支持 `-o table`（默认）、`json`、`csv` 或 `tsv`，`--rich-ascii` 把表格边框换成 ASCII。也没有 `--limit`：`value-samples` 由内省器自己的默认值（`VALUE_SAMPLE_LIMIT`，100）界定，因此这个数字不会被写两遍。 diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/cli/README.md b/src/rhosocial/activerecord/backend/impl/bigquery/cli/README.md index a1e4bdb..d73bc05 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/cli/README.md +++ b/src/rhosocial/activerecord/backend/impl/bigquery/cli/README.md @@ -21,6 +21,7 @@ different answers, and only one of them is what happened. | :--- | :--- | :--- | | `tables` | `INFORMATION_SCHEMA.TABLES` | Tables and views, with `table_type` distinguishing `BASE TABLE`, `VIEW`, `MATERIALIZED VIEW` and `EXTERNAL`. | | `views` | `INFORMATION_SCHEMA.VIEWS` | Views, with their definitions. | +| `routines` | `INFORMATION_SCHEMA.ROUTINES` | Persistent UDFs (scalar, aggregate and table functions) and stored procedures, with their definition, return type and determinism. | | `columns` | `INFORMATION_SCHEMA.COLUMNS` | One table's columns: name, type, nullability, ordinal position, default, and the generation and identity facts a real server answers. | | `datasets` | `INFORMATION_SCHEMA.SCHEMATA` | The project's datasets. A dataset is BigQuery's namespace level above a table, so it stands where a schema would elsewhere. | | `database` | `SELECT @@project_id` | The project this run connected to. The project-level question, so it takes no dataset and no name. | @@ -33,11 +34,15 @@ different answers, and only one of them is what happened. | `runtime` | — | Refused: BigQuery reports no runtime metrics. | | `index-stats` | — | Refused: there are no per-index counters. | | `dependencies` | — | Refused: no view names an object's dependants. | -| `ddl` | — | Refused: `TABLES.ddl` describes tables and external tables, not a routine or a view's definition. | -`NAME` is required for `columns`, `partitions`, `ddl`, `column-profiles`, -`value-samples` and `join-coverage`; optional for `dependencies`; refused for -`database`. The list types take no name. A `NAME` given where it is refused +`ddl` is not offered at all: `supports_ddl_extraction()` answers `False` and +no extraction statement exists here, so a subcommand would only ever be a +refusal. `TABLES.ddl` is a field of the table listing, not a DDL-extraction +family. + +`NAME` is required for `columns`, `partitions`, `column-profiles`, +`value-samples` and `join-coverage`; optional for `routines` and +`dependencies`; refused for `database`. The list types take no name. A `NAME` given where it is refused exits 2 rather than being parsed and dropped, and a flag the type does not read (`--column`, `--right-table`, `--right-column`) is refused the same way. @@ -75,6 +80,12 @@ exits 2 rather than being parsed and dropped, and a flag the type does not read `column-profiles` reads the whole table and is the most expensive thing this CLI can be asked. The sentence on stderr before the rows says so; each row's `source='aggregate'` says the figures are exact counts. +* **There are no type, sequence or operator types here.** BigQuery has no + standalone user-defined types, no sequences and no user-defined + operators, so none of those families is invented. Identity columns are + reported under `columns`, not as a sequence family. Reservations, + capacity commitments and job history are Admin API / operational + resources rather than schema objects, so they are out of scope too. * **Nothing here was run against a live project.** The statements are written from the official documentation (cited beside each one in the source). Verify the first run against your project before relying on the @@ -92,9 +103,11 @@ exits 2 rather than being parsed and dropped, and a flag the type does not read python -m rhosocial.activerecord.backend.impl.bigquery introspect datasets --project my-project python -m rhosocial.activerecord.backend.impl.bigquery introspect database --project my-project -# One dataset's tables and views +# One dataset's tables, views and routines python -m rhosocial.activerecord.backend.impl.bigquery introspect tables --dataset events -o json python -m rhosocial.activerecord.backend.impl.bigquery introspect views --dataset events +python -m rhosocial.activerecord.backend.impl.bigquery introspect routines --dataset events +python -m rhosocial.activerecord.backend.impl.bigquery introspect routines add_one --dataset events # One table's columns, and its partitions python -m rhosocial.activerecord.backend.impl.bigquery introspect columns orders --dataset events @@ -107,7 +120,7 @@ python -m rhosocial.activerecord.backend.impl.bigquery introspect join-coverage --column buyer_id --right-table users --right-column id --dataset events # A family this project declines: refused, not answered with an empty row -python -m rhosocial.activerecord.backend.impl.bigquery introspect ddl orders --dataset events +python -m rhosocial.activerecord.backend.impl.bigquery introspect unused-indexes --dataset events # Against a local emulator python -m rhosocial.activerecord.backend.impl.bigquery introspect tables \ diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/cli/introspect.py b/src/rhosocial/activerecord/backend/impl/bigquery/cli/introspect.py index 919158b..e0afc61 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/cli/introspect.py +++ b/src/rhosocial/activerecord/backend/impl/bigquery/cli/introspect.py @@ -1,6 +1,7 @@ # src/rhosocial/activerecord/backend/impl/bigquery/cli/introspect.py """The introspection subcommands: the catalogue questions, the engine's own -``partitions`` family, and the nine families the dialect answers or declines. +``partitions`` family, the routines family, and the nine families the +dialect answers or declines. Each one wires ``backend.introspector`` to an output format: @@ -8,6 +9,7 @@ Command Introspector question ``tables`` ``list_tables()`` ``views`` ``list_views()`` +``routines [NAME]`` ``list_routines()`` / ``get_routine_info()`` ``columns NAME`` ``list_columns()`` ``datasets`` ``list_datasets()`` (the SCHEMATA view) ``database`` ``get_database_info()`` (@@project_id) @@ -17,10 +19,12 @@ ``join-coverage NAME`` ``estimate_join_coverage()`` ======================== ======================================= -The six families this backend declines -- ``unused-indexes``, -``table-stats``, ``runtime``, ``index-stats``, ``dependencies`` and -``ddl`` -- are listed too, so that asking one of them is answered by the -capability gate rather than by an "unknown command". +The five families this backend declines -- ``unused-indexes``, +``table-stats``, ``runtime``, ``index-stats`` and ``dependencies`` -- are +listed too, so that asking one of them is answered by the capability gate +rather than by an "unknown command". ``ddl`` is not offered at all: the +dialect's ``supports_ddl_extraction()`` is False and no extraction statement +exists here, so the subcommand would only ever be a refusal. ``--dataset`` selects the dataset every question is scoped to, and the schema argument each question receives is BigQuery's **dataset** -- the two @@ -72,16 +76,20 @@ #: Both are declared by the dialect, so a CLI that could not ask them would #: be "answerable but unaskable", the defect §6 exists to prevent. #: -#: BigQuery declares three of the nine families beyond its catalogue and its +#: BigQuery declares three of the families beyond its catalogue and its #: partitions view: the three that read the *rows* (a profile, a value #: listing, an overlap measurement), because ``INFORMATION_SCHEMA`` describes #: what a column declares and no view reports what its rows hold. The other -#: six answer False -- there is no unused-index view, no usage counters, no -#: runtime metrics and no dependency graph in ``INFORMATION_SCHEMA`` -- so -#: their subcommands exist to be refused rather than to run. +#: five answer False -- there is no unused-index view, no usage counters, no +#: runtime metrics, no index counters and no dependency graph in +#: ``INFORMATION_SCHEMA`` -- so their subcommands exist to be refused rather +#: than to run. ``ddl`` is absent from this mapping because +#: ``supports_ddl_extraction()`` is False and there is no extraction statement +#: here; a subcommand whose capability is permanently false is not advertised. _TYPE_CAPABILITIES = { "tables": "supports_table_introspection", "views": "supports_view_introspection", + "routines": "supports_routine_introspection", "columns": "supports_column_introspection", "datasets": "supports_dataset_introspection", "database": "supports_database_info", @@ -91,7 +99,6 @@ "runtime": "supports_runtime_stats", "index-stats": "supports_index_stats", "dependencies": "supports_object_dependencies", - "ddl": "supports_ddl_extraction", "column-profiles": "supports_column_data_profile", "value-samples": "supports_column_value_sample", "join-coverage": "supports_join_coverage", @@ -113,18 +120,20 @@ #: ``--help`` print, and the two tables cover the same keys: a subcommand #: offered without a rule is one whose NAME decision nobody has made. #: -#: ``tables``, ``views`` and ``datasets`` are listings over the whole -#: dataset or project and take no positional at all; argparse refuses a -#: stray one. ``database`` answers for the project the run connected to, and -#: the four counter families for the dataset as a whole, so a name beside -#: any of them is refused rather than parsed and dropped. ``dependencies`` -#: is the one family that can also answer for one object's edges, so its -#: name is optional. ``partitions`` is a one-table question: without a name +#: ``tables`` and ``datasets`` are listings over the whole dataset or +#: project and take no positional at all; argparse refuses a stray one. +#: ``database`` answers for the project the run connected to, and the four +#: counter families for the dataset as a whole, so a name beside any of them +#: is refused rather than parsed and dropped. ``routines`` is a listing that +#: can also answer for one named routine, and ``dependencies`` can answer for +#: one object's edges, so both names are optional. ``partitions`` is a +#: one-table question: without a name #: the read is the dataset-wide one the storage reducer asks, which is a #: different question the statement has no filter for. _NAME_RULE = { "tables": "forbidden", "views": "forbidden", + "routines": "optional", "columns": "required", "datasets": "forbidden", "database": "forbidden", @@ -134,7 +143,6 @@ "runtime": "forbidden", "index-stats": "forbidden", "dependencies": "optional", - "ddl": "required", "column-profiles": "required", "value-samples": "required", "join-coverage": "required", @@ -148,11 +156,11 @@ ) #: The subcommands that can also answer for one named object but answer the -#: whole family without it. ``dependencies`` narrows to one object's edges -#: when a name is given; omitting it asks for the whole graph. BigQuery -#: declines the family, so both spellings reach the same refusal -- but the -#: name still has to parse, or the refusal would be argparse's instead of -#: the capability answer. +#: whole family without it. ``routines`` narrows to one routine when a name +#: is given; ``dependencies`` narrows to one object's edges. BigQuery +#: declines the dependency family, so both of its spellings reach the same +#: refusal -- but the name still has to parse, or the refusal would be +#: argparse's instead of the capability answer. _OPTIONAL_NAME_COMMANDS = tuple( name for name, rule in _NAME_RULE.items() if rule == "optional" ) @@ -166,6 +174,10 @@ %(prog)s tables --project my-project --dataset my_dataset %(prog)s views --project my-project --dataset my_dataset + # The dataset's routines, and one named routine + %(prog)s routines --project my-project --dataset my_dataset + %(prog)s routines my_function --project my-project --dataset my_dataset + # One table's columns, and its partitions %(prog)s columns orders --project my-project --dataset my_dataset %(prog)s partitions orders --project my-project --dataset my_dataset @@ -179,7 +191,6 @@ # The families BigQuery declines -- refused, not answered with an empty row %(prog)s unused-indexes --project my-project --dataset my_dataset %(prog)s dependencies --project my-project --dataset my_dataset - %(prog)s ddl orders --project my-project --dataset my_dataset # JSON output, and the environment-variable form %(prog)s tables -o json @@ -271,6 +282,7 @@ def create_parser(subparsers): _HELP = { "tables": "List the tables and views in a dataset", "views": "List the views in a dataset", + "routines": "List a dataset's routines, or report one named routine", "columns": "List one table's columns", "datasets": "List the project's datasets", # The project-level question: BigQuery answers it from the documented @@ -290,7 +302,8 @@ def create_parser(subparsers): "runtime": "Live runtime measurements (not supported by BigQuery)", "index-stats": "Per-index usage counters (not supported by BigQuery)", "dependencies": "The object dependency graph (not supported by BigQuery)", - "ddl": "The DDL that recreates one table (not supported by BigQuery)", + # ``ddl`` is not offered: supports_ddl_extraction() is False and no + # extraction statement exists, so the subcommand would only be refused. # The three families that read the rows rather than the catalogue. "column-profiles": "Profile every column of a table from its data", "value-samples": "List one column's most frequent values", @@ -300,7 +313,6 @@ def create_parser(subparsers): _POSITIONAL_HELP = { "columns": "Table name (required)", "partitions": "Table name (required)", - "ddl": "Table name (required)", "column-profiles": "Table name (required)", "value-samples": "Table name (required)", "join-coverage": "Left table name (required)", @@ -559,11 +571,20 @@ def _index_usage_stats(introspector): def _database_info(introspector): return introspector.get_database_info() + def _routines(introspector): + # A family with one listing and one named-object question: the name + # is optional, so it narrows the same catalogue read rather than + # selecting a different method. + if args.name: + return introspector.get_routine_info(args.name, schema=schema) + return introspector.list_routines(schema=schema) + return { "tables": lambda introspector: introspector.list_tables( schema=schema, include_system=args.include_system ), "views": lambda introspector: introspector.list_views(schema=schema), + "routines": _routines, "columns": lambda introspector: introspector.list_columns( args.name, schema=schema ), @@ -587,9 +608,6 @@ def _database_info(introspector): object_name=args.name, schema=schema ), # ---- the one-object and two-object families ------------------- - "ddl": lambda introspector: introspector.get_ddl_statement( - args.name, object_type=None, schema=schema - ), "column-profiles": lambda introspector: introspector.list_column_data_profiles( args.name, schema=schema ), @@ -618,40 +636,6 @@ def _async_questions(args) -> Dict[str, Callable[[Any], Any]]: return _questions(args) -def _display_ddl(args, provider, statement) -> None: - """Show one object's DDL as the engine handed it back. - - A DDL text is a block, not a field of a row: a bordered cell wraps every - one of its lines and reads worse than the text itself. So the terminal - format prints the engine's own text, and the formats a caller parses - keep the record -- schema, object type, object name, ``native`` and the - text -- which is what ``native`` is for: it says the engine retrieved - this rather than the library assembling it. - """ - record = _serialize_for_output(statement) - if isinstance(record, dict): - text = record.get("ddl") - object_name = record.get("object_name") or args.name - object_type = record.get("object_type") or "OBJECT" - else: # pragma: no cover - a statement is a record, not a scalar - text, object_name, object_type = statement, args.name, "OBJECT" - if not text: - # Not "no such table": the object exists and the engine answers - # with no text rather than an assembled fragment that would read as - # authoritative. The two are worth telling apart, so this exits - # without printing an empty row. - print( - f"Error: no DDL available for '{args.name}'", - file=sys.stderr, - ) - sys.exit(1) - if args.output == "table": - print(f"-- {object_type} {object_name}") - print(text) - return - provider.display_results([record], title=f"DDL of {object_name}") - - def _report(result: Any, args, provider) -> None: """Print one question's result. @@ -659,8 +643,20 @@ def _report(result: Any, args, provider) -> None: together -- columns and primary key -- are now the ``columns`` command and the key columns it reports. """ - if args.command == "ddl": - _display_ddl(args, provider, result) + if args.command == "routines" and getattr(args, "name", None): + # A named-routine answer is one record, not a listing. ``None`` is + # "the dataset has no routine with that name", which is not an empty + # listing and should not exit 0 with a blank table. + if result is None: + print( + f"Error: no routine named '{args.name}' in the dataset", + file=sys.stderr, + ) + sys.exit(1) + provider.display_results( + [_serialize_for_output(result)], + title=_TITLES.get(args.command, args.command.title()), + ) return if args.command == "database": @@ -728,6 +724,7 @@ def _report(result: Any, args, provider) -> None: #: families read better as what they measure than as their subcommand name. _TITLES = { "database": "Database Info", + "routines": "Routines", "partitions": "Partitions", "unused-indexes": "Indexes with no recorded use", "table-stats": "Table usage counters", diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/expression/introspection.py b/src/rhosocial/activerecord/backend/impl/bigquery/expression/introspection.py index c8cf0eb..40be889 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/expression/introspection.py +++ b/src/rhosocial/activerecord/backend/impl/bigquery/expression/introspection.py @@ -48,6 +48,7 @@ primary keys ``INFORMATION_SCHEMA.TABLE_CONSTRAINTS`` joined to ``KEY_COLUMN_USAGE`` views ``INFORMATION_SCHEMA.VIEWS`` +routines ``INFORMATION_SCHEMA.ROUTINES`` datasets ``INFORMATION_SCHEMA.SCHEMATA`` table options ``INFORMATION_SCHEMA.TABLE_OPTIONS`` partitions ``INFORMATION_SCHEMA.PARTITIONS`` @@ -151,6 +152,7 @@ "BigQueryIndexInfoQueryStatement", "BigQueryViewListQueryStatement", "BigQueryViewInfoQueryStatement", + "BigQueryRoutineListQueryStatement", "BigQueryDatasetListQueryStatement", "BigQueryTableOptionsQueryStatement", "BigQueryPartitionListQueryStatement", @@ -382,6 +384,45 @@ def format_method(self) -> str: return "format_bigquery_view_info_query_statement" +class BigQueryRoutineListQueryStatement(IntrospectionExpression): + """The ``INFORMATION_SCHEMA.ROUTINES`` question. + + BigQuery routines are the persistent user-defined functions, aggregate + functions, table functions and stored procedures a dataset holds. The + view is dataset-scoped, like ``VIEWS``, and the documented projection is + asked for in the order the catalog documents. + + The ``routine_name`` slot is how one named routine is asked for: with a + name the same question is filtered to that row, which is the ``get`` half + of the family. Omitting it lists the dataset's routines. + + Args: + dialect: The BigQuery dialect that renders this statement. + schema: The dataset, or a ``project.dataset`` pair. Required. + catalog: The project to qualify the dataset with, or ``None``. + routine_name: The routine to narrow to, or ``None`` for the listing. + + https://cloud.google.com/bigquery/docs/information-schema-routines + """ + + def __init__( + self, + dialect: "BigQueryDialect", + schema: str, + catalog: Optional[str] = None, + routine_name: Optional[str] = None, + ): + """Initialize the routine list statement.""" + super().__init__(dialect, schema) + self.catalog = catalog + self.routine_name = routine_name + + @property + def format_method(self) -> str: + """The dialect formatting method that renders this expression.""" + return "format_bigquery_routine_list_query_statement" + + class BigQueryDatasetListQueryStatement(IntrospectionExpression): """The ``INFORMATION_SCHEMA.SCHEMATA`` question (datasets). diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/introspection/__init__.py b/src/rhosocial/activerecord/backend/impl/bigquery/introspection/__init__.py index 18525e1..592bf94 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/introspection/__init__.py +++ b/src/rhosocial/activerecord/backend/impl/bigquery/introspection/__init__.py @@ -14,6 +14,7 @@ from .types import ( BigQueryDatasetInfo, BigQueryPartitionInfo, + BigQueryRoutineInfo, BigQueryTableOptionInfo, BigQueryTableStorageInfo, ) @@ -26,6 +27,7 @@ "BigQueryIntrospector", "BigQueryDatasetInfo", "BigQueryPartitionInfo", + "BigQueryRoutineInfo", "BigQueryTableOptionInfo", "BigQueryTableStorageInfo", ] diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/introspection/introspector.py b/src/rhosocial/activerecord/backend/impl/bigquery/introspection/introspector.py index 72ea78a..33f4c23 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/introspection/introspector.py +++ b/src/rhosocial/activerecord/backend/impl/bigquery/introspection/introspector.py @@ -13,6 +13,7 @@ indexes none -- BigQuery has no index catalogue primary key ``TABLE_CONSTRAINTS`` + ``KEY_COLUMN_USAGE`` views ``INFORMATION_SCHEMA.VIEWS`` +routines ``INFORMATION_SCHEMA.ROUTINES`` datasets ``INFORMATION_SCHEMA.SCHEMATA`` table options ``INFORMATION_SCHEMA.TABLE_OPTIONS`` partitions ``INFORMATION_SCHEMA.PARTITIONS`` @@ -75,6 +76,7 @@ * ``TABLES`` https://cloud.google.com/bigquery/docs/information-schema-tables * ``COLUMNS`` https://cloud.google.com/bigquery/docs/information-schema-columns * ``VIEWS`` https://cloud.google.com/bigquery/docs/information-schema-views +* ``ROUTINES`` https://cloud.google.com/bigquery/docs/information-schema-routines * ``TABLE_CONSTRAINTS`` https://cloud.google.com/bigquery/docs/information-schema-table-constraints * ``KEY_COLUMN_USAGE`` @@ -134,6 +136,7 @@ BigQueryDatasetListQueryStatement, BigQueryIndexInfoQueryStatement, BigQueryPartitionListQueryStatement, + BigQueryRoutineListQueryStatement, BigQueryTableListQueryStatement, BigQueryTableOptionsQueryStatement, BigQueryViewInfoQueryStatement, @@ -142,6 +145,7 @@ from .types import ( BigQueryDatasetInfo, BigQueryPartitionInfo, + BigQueryRoutineInfo, BigQueryTableOptionInfo, BigQueryTableStorageInfo, ) @@ -488,6 +492,25 @@ def view_info_statement( self.dialect, view_name=view_name, schema=dataset, catalog=catalog ) + def routine_list_statement( + self, *, schema: Optional[str], routine_name: Optional[str] = None + ) -> "BigQueryRoutineListQueryStatement": + """Return the ``INFORMATION_SCHEMA.ROUTINES`` statement. + + With no name the statement lists the dataset's routines; with one it + filters the same question to that row, which is how + :meth:`get_routine_info` asks. + + https://cloud.google.com/bigquery/docs/information-schema-routines + """ + catalog, dataset = self._dataset_scoped(schema) + return BigQueryRoutineListQueryStatement( + self.dialect, + schema=dataset, + catalog=catalog, + routine_name=routine_name, + ) + def trigger_list_statement( self, *, schema: Optional[str], table_name: Optional[str] = None ) -> "NoReturn": @@ -858,6 +881,41 @@ def parse_triggers( del rows, schema return [] + def parse_routines( + self, rows: List[Dict[str, Any]], schema: Optional[str] + ) -> List[BigQueryRoutineInfo]: + """Parse ``ROUTINES`` rows into one record per routine. + + ``is_deterministic`` arrives as the catalog's ``YES``/``NO``/``NULL`` + and is read as ``True``/``False``/``None``; every other slot keeps + the catalog's own spelling. ``specific_schema`` and + ``routine_schema`` both name the dataset, so a row that carries + neither falls back to the schema argument the question was scoped to. + """ + label = self._schema_label(schema) + return [ + BigQueryRoutineInfo( + routine_name=_row_get(row, "routine_name") or "", + routine_type=_row_get(row, "routine_type") or "", + specific_catalog=_row_get(row, "specific_catalog"), + specific_schema=_row_get(row, "specific_schema") or label, + specific_name=_row_get(row, "specific_name"), + routine_catalog=_row_get(row, "routine_catalog"), + routine_schema=_row_get(row, "routine_schema") or label, + data_type=_row_get(row, "data_type"), + routine_body=_row_get(row, "routine_body"), + routine_definition=_row_get(row, "routine_definition"), + external_language=_row_get(row, "external_language"), + is_deterministic=_as_bool(_row_get(row, "is_deterministic")), + security_type=_row_get(row, "security_type"), + created=_row_get(row, "created"), + last_altered=_row_get(row, "last_altered"), + connection=_row_get(row, "connection"), + ddl=_row_get(row, "ddl"), + ) + for row in rows + ] + def parse_datasets(self, rows: List[Dict[str, Any]]) -> List[BigQueryDatasetInfo]: return [ BigQueryDatasetInfo( @@ -1102,6 +1160,55 @@ def list_datasets(self) -> List[BigQueryDatasetInfo]: self._cache.set(key, result) return result + def list_routines( + self, schema: Optional[str] = None + ) -> List[BigQueryRoutineInfo]: + """List a dataset's routines from ``INFORMATION_SCHEMA.ROUTINES``. + + One row per persistent user-defined function, aggregate function, + table function or stored procedure. The cache key carries the schema + the question was scoped to, so two datasets do not answer each other. + """ + key = self._cache.key( + IntrospectionScope.SCHEMA, + schema or self.default_schema, + extra="routines", + ) + cached = self._cache.get(key) + if cached is not None: + return cached + sql, params = self.routine_list_statement(schema=schema).to_sql() + result = self.parse_routines(self._executor.execute(sql, params), schema) + self._cache.set(key, result) + return result + + def get_routine_info( + self, routine_name: str, schema: Optional[str] = None + ) -> Optional[BigQueryRoutineInfo]: + """Return one named routine, or ``None`` when the dataset has none. + + The question is the same ``ROUTINES`` read as :meth:`list_routines`, + filtered by ``routine_name``. A missing name is not an error here: + the caller decides whether "no such routine" is a failure. + """ + key = self._cache.key( + IntrospectionScope.SCHEMA, + routine_name, + schema=schema, + extra="routine", + ) + cached = self._cache.get(key) + if cached is not None: + return cached + sql, params = self.routine_list_statement( + schema=schema, routine_name=routine_name + ).to_sql() + rows = self.parse_routines(self._executor.execute(sql, params), schema) + if not rows: + return None + self._cache.set(key, rows[0]) + return rows[0] + def list_table_options( self, table_name: str, schema: Optional[str] = None ) -> List[BigQueryTableOptionInfo]: @@ -1342,6 +1449,52 @@ async def list_datasets(self) -> List[BigQueryDatasetInfo]: self._cache.set(key, result) return result + async def list_routines( + self, schema: Optional[str] = None + ) -> List[BigQueryRoutineInfo]: + """List a dataset's routines (async). + + The async twin of :meth:`BigQueryIntrospector.list_routines`: same + statement, same parser and same cache key. + """ + key = self._cache.key( + IntrospectionScope.SCHEMA, + schema or self.default_schema, + extra="routines", + ) + cached = self._cache.get(key) + if cached is not None: + return cached + sql, params = self.routine_list_statement(schema=schema).to_sql() + result = self.parse_routines(await self._executor.execute(sql, params), schema) + self._cache.set(key, result) + return result + + async def get_routine_info( + self, routine_name: str, schema: Optional[str] = None + ) -> Optional[BigQueryRoutineInfo]: + """Return one named routine, or ``None`` (async). + + The async twin of :meth:`BigQueryIntrospector.get_routine_info`. + """ + key = self._cache.key( + IntrospectionScope.SCHEMA, + routine_name, + schema=schema, + extra="routine", + ) + cached = self._cache.get(key) + if cached is not None: + return cached + sql, params = self.routine_list_statement( + schema=schema, routine_name=routine_name + ).to_sql() + rows = self.parse_routines(await self._executor.execute(sql, params), schema) + if not rows: + return None + self._cache.set(key, rows[0]) + return rows[0] + async def list_table_options( self, table_name: str, schema: Optional[str] = None ) -> List[BigQueryTableOptionInfo]: diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/introspection/types.py b/src/rhosocial/activerecord/backend/impl/bigquery/introspection/types.py index ed18e68..4cddec5 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/introspection/types.py +++ b/src/rhosocial/activerecord/backend/impl/bigquery/introspection/types.py @@ -8,6 +8,9 @@ ``BigQueryDatasetInfo`` A dataset, from ``INFORMATION_SCHEMA.SCHEMATA``. +``BigQueryRoutineInfo`` + A persistent function or procedure, from + ``INFORMATION_SCHEMA.ROUTINES``. ``BigQueryTableOptionInfo`` One table option, from ``INFORMATION_SCHEMA.TABLE_OPTIONS``. ``BigQueryPartitionInfo`` @@ -26,6 +29,8 @@ * ``INFORMATION_SCHEMA.SCHEMATA`` https://cloud.google.com/bigquery/docs/information-schema-datasets-schemata +* ``INFORMATION_SCHEMA.ROUTINES`` + https://cloud.google.com/bigquery/docs/information-schema-routines * ``INFORMATION_SCHEMA.TABLE_OPTIONS`` https://cloud.google.com/bigquery/docs/information-schema-table-options * ``INFORMATION_SCHEMA.PARTITIONS`` @@ -62,6 +67,53 @@ class BigQueryDatasetInfo: sync_status: Optional[str] = None +@dataclass +class BigQueryRoutineInfo: + """One row of ``INFORMATION_SCHEMA.ROUTINES``. + + A routine is a persistent user-defined function or a stored procedure. + ``routine_type`` is the catalog's own word -- ``FUNCTION``, + ``AGGREGATE FUNCTION``, ``TABLE FUNCTION`` or ``PROCEDURE`` -- and + ``data_type`` is the declared return type, which is ``NULL`` for a + procedure. ``routine_body`` says whether the body is SQL (``SQL``) or + external (``EXTERNAL`` for a JavaScript UDF), and + ``routine_definition`` carries the body's text. + + Field names follow the catalog, lowercased, so each one can be traced + back to the documented column without a mapping table. The + ``specific_*`` and ``routine_*`` pairs both name the project and dataset + in the catalog's own way; they are kept because SQL-standard + ``SPECIFIC_*`` identity is what distinguishes overloaded routines. + + https://cloud.google.com/bigquery/docs/information-schema-routines + """ + + routine_name: str + routine_type: str + specific_catalog: Optional[str] = None + specific_schema: Optional[str] = None + specific_name: Optional[str] = None + routine_catalog: Optional[str] = None + routine_schema: Optional[str] = None + #: The declared return type; ``None`` for a stored procedure. + data_type: Optional[str] = None + #: ``SQL``, or ``EXTERNAL`` for a JavaScript UDF. + routine_body: Optional[str] = None + routine_definition: Optional[str] = None + #: ``JAVASCRIPT`` for a JavaScript UDF, ``None`` for a SQL routine. + external_language: Optional[str] = None + #: ``True``/``False`` from the catalog's ``YES``/``NO``; ``None`` if unknown. + is_deterministic: Optional[bool] = None + #: Documented as always ``NULL``. + security_type: Optional[str] = None + created: Optional[datetime] = None + last_altered: Optional[datetime] = None + #: The remote-function connection, when the routine has one. + connection: Optional[str] = None + #: The ``CREATE FUNCTION`` / ``CREATE PROCEDURE`` text the catalog reports. + ddl: Optional[str] = None + + @dataclass class BigQueryTableOptionInfo: """One row of ``INFORMATION_SCHEMA.TABLE_OPTIONS``. @@ -137,6 +189,7 @@ class BigQueryTableStorageInfo: __all__ = [ "BigQueryDatasetInfo", + "BigQueryRoutineInfo", "BigQueryTableOptionInfo", "BigQueryPartitionInfo", "BigQueryTableStorageInfo", diff --git a/src/rhosocial/activerecord/backend/impl/bigquery/mixins/introspection.py b/src/rhosocial/activerecord/backend/impl/bigquery/mixins/introspection.py index d82e688..866eeee 100644 --- a/src/rhosocial/activerecord/backend/impl/bigquery/mixins/introspection.py +++ b/src/rhosocial/activerecord/backend/impl/bigquery/mixins/introspection.py @@ -67,6 +67,7 @@ BigQueryColumnInfoQueryStatement, BigQueryDatabaseInfoQueryStatement, BigQueryIndexInfoQueryStatement, + BigQueryRoutineListQueryStatement, BigQueryTableListQueryStatement, BigQueryViewInfoQueryStatement, BigQueryViewListQueryStatement, @@ -96,6 +97,7 @@ BigQueryDatasetListQueryStatement, BigQueryIndexInfoQueryStatement, BigQueryPartitionListQueryStatement, + BigQueryRoutineListQueryStatement, BigQueryTableListQueryStatement, BigQueryTableOptionsQueryStatement, BigQueryViewInfoQueryStatement, @@ -156,6 +158,17 @@ def supports_dataset_introspection(self) -> bool: schema under it, so this is the only namespace switch.""" return True + def supports_routine_introspection(self) -> bool: + """BigQuery answers for its routines from ``INFORMATION_SCHEMA.ROUTINES``. + + A routine is a persistent user-defined function (scalar, aggregate or + table function) or a stored procedure. The view is dataset-scoped, + like ``VIEWS``, and documents one row per routine. + + https://cloud.google.com/bigquery/docs/information-schema-routines + """ + return True + # ------------------------------------------------------------------ # # Capability declaration # ------------------------------------------------------------------ # @@ -705,6 +718,60 @@ def format_bigquery_view_info_query_statement( ) return self.format_query_statement(query) + def format_bigquery_routine_list_query_statement( + self, expr: "BigQueryRoutineListQueryStatement" + ) -> Tuple[str, tuple]: + """Render the ``ROUTINES`` question. + + The projection is the documented column order. ``routine_type`` + distinguishes ``FUNCTION``, ``AGGREGATE FUNCTION``, ``TABLE + FUNCTION`` and ``PROCEDURE``; ``data_type`` is the return type + (``NULL`` for a procedure); ``routine_definition`` carries the SQL + body or the + JavaScript definition; and ``connection`` names the remote-function + connection when one backs the routine. + + A ``routine_name`` narrows the same question to one row, which is + how the ``get`` half of the family is asked. The name is a bind + parameter, like every other caller value in this dialect. + + https://cloud.google.com/bigquery/docs/information-schema-routines + """ + condition = None + if expr.routine_name is not None: + condition = ComparisonPredicate( + self, + ComparisonPredicate.coerce_operator("="), + Column(self, "routine_name"), + Literal(self, expr.routine_name), + ) + query = QueryStatement( + self, + select=self._columns( + "specific_catalog", + "specific_schema", + "specific_name", + "routine_catalog", + "routine_schema", + "routine_name", + "routine_type", + "data_type", + "routine_body", + "routine_definition", + "external_language", + "is_deterministic", + "security_type", + "created", + "last_altered", + "connection", + "ddl", + ), + from_=self._information_schema_source("ROUTINES", expr.catalog, expr.schema), + where=condition, + order_by=self._order_by_first("routine_name"), + ) + return self.format_query_statement(query) + def format_bigquery_dataset_list_query_statement( self, expr: "BigQueryDatasetListQueryStatement" ) -> Tuple[str, tuple]: diff --git a/tests/rhosocial/activerecord_bigquery_test/feature/backend/dialect/test_bigquery_kind_identity.py b/tests/rhosocial/activerecord_bigquery_test/feature/backend/dialect/test_bigquery_kind_identity.py index 0d7e279..bd2ae4f 100644 --- a/tests/rhosocial/activerecord_bigquery_test/feature/backend/dialect/test_bigquery_kind_identity.py +++ b/tests/rhosocial/activerecord_bigquery_test/feature/backend/dialect/test_bigquery_kind_identity.py @@ -65,11 +65,11 @@ def test_the_package_defines_the_materialized_view_and_introspection_statements( """A guard on the sweep below: an empty package would pass it silently. The set grew once, when the introspection statements arrived: the - four materialized-view statements and the ten questions this backend - asks -- the nine INFORMATION_SCHEMA ones and the data-profile family, - which reads the rows rather than a view. A class added without - updating this guard is caught here rather than passing the - parametrised tests silently. + four materialized-view statements and the eleven questions this + backend asks -- the ten INFORMATION_SCHEMA ones (including ROUTINES) + and the data-profile family, which reads the rows rather than a view. + A class added without updating this guard is caught here rather than + passing the parametrised tests silently. """ assert set(NODES) == { f"{BIGQUERY_EXPR_PKG}.materialized_view.BigQueryAlterMaterializedViewSetOptionsStatement", @@ -82,6 +82,7 @@ def test_the_package_defines_the_materialized_view_and_introspection_statements( f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryIndexInfoQueryStatement", f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryViewListQueryStatement", f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryViewInfoQueryStatement", + f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryRoutineListQueryStatement", f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryColumnDataProfileQueryStatement", f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryDatasetListQueryStatement", f"{BIGQUERY_EXPR_PKG}.introspection.BigQueryTableOptionsQueryStatement", diff --git a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_cli_introspect.py b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_cli_introspect.py index e960a0a..be0f767 100644 --- a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_cli_introspect.py +++ b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_cli_introspect.py @@ -34,6 +34,7 @@ from rhosocial.activerecord.backend.impl.bigquery.cli import introspect as cli_introspect from rhosocial.activerecord.backend.impl.bigquery.introspection.types import ( BigQueryPartitionInfo, + BigQueryRoutineInfo, ) @@ -130,6 +131,30 @@ def list_datasets(self): } ] + def list_routines(self, schema=None): + self.calls.append(("list_routines", schema)) + return [ + BigQueryRoutineInfo( + routine_name="add_one", + routine_type="FUNCTION", + routine_schema=schema, + data_type="INT64", + is_deterministic=True, + ) + ] + + def get_routine_info(self, routine_name, schema=None): + self.calls.append(("get_routine_info", routine_name, schema)) + if routine_name != "add_one": + return None + return BigQueryRoutineInfo( + routine_name="add_one", + routine_type="FUNCTION", + routine_schema=schema, + data_type="INT64", + is_deterministic=True, + ) + def get_database_info(self): self.calls.append(("get_database_info",)) return IntrospectionDatabaseInfo( @@ -214,6 +239,7 @@ def test_every_subcommand_is_registered(): assert set(COMMAND_NAMES) == { "tables", "views", + "routines", "columns", "datasets", "database", @@ -223,7 +249,6 @@ def test_every_subcommand_is_registered(): "runtime", "index-stats", "dependencies", - "ddl", "column-profiles", "value-samples", "join-coverage", @@ -261,6 +286,18 @@ def test_an_unknown_command_exits_non_zero(): assert exc_info.value.code == 2 +def test_ddl_is_not_an_advertised_command(): + """``ddl`` is not offered: its capability is permanently false here. + + ``supports_ddl_extraction()`` answers ``False`` and no extraction + statement exists, so argparse rejects the name as an invalid choice + (exit 2) rather than the CLI advertising it and refusing it later. + """ + with pytest.raises(SystemExit) as exc_info: + _run(["ddl", "orders"]) + assert exc_info.value.code == 2 + + # --------------------------------------------------------------------------- # The subcommands, in each format # --------------------------------------------------------------------------- @@ -348,6 +385,49 @@ def test_datasets_lists_datasets_as_json(built, capsys): assert built[0].introspector.calls == [("list_datasets",)] +def test_routines_lists_the_dataset_routines_as_json(built, capsys): + _run(["routines", "--dataset", "my_dataset", "-o", "json"]) + payload = json.loads(capsys.readouterr().out) + assert [row["routine_name"] for row in payload] == ["add_one"] + assert payload[0]["routine_type"] == "FUNCTION" + assert payload[0]["is_deterministic"] is True + assert built[0].introspector.calls == [("list_routines", "my_dataset")] + + +def test_a_named_routine_is_one_record_as_json(built, capsys): + _run(["routines", "add_one", "--dataset", "my_dataset", "-o", "json"]) + payload = json.loads(capsys.readouterr().out) + assert payload == [ + { + "routine_name": "add_one", + "routine_type": "FUNCTION", + "specific_catalog": None, + "specific_schema": None, + "specific_name": None, + "routine_catalog": None, + "routine_schema": "my_dataset", + "data_type": "INT64", + "routine_body": None, + "routine_definition": None, + "external_language": None, + "is_deterministic": True, + "security_type": None, + "created": None, + "last_altered": None, + "connection": None, + "ddl": None, + } + ] + assert built[0].introspector.calls == [("get_routine_info", "add_one", "my_dataset")] + + +def test_a_missing_routine_exits_non_zero(built, capsys): + with pytest.raises(SystemExit) as exc_info: + _run(["routines", "missing", "--dataset", "my_dataset", "-o", "json"]) + assert exc_info.value.code == 1 + assert "no routine named 'missing'" in capsys.readouterr().err + + def test_columns_lists_one_tables_columns_as_json(built, capsys): _run(["columns", "orders", "--dataset", "my_dataset", "-o", "json"]) payload = json.loads(capsys.readouterr().out) diff --git a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspect_families_cli.py b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspect_families_cli.py index f904359..9f9f711 100644 --- a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspect_families_cli.py +++ b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspect_families_cli.py @@ -1,5 +1,5 @@ # tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspect_families_cli.py -"""The nine families, as the BigQuery CLI asks for them. +"""The eight families, as the BigQuery CLI asks for them. No project and no credentials are needed here. What is checked is the wiring the CLI owns: that each type reaches its introspector method, that NAME and @@ -9,13 +9,14 @@ row. Whether the answer is right is the introspector's question, and the live tests carry it. -Six of the nine are gated, not answered: ``INFORMATION_SCHEMA`` describes +Five of the eight are gated, not answered: ``INFORMATION_SCHEMA`` describes objects rather than a statistics collector, so ``unused-indexes``, -``table-stats``, ``runtime``, ``index-stats``, ``dependencies`` and ``ddl`` -are refused at the capability question. Their branches are still asserted -here, by holding the gate open, because "the CLI can ask this and is told no" -and "the CLI cannot ask this" are different contracts and only one of them is -true. +``table-stats``, ``runtime``, ``index-stats`` and ``dependencies`` are refused +at the capability question. Their branches are still asserted here, by holding +the gate open, because "the CLI can ask this and is told no" and "the CLI +cannot ask this" are different contracts and only one of them is true. ``ddl`` +is not offered at all -- ``supports_ddl_extraction()`` is False and no +extraction statement exists -- so it is absent from the advertised surface. Three answer, and the profile speaks for the one cost a reader has to know: BigQuery bills bytes read, so a profile is a full scan. That sentence belongs @@ -44,12 +45,14 @@ from rhosocial.activerecord.backend.introspection.types import ( ColumnDataProfile, ColumnValueFrequency, - DDLStatement, JoinCoverage, TableType, ) +from rhosocial.activerecord.backend.impl.bigquery.introspection.types import ( + BigQueryRoutineInfo, +) -#: The nine families this change wires, and the switch that gates each. +#: The eight families this change wires, and the switch that gates each. #: The dialect declares three of them; the rest it declines, and the refusal #: is the answer this backend gives rather than an empty row. NEW_FAMILIES = { @@ -58,7 +61,6 @@ "runtime": "supports_runtime_stats", "index-stats": "supports_index_stats", "dependencies": "supports_object_dependencies", - "ddl": "supports_ddl_extraction", "column-profiles": "supports_column_data_profile", "value-samples": "supports_column_value_sample", "join-coverage": "supports_join_coverage", @@ -79,10 +81,10 @@ "join-coverage", ) -#: The two questions the dialect answers from the catalogue, beyond the -#: listings: the project-level database question and this engine's own -#: partitions view. Neither is declined, so neither is a refusal. -CATALOGUE_FAMILIES = ("database", "partitions") +#: The three questions the dialect answers from the catalogue, beyond the +#: listings: the project-level database question, the routines family and +#: this engine's own partitions view. None is declined, so none is a refusal. +CATALOGUE_FAMILIES = ("database", "routines", "partitions") class _RecordingIntrospector: @@ -246,6 +248,7 @@ def test_the_table_says_what_each_commands_name_means(self): assert introspect_cli._NAME_RULE == { "tables": "forbidden", "views": "forbidden", + "routines": "optional", "columns": "required", "datasets": "forbidden", "database": "forbidden", @@ -255,7 +258,6 @@ def test_the_table_says_what_each_commands_name_means(self): "runtime": "forbidden", "index-stats": "forbidden", "dependencies": "optional", - "ddl": "required", "column-profiles": "required", "value-samples": "required", "join-coverage": "required", @@ -360,52 +362,6 @@ def test_the_list_is_the_four_formats_the_factory_answers(self): assert output_mod.OUTPUT_CHOICES == ["table", "json", "csv", "tsv"] -class TestTheDdlBranch: - """The schema object, not a name, is what a caller parses.""" - - def _statement(self) -> DDLStatement: - return DDLStatement( - schema=None, - object_type="TABLE", - object_name="orders", - ddl="CREATE TABLE orders (id INT64)", - native=True, - ) - - def test_the_table_is_asked_for_by_name(self, monkeypatch): - introspector = _RecordingIntrospector({"get_ddl_statement": self._statement()}) - - _run_sync(monkeypatch, introspector, _args("ddl", "orders"), gate_open=True) - - assert introspector.calls == [ - ("get_ddl_statement", ("orders",), {"object_type": None, "schema": "my_dataset"}) - ] - - def test_a_missing_name_is_an_incomplete_invocation(self, monkeypatch, capsys): - """A missing required NAME exits 1, like a missing ``--dataset``. - - ``ddl`` is refused by this backend's gate, so the gate is held open - to reach the branch's own check: the exit code is the difference - between "you did not finish the invocation" (1) and "this project - will not answer that" (2), and only one of them is a defect in the - request. - """ - with pytest.raises(SystemExit) as excinfo: - _run_sync(monkeypatch, _RecordingIntrospector({}), _args("ddl"), gate_open=True) - - assert excinfo.value.code == 1 - assert "required" in capsys.readouterr().err - - def test_no_ddl_is_an_error_not_an_empty_listing(self, monkeypatch, capsys): - introspector = _RecordingIntrospector({"get_ddl_statement": None}) - - with pytest.raises(SystemExit) as excinfo: - _run_sync(monkeypatch, introspector, _args("ddl", "orders"), gate_open=True) - - assert excinfo.value.code == 1 - assert "no DDL available" in capsys.readouterr().err - - class TestTheColumnProfilesBranch: """One scan, billed, and the cost said before the rows.""" @@ -672,6 +628,66 @@ def test_a_name_is_refused(self, capsys): assert "does not take an object name" in capsys.readouterr().err +class TestTheRoutinesBranch: + """A listing, and the same read narrowed to one named routine.""" + + def _routine(self) -> BigQueryRoutineInfo: + return BigQueryRoutineInfo( + routine_name="add_one", + routine_type="FUNCTION", + specific_schema="my_dataset", + routine_schema="my_dataset", + data_type="INT64", + routine_body="SQL", + routine_definition="x + 1", + is_deterministic=True, + ) + + def test_an_unnamed_run_lists_the_dataset_routines(self, monkeypatch): + introspector = _RecordingIntrospector({"list_routines": [self._routine()]}) + + _run_sync(monkeypatch, introspector, _args("routines")) + + assert introspector.calls == [("list_routines", (), {"schema": "my_dataset"})] + + def test_a_named_run_asks_for_one_routine(self, monkeypatch): + introspector = _RecordingIntrospector({"get_routine_info": self._routine()}) + + _run_sync(monkeypatch, introspector, _args("routines", "add_one")) + + assert introspector.calls == [ + ("get_routine_info", ("add_one",), {"schema": "my_dataset"}) + ] + + def test_a_missing_routine_is_an_error_not_a_blank_listing(self, monkeypatch, capsys): + introspector = _RecordingIntrospector({"get_routine_info": None}) + + with pytest.raises(SystemExit) as excinfo: + _run_sync(monkeypatch, introspector, _args("routines", "missing")) + + assert excinfo.value.code == 1 + assert "no routine named 'missing'" in capsys.readouterr().err + + def test_the_named_answer_is_one_record(self, monkeypatch, capsys): + introspector = _RecordingIntrospector({"get_routine_info": self._routine()}) + + _run_sync(monkeypatch, introspector, _args("routines", "add_one")) + + payload = json.loads(capsys.readouterr().out) + assert payload[0]["routine_name"] == "add_one" + assert payload[0]["routine_type"] == "FUNCTION" + assert payload[0]["is_deterministic"] is True + + def test_the_family_is_gated_by_its_own_probe(self): + assert ( + introspect_cli._TYPE_CAPABILITIES["routines"] + == "supports_routine_introspection" + ) + assert BigQueryDialect.supports_routine_introspection( + BigQueryDialect() + ) is True + + class TestThePartitionsBranch: """One table, and one row per partition it holds.""" @@ -716,7 +732,7 @@ def test_both_families_that_read_a_column_may_be_given_one(self): introspect_cli._refuse_unused_arguments(_args(command, column="status")) def test_a_required_name_is_required_for_the_new_families(self, monkeypatch, capsys): - for command in ("ddl", "column-profiles", "value-samples"): + for command in ("column-profiles", "value-samples"): with pytest.raises(SystemExit) as excinfo: _run_sync( monkeypatch, @@ -774,7 +790,6 @@ def test_the_declined_families_are_refused_by_this_dialect(self, capsys): "runtime", "index-stats", "dependencies", - "ddl", ): with pytest.raises(SystemExit) as excinfo: introspect_cli._require_capability( @@ -794,18 +809,6 @@ class TestTheAsyncPath: ("table-stats", "list_table_usage_stats", {}, [], ((), {"schema": "my_dataset"})), ("runtime", "list_runtime_metrics", {}, [], ((), {"schema": "my_dataset"})), ("index-stats", "list_index_usage_stats", {}, [], ((), {"schema": "my_dataset"})), - ( - "ddl", - "get_ddl_statement", - {"name": "orders"}, - DDLStatement( - object_type="TABLE", - object_name="orders", - ddl="CREATE TABLE orders (id INT64)", - native=True, - ), - (("orders",), {"object_type": None, "schema": "my_dataset"}), - ), ( "column-profiles", "list_column_data_profiles", @@ -860,15 +863,29 @@ class TestTheAsyncPath: [], (("orders",), {"schema": "my_dataset"}), ), + ( + "routines", + "list_routines", + {}, + [], + ((), {"schema": "my_dataset"}), + ), + ( + "routines", + "get_routine_info", + {"name": "add_one"}, + BigQueryRoutineInfo(routine_name="add_one", routine_type="FUNCTION"), + (("add_one",), {"schema": "my_dataset"}), + ), ], ) def test_the_async_branch_calls_the_same_method_with_the_same_arguments( self, command, method, args_kwargs, result, expected, monkeypatch ): - # ``result`` is not ``[]`` everywhere: ``ddl`` answers ``None`` for - # "no statement", and a stub that returned an empty list there would - # send the branch down the display path instead of the call it is - # being asked to reach. + # ``result`` is not ``[]`` everywhere: the row families answer a + # record or ``None`` for "no measurement", and a stub that returned + # an empty list there would send the branch down the display path + # instead of the call it is being asked to reach. introspector = _AsyncRecordingIntrospector({method: result}) _run_async( @@ -896,9 +913,9 @@ def test_the_async_profile_says_the_scan_is_billed(self, monkeypatch, capsys): class TestTheTypeSurfaceExtension: def test_the_advertised_surface_is_fifteen_types(self): - """The list the CLI's README states: the four listings, the project - and the partitions questions, the six declined families and the three - that read the rows.""" + """The list the CLI's README states: the five listings, the project, + routines and partitions questions, the five declined families and the + three that read the rows.""" assert len(introspect_cli.INTROSPECT_TYPES) == 15 for name in CATALOGUE_FAMILIES: assert name in introspect_cli.INTROSPECT_TYPES diff --git a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_parsing.py b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_parsing.py index 1b493fa..c21b9d7 100644 --- a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_parsing.py +++ b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_parsing.py @@ -525,6 +525,158 @@ def test_partitions_parse_counts_and_tiers(introspector): assert partitions[0].last_modified_time == modified +def test_routines_parse_the_documented_fields(introspector): + created = datetime(2026, 10, 24, 12, 0, 0) + routines = introspector.parse_routines( + [ + { + "specific_catalog": "my-project", + "specific_schema": "my_dataset", + "specific_name": "add_one", + "routine_catalog": "my-project", + "routine_schema": "my_dataset", + "routine_name": "add_one", + "routine_type": "FUNCTION", + "data_type": "INT64", + "routine_body": "SQL", + "routine_definition": "x + 1", + "external_language": None, + "is_deterministic": "YES", + "security_type": None, + "created": created, + "last_altered": created, + "connection": None, + "ddl": "CREATE FUNCTION myproject.mydataset.add_one(x INT64) AS (x + 1)", + }, + { + "routine_name": "load_orders", + "routine_type": "PROCEDURE", + "data_type": None, + "routine_body": "SQL", + "routine_definition": "INSERT INTO orders SELECT * FROM staging", + "is_deterministic": "NO", + }, + { + "routine_name": "js_regexp", + "routine_type": "FUNCTION", + "data_type": "STRING", + "routine_body": "EXTERNAL", + "routine_definition": "return x.match(/a/);", + "external_language": "JAVASCRIPT", + "is_deterministic": None, + "connection": "us.my-connection", + }, + ], + "my_dataset", + ) + add_one, load_orders, js_regexp = routines + assert add_one.routine_name == "add_one" + assert add_one.routine_type == "FUNCTION" + assert add_one.data_type == "INT64" + assert add_one.routine_body == "SQL" + assert add_one.routine_definition == "x + 1" + assert add_one.is_deterministic is True + assert add_one.created == created + assert add_one.ddl.startswith("CREATE FUNCTION") + assert add_one.specific_schema == "my_dataset" + assert load_orders.routine_type == "PROCEDURE" + assert load_orders.data_type is None + assert load_orders.is_deterministic is False + # A row carrying no schema slots falls back to the question's dataset. + assert load_orders.routine_schema == "my_dataset" + assert js_regexp.routine_body == "EXTERNAL" + assert js_regexp.external_language == "JAVASCRIPT" + assert js_regexp.is_deterministic is None + assert js_regexp.connection == "us.my-connection" + + +def test_routines_accept_uppercase_row_keys(introspector): + """A driver or fixture that preserves the catalog's own spelling.""" + routines = introspector.parse_routines( + [{"ROUTINE_NAME": "add_one", "ROUTINE_TYPE": "FUNCTION", "IS_DETERMINISTIC": "YES"}], + "my_dataset", + ) + assert routines[0].routine_name == "add_one" + assert routines[0].routine_type == "FUNCTION" + assert routines[0].is_deterministic is True + + +class _CannedRoutineExecutor: + """Answers canned routine rows and records the question.""" + + def __init__(self, rows): + self.rows = rows + self.calls = [] + + def execute(self, sql, params=()): + self.calls.append((sql, params)) + return self.rows + + +class _CannedRoutineAsyncExecutor(_CannedRoutineExecutor): + async def execute(self, sql, params=()): + self.calls.append((sql, params)) + return self.rows + + +ROUTINE_ROWS = [ + { + "routine_name": "add_one", + "routine_type": "FUNCTION", + "specific_schema": "my_dataset", + "routine_schema": "my_dataset", + "data_type": "INT64", + "is_deterministic": "YES", + } +] + + +def test_list_routines_asks_the_routines_view_once_and_caches_it(): + executor = _CannedRoutineExecutor(ROUTINE_ROWS) + introspector_obj = BigQueryIntrospector(_BackendStandIn(), executor) + + routines = introspector_obj.list_routines(schema="my_dataset") + assert [r.routine_name for r in routines] == ["add_one"] + assert len(executor.calls) == 1 + assert "INFORMATION_SCHEMA.ROUTINES" in executor.calls[0][0] + assert executor.calls[0][1] == () + + # The second read is the cached answer, not a second question. + assert introspector_obj.list_routines(schema="my_dataset") is routines + assert len(executor.calls) == 1 + + +def test_get_routine_info_filters_by_name_and_returns_none_when_absent(): + executor = _CannedRoutineExecutor(ROUTINE_ROWS) + introspector_obj = BigQueryIntrospector(_BackendStandIn(), executor) + + info = introspector_obj.get_routine_info("add_one", schema="my_dataset") + assert info.routine_name == "add_one" + sql, params = executor.calls[0] + assert "INFORMATION_SCHEMA.ROUTINES" in sql + assert "WHERE `routine_name` = ?" in sql + assert params == ("add_one",) + + executor.rows = [] + executor.calls = [] + introspector_obj2 = BigQueryIntrospector(_BackendStandIn(), executor) + assert introspector_obj2.get_routine_info("missing", schema="my_dataset") is None + assert executor.calls[0][1] == ("missing",) + + +async def test_list_routines_and_get_routine_info_async(): + executor = _CannedRoutineAsyncExecutor(ROUTINE_ROWS) + introspector_obj = AsyncBigQueryIntrospector(_BackendStandIn(), executor) + + routines = await introspector_obj.list_routines(schema="my_dataset") + assert [r.routine_name for r in routines] == ["add_one"] + assert "INFORMATION_SCHEMA.ROUTINES" in executor.calls[0][0] + + info = await introspector_obj.get_routine_info("add_one", schema="my_dataset") + assert info.routine_name == "add_one" + assert executor.calls[1][1] == ("add_one",) + + def test_database_info_reports_the_project_and_the_backend_level(introspector): info = introspector.parse_database_info([{"catalog_name": "my-project"}]) assert info.name == "my-project" diff --git a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_sql.py b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_sql.py index 2a3ee0d..f6662f4 100644 --- a/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_sql.py +++ b/tests/rhosocial/activerecord_bigquery_test/feature/backend/introspection/test_introspection_sql.py @@ -48,6 +48,7 @@ BigQueryDatasetListQueryStatement, BigQueryIndexInfoQueryStatement, BigQueryPartitionListQueryStatement, + BigQueryRoutineListQueryStatement, BigQueryTableListQueryStatement, BigQueryTableOptionsQueryStatement, BigQueryViewInfoQueryStatement, @@ -253,6 +254,46 @@ def test_single_view_query_binds_the_view_name(dialect): assert params == ("my_view",) +def test_routines_query_lists_the_documented_projection(dialect): + sql, params = BigQueryRoutineListQueryStatement( + dialect, schema="my_dataset", catalog="my-project" + ).to_sql() + assert "FROM `my-project`.`my_dataset`.INFORMATION_SCHEMA.ROUTINES" in sql + for column in ( + "specific_catalog", + "specific_schema", + "specific_name", + "routine_catalog", + "routine_schema", + "routine_name", + "routine_type", + "data_type", + "routine_body", + "routine_definition", + "external_language", + "is_deterministic", + "security_type", + "created", + "last_altered", + "connection", + "ddl", + ): + assert f"`{column}`" in sql + assert "WHERE" not in sql + assert sql.rstrip().endswith("ORDER BY `routine_name`") + assert params == () + + +def test_routine_query_binds_the_name_when_one_is_asked_for(dialect): + sql, params = BigQueryRoutineListQueryStatement( + dialect, schema="my_dataset", catalog="my-project", routine_name="add_one" + ).to_sql() + assert "FROM `my-project`.`my_dataset`.INFORMATION_SCHEMA.ROUTINES" in sql + assert "WHERE `routine_name` = ?" in sql + assert "ORDER BY `routine_name`" in sql + assert params == ("add_one",) + + def test_dataset_query_renders_the_project_and_region_qualifiers(dialect): sql, params = BigQueryDatasetListQueryStatement( dialect, catalog="my-project", location="US" @@ -362,6 +403,21 @@ def test_every_builder_binds_caller_data_only(introspector): table_name="orders", schema="my_dataset" ).to_sql() assert params == ("orders",) + _sql, params = introspector.routine_list_statement( + schema="my_dataset", routine_name="add_one" + ).to_sql() + assert params == ("add_one",) + + +def test_the_routine_builder_uses_the_configured_project_and_dataset(introspector): + introspector, _executor = introspector + sql, params = introspector.routine_list_statement(schema="my_dataset").to_sql() + assert "FROM `my-project`.`my_dataset`.INFORMATION_SCHEMA.ROUTINES" in sql + assert params == () + + +def test_the_routine_probe_declares_the_family(dialect): + assert dialect.supports_routine_introspection() is True def test_the_foreign_key_builder_refuses(introspector):