Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:

- name: Install activerecord and testsuite
run: |
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@release/v1.0.0.dev30" # NOSONAR
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@feature/partition-maintenance" # NOSONAR
pip install "rhosocial-activerecord-testsuite @ git+https://github.com/rhosocial/python-activerecord-testsuite.git@release/v1.0.0.dev17"

- name: Install dependencies
Expand Down Expand Up @@ -278,7 +278,7 @@ jobs:

- name: Install activerecord and testsuite
run: |
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@release/v1.0.0.dev30" # NOSONAR
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@feature/partition-maintenance" # NOSONAR
pip install "rhosocial-activerecord-testsuite @ git+https://github.com/rhosocial/python-activerecord-testsuite.git@release/v1.0.0.dev17"

- name: Install dependencies
Expand Down Expand Up @@ -385,7 +385,7 @@ jobs:

- name: Install activerecord and testsuite
run: |
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@release/v1.0.0.dev30" # NOSONAR
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@feature/partition-maintenance" # NOSONAR
pip install "rhosocial-activerecord-testsuite @ git+https://github.com/rhosocial/python-activerecord-testsuite.git@release/v1.0.0.dev17"

- name: Install dependencies
Expand Down Expand Up @@ -488,7 +488,7 @@ jobs:

- name: Install activerecord and testsuite
run: |
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@release/v1.0.0.dev30" # NOSONAR
pip install "rhosocial-activerecord @ git+https://github.com/rhosocial/python-activerecord.git@feature/partition-maintenance" # NOSONAR
pip install "rhosocial-activerecord-testsuite @ git+https://github.com/rhosocial/python-activerecord-testsuite.git@release/v1.0.0.dev17"

- name: Install dependencies
Expand Down
1 change: 1 addition & 0 deletions changelog.d/67.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added PostgreSQL partition planning and maintenance helpers: idempotent nsure, recursive list, alidate, drop_before and explicit drop_branch, with sync and async executors.
93 changes: 91 additions & 2 deletions docs/en_US/postgres_specific_features/partition.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,98 @@ from rhosocial.activerecord.backend.impl.postgres.expression.ddl.partition impor

partition = PostgresCreatePartitionStatement(
dialect, parent_table="orders", partition_name="orders_2024_q1",
partition_type="RANGE", bounds="FROM ('2024-01-01') TO ('2024-04-01')",
partition_type="RANGE",
partition_values={"from": "2024-01-01", "to": "2024-04-01"},
)
```

## Nested Partitions

PostgreSQL lets a partition be partitioned again, and it lets each branch of a
HASH parent pick its own child strategy. Pass a `PartitionClause` to
`PostgresCreatePartitionStatement`; the rendered order is `FOR VALUES` then
`PARTITION BY`.

```python
from rhosocial.activerecord.backend.expression import Column
from rhosocial.activerecord.backend.expression.statements import (
PartitionClause, PartitionStrategy,
)

# CREATE TABLE logs_p0 PARTITION OF logs
# FOR VALUES WITH (MODULUS 4, REMAINDER 0)
# PARTITION BY RANGE (created_at)
branch = PostgresCreatePartitionStatement(
dialect, parent_table="logs", partition_name="logs_p0",
partition_type="HASH", partition_values={"modulus": 4, "remainder": 0},
partition_clause=PartitionClause(
dialect, PartitionStrategy.RANGE, [Column(dialect, "created_at")],
),
)

# CREATE TABLE logs_p0_2026_01 PARTITION OF logs_p0
# FOR VALUES FROM ('2026-01-01 00:00:00+00') TO ('2026-02-01 00:00:00+00')
leaf = PostgresCreatePartitionStatement(
dialect, parent_table="logs_p0", partition_name="logs_p0_2026_01",
partition_type="RANGE",
partition_values={"from": "2026-01-01 00:00:00+00",
"to": "2026-02-01 00:00:00+00"},
)
```

A parent primary key must contain every level's partition key. With a HASH
parent on `tenant_bucket` and RANGE children on `created_at`, the parent key is
`(id, tenant_bucket, created_at)`; PostgreSQL rejects a child that is itself
partitioned when the inherited key omits `created_at`.

## Partition Maintenance

`PostgresPartitionPlan` describes the tree without naming individual
partitions; `PostgresPartitionMaintenance` (and its async twin) reconciles it
with the catalog. The module is owner-agnostic: table names and retention
decisions stay with the application.

```python
from datetime import datetime, timezone
from rhosocial.activerecord.backend.impl.postgres.partition import (
HashBranch, PostgresPartitionMaintenance, PostgresPartitionPlan, RangePolicy,
)

start = datetime(2026, 1, 1, tzinfo=timezone.utc)
plan = PostgresPartitionPlan(
parent="logs", method="HASH", columns=("tenant_bucket",), modulus=4,
branches=(
HashBranch(0, RangePolicy(columns=("created_at",), start=start,
interval="month", count=12)),
HashBranch(1, RangePolicy(columns=("created_at",), start=start,
interval="quarter", count=4)),
HashBranch(2, RangePolicy(columns=("created_at",), start=start,
interval="year", count=2)),
HashBranch(3, RangePolicy(columns=("created_at",),
bounds=((start, start.replace(day=16)),))),
),
)

maintenance = PostgresPartitionMaintenance(backend)
report = maintenance.ensure(plan) # idempotent gap filling
report.missing # (parent, child) pairs
maintenance.list(plan).orphans # catalog-only partitions
maintenance.validate(plan).issues # gaps / overlaps / mismatches
maintenance.drop_before(plan, cutoff, dry_run=True)# per-branch retention
```

* `ensure` creates only missing partitions. PostgreSQL DDL is transactional, so
each unit (one branch with its leaves, or all leaves of a flat plan) commits
independently.
* `list` reads `pg_inherits`/`pg_class` recursively and keeps the server's
bound text verbatim.
* `drop_before` only drops true leaves; a branch that is itself partitioned is
never cascaded. It defaults to `dry_run=True`.
* `drop_branch(plan, plan.branches[0])` is the explicit whole-branch removal.
* A flat plan (`policy=RangePolicy(...)` with no `branches`) produces the
single-level shape used by monthly log tables.


## Partition Management

PostgreSQL does not support `ATTACH PARTITION CONCURRENTLY`. PostgreSQL 14+ supports
Expand All @@ -36,7 +124,8 @@ from rhosocial.activerecord.backend.impl.postgres.expression.ddl.partition impor
# ATTACH PARTITION
attach = PostgresAttachPartitionStatement(
dialect, parent_table="orders", partition_name="orders_new",
partition_type="RANGE", bounds="FROM ('2024-07-01') TO ('2024-10-01')",
partition_type="RANGE",
partition_values={"from": "2024-07-01", "to": "2024-10-01"},
)

# DETACH PARTITION CONCURRENTLY (PG 14+)
Expand Down
87 changes: 85 additions & 2 deletions docs/zh_CN/postgres_specific_features/partition.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,93 @@ partition = PostgresCreatePartitionStatement(
parent_table="orders",
partition_name="orders_2024_q1",
partition_type="RANGE",
bounds="FROM ('2024-01-01') TO ('2024-04-01')",
partition_values={"from": "2024-01-01", "to": "2024-04-01"},
)
# sql: 'CREATE TABLE "orders_2024_q1" PARTITION OF "orders" FOR VALUES FROM ('2024-01-01') TO ('2024-04-01')'
```

## 多级(子)分区

PostgreSQL 允许一个分区继续分区,且 HASH 父表的每个一级分支可以各自选择
二级策略。把 `PartitionClause` 传给 `PostgresCreatePartitionStatement`,
渲染顺序为 `FOR VALUES` → `PARTITION BY`。

```python
from rhosocial.activerecord.backend.expression import Column
from rhosocial.activerecord.backend.expression.statements import (
PartitionClause, PartitionStrategy,
)

# CREATE TABLE logs_p0 PARTITION OF logs
# FOR VALUES WITH (MODULUS 4, REMAINDER 0)
# PARTITION BY RANGE (created_at)
branch = PostgresCreatePartitionStatement(
dialect, parent_table="logs", partition_name="logs_p0",
partition_type="HASH", partition_values={"modulus": 4, "remainder": 0},
partition_clause=PartitionClause(
dialect, PartitionStrategy.RANGE, [Column(dialect, "created_at")],
),
)

# CREATE TABLE logs_p0_2026_01 PARTITION OF logs_p0
# FOR VALUES FROM ('2026-01-01 00:00:00+00') TO ('2026-02-01 00:00:00+00')
leaf = PostgresCreatePartitionStatement(
dialect, parent_table="logs_p0", partition_name="logs_p0_2026_01",
partition_type="RANGE",
partition_values={"from": "2026-01-01 00:00:00+00",
"to": "2026-02-01 00:00:00+00"},
)
```

父表主键必须包含**每一级**的分区键。若一级按 `tenant_bucket` HASH、二级按
`created_at` RANGE,父表主键应为 `(id, tenant_bucket, created_at)`;当继承的
唯一键缺少 `created_at` 时,PostgreSQL 会拒绝“本身又分区”的一级子表。

## 分区维护

`PostgresPartitionPlan` 只描述树(父表、分区键、分支策略、命名),不写死具体
分区名;`PostgresPartitionMaintenance`(及其异步版本)把它与目录状态对齐。
模块不绑定任何业务表名或保留策略,这些仍由应用决定。

```python
from datetime import datetime, timezone
from rhosocial.activerecord.backend.impl.postgres.partition import (
HashBranch, PostgresPartitionMaintenance, PostgresPartitionPlan, RangePolicy,
)

start = datetime(2026, 1, 1, tzinfo=timezone.utc)
plan = PostgresPartitionPlan(
parent="logs", method="HASH", columns=("tenant_bucket",), modulus=4,
branches=(
HashBranch(0, RangePolicy(columns=("created_at",), start=start,
interval="month", count=12)),
HashBranch(1, RangePolicy(columns=("created_at",), start=start,
interval="quarter", count=4)),
HashBranch(2, RangePolicy(columns=("created_at",), start=start,
interval="year", count=2)),
HashBranch(3, RangePolicy(columns=("created_at",),
bounds=((start, start.replace(day=16)),))),
),
)

maintenance = PostgresPartitionMaintenance(backend)
report = maintenance.ensure(plan) # 幂等补建缺失分区
report.missing # (父表, 缺失分区) 列表
maintenance.list(plan).orphans # 计划外的分区
maintenance.validate(plan).issues # 缺口 / 重叠 / 策略不符
maintenance.drop_before(plan, cutoff, dry_run=True) # 按分支策略回收
```

* `ensure` 只补建缺失项;PostgreSQL 的 DDL 是事务性的,每个单元(一个一级
分支及其叶子,或单层计划的全部叶子)独立提交。
* `list` 递归读取 `pg_inherits` / `pg_class`,并原样保留服务器给出的 bound 文本。
* `drop_before` 只删除真正的叶子;自身已分区的一级分支不会被级联删除,默认
`dry_run=True`。
* `drop_branch(plan, plan.branches[0])` 是删除整个一级分支的显式操作。
* 单层计划(只给 `policy=RangePolicy(...)`、不给 `branches`)生成按月日志表
所需的扁平形态。


## 分区管理

PostgreSQL 不支持 `ATTACH PARTITION CONCURRENTLY`。PostgreSQL 14+ 支持
Expand All @@ -43,7 +125,8 @@ from rhosocial.activerecord.backend.impl.postgres.expression.ddl.partition impor
# 附加分区
attach = PostgresAttachPartitionStatement(
dialect, parent_table="orders", partition_name="orders_new",
partition_type="RANGE", bounds="FROM ('2024-07-01') TO ('2024-10-01')",
partition_type="RANGE",
partition_values={"from": "2024-07-01", "to": "2024-10-01"},
)
# sql: 'ALTER TABLE "orders" ATTACH PARTITION "orders_new" FOR VALUES FROM ('2024-07-01') TO ('2024-10-01')'

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
Table partitioning - PostgreSQL 10+.

This example demonstrates:
1. CREATE TABLE ... PARTITION BY RANGE
2. Creating individual partitions
3. Inserting data into partitioned table
1. CREATE TABLE ... PARTITION BY RANGE with expression nodes
2. Creating individual partitions (CREATE TABLE ... PARTITION OF)
3. Inserting data into a partitioned table
4. Querying partitioned tables
"""

Expand All @@ -14,17 +14,25 @@
import os
from rhosocial.activerecord.backend.impl.postgres.backend import PostgresBackend
from rhosocial.activerecord.backend.impl.postgres.config import PostgresConnectionConfig
from rhosocial.activerecord.backend.impl.postgres.expression.ddl.partition import (
from rhosocial.activerecord.backend.impl.postgres.expression import (
PostgresCreatePartitionStatement,
PostgresSerialType,
)
from rhosocial.activerecord.backend.expression import (
ColumnDefinition,
ColumnConstraint,
ColumnNotNullConstraint,
CreateTableStatement,
DropTableStatement,
InsertStatement,
PartitionClause,
PartitionStrategy,
ValuesSource,
QueryStatement,
)
from rhosocial.activerecord.backend.expression.objects import Table
from rhosocial.activerecord.backend.expression.core import Literal, AllColumnsWildcard, Column
from rhosocial.activerecord.backend.expression.types import DateType, DecimalType, VarCharType
from rhosocial.activerecord.backend.options import ExecutionOptions
from rhosocial.activerecord.backend.schema import StatementType

Expand Down Expand Up @@ -55,17 +63,37 @@
# SECTION: Business Logic (the pattern to learn)
# ============================================================

# 1. Create partitioned parent table using raw SQL
# (Expression API does not yet support PARTITION BY syntax)
parent_sql = """
CREATE TABLE orders (
id SERIAL,
order_date DATE NOT NULL,
customer_name VARCHAR(100),
amount NUMERIC(10,2)
) PARTITION BY RANGE (order_date)
"""
backend.execute(parent_sql)
# 1. Create the partitioned parent table with the expression API.
# PARTITION BY is a clause node on CreateTableStatement; the SQL is the
# dialect's, not the caller's.
parent_expr = CreateTableStatement(
dialect=dialect,
table=Table(dialect, 'orders'),
columns=[
ColumnDefinition(
dialect, 'id', PostgresSerialType(dialect=dialect),
),
ColumnDefinition(
dialect, 'order_date', DateType(dialect=dialect),
constraints=[
ColumnConstraint(dialect, branch=ColumnNotNullConstraint(dialect)),
],
),
ColumnDefinition(
dialect, 'customer_name', VarCharType(length=100, dialect=dialect),
),
ColumnDefinition(
dialect, 'amount', DecimalType(precision=10, scale=2, dialect=dialect),
),
],
partition=PartitionClause(
dialect=dialect,
method=PartitionStrategy.RANGE,
keys=[Column(dialect, 'order_date')],
),
)
sql, params = parent_expr.to_sql()
backend.execute(sql, params)
print("Created partitioned table: orders")

# 2. Create individual partitions using raw SQL
Expand Down Expand Up @@ -149,8 +177,13 @@
# Key points:
# 1. Requires PostgreSQL 10+ for declarative partitioning
# 2. PARTITION BY RANGE is the most common strategy
# 3. Other strategies: PARTITION BY LIST, PARTITION BY HASH
# 3. Other strategies: PARTITION BY LIST, PARTITION BY HASH (PG 11+)
# 4. PostgreSQL automatically routes INSERT to correct partition
# 5. Queries on parent table are transparent across all partitions
# 6. Individual partitions can be queried directly for performance
# 7. Expression API does not yet support PARTITION BY - use raw SQL
# 7. Nested partitioning is supported: a partition created with
# PostgresCreatePartitionStatement(partition_clause=...) is itself
# PARTITION BY <strategy>; PostgreSQL lets each HASH branch choose its own
# child strategy. See PostgresPartitionPlan for the tree-maintenance API.
# 8. A partitioned table's primary key must include every level's partition
# key, otherwise the server rejects the child that is itself partitioned.
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@
PostgresDetachPartitionStatement,
PostgresAttachPartitionStatement,
PostgresPartitionMetadataQueryStatement,
PostgresPartitionTreeQueryStatement,
PostgresAlterIndexStatement,
PostgresAlterIndexActionType,
PostgresReindexStatement,
Expand Down Expand Up @@ -231,6 +232,7 @@
"PostgresDetachPartitionStatement",
"PostgresAttachPartitionStatement",
"PostgresPartitionMetadataQueryStatement",
"PostgresPartitionTreeQueryStatement",
"PostgresAlterIndexStatement",
"PostgresAlterIndexActionType",
"PostgresReindexStatement",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@
PostgresDetachPartitionStatement,
PostgresAttachPartitionStatement,
PostgresPartitionMetadataQueryStatement,
PostgresPartitionTreeQueryStatement,
)
from .index import (
PostgresAlterIndexStatement,
Expand Down Expand Up @@ -225,6 +226,7 @@
"PostgresDetachPartitionStatement",
"PostgresAttachPartitionStatement",
"PostgresPartitionMetadataQueryStatement",
"PostgresPartitionTreeQueryStatement",
# index
"PostgresAlterIndexStatement",
"PostgresAlterIndexActionType",
Expand Down
Loading
Loading