Skip to content

Commit 3859ce6

Browse files
authored
refactor(backend): postgres backend improvements and CI update (#12)
* refactor(backend): reorganize mixins/protocols into module directories Split large single-file modules into proper package structures: - mixins.py → mixins/ (with ddl/, dml/, extensions/, types/ submodules) - protocols.py → protocols/ (matching structure) Add PostgreSQL-specific expression classes for: - DML: VACUUM, ANALYZE - DDL: PARTITION, REINDEX, STATISTICS, COMMENT, MATERIALIZED VIEW, ENUM Update documentation with version-specific features tables: - Parallel query, stored procedures, extended statistics - PostgreSQL 15+, 16+, 17+ new features Add pytest-mock to test dependencies. * style: replace copyright headers with relative path comments * chore: bump rhosocial-activerecord to 1.0.0.dev19 in CI * test(backend): add unit tests for PostgreSQL dialect features Add comprehensive unit tests to improve code coverage from 69% to 74%. New test files: - test_stored_procedure.py: stored procedure DDL/DML - test_extended_statistics.py: CREATE/DROP STATISTICS - test_trigger.py: CREATE/DROP TRIGGER statements - test_index_pg.py: REINDEX and CREATE INDEX with PG options - test_range_functions.py: range operators and functions - test_extensions.py: hstore, ltree, intarray, pg_trgm, pgvector, PostGIS Total: 186 new test cases, all passing. * docs: add changelog fragment for test additions (PR #12)
1 parent ef7ab70 commit 3859ce6

90 files changed

Lines changed: 10043 additions & 2448 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ jobs:
6767
6868
- name: Install activerecord and testsuite from PyPI
6969
run: |
70-
pip install rhosocial-activerecord==1.0.0.dev18
70+
pip install rhosocial-activerecord==1.0.0.dev19
7171
pip install rhosocial-activerecord-testsuite==1.0.0.dev7
7272
7373
- name: Create postgres scenario config

‎changelog.d/12.added.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Added comprehensive unit tests for PostgreSQL dialect features, improving code coverage from 69% to 74%. New tests cover stored procedures, extended statistics, triggers, indexes, range functions, and extensions (hstore, ltree, intarray, pg_trgm, pgvector, PostGIS).

‎docs/en_US/introduction/supported_versions.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,56 @@
3737
| Stored generated columns | 12+ |
3838
| JSON path queries | 12+ |
3939

40+
### Parallel Query Features
41+
42+
| Feature | Minimum Version |
43+
|---------|-----------------|
44+
| Parallel sequential scan | 9.6+ |
45+
| Parallel Append | 10+ |
46+
| Parallel index scan | 10+ |
47+
| Gather Merge | 10+ |
48+
| Parallel index-only scan | 11+ |
49+
| Parallel hash join | 11+ |
50+
51+
### Stored Procedures
52+
53+
| Feature | Minimum Version |
54+
|---------|-----------------|
55+
| CALL statement | 11+ |
56+
| Transaction control in procedures | 11+ |
57+
| SQL-standard function body | 14+ |
58+
59+
### Extended Statistics
60+
61+
| Feature | Minimum Version |
62+
|---------|-----------------|
63+
| CREATE STATISTICS | 10+ |
64+
| Functional dependencies | 10+ |
65+
| NDistinct statistics | 10+ |
66+
| MCV (Most Common Values) | 12+ |
67+
68+
### PostgreSQL 15+ Features
69+
70+
| Feature | Minimum Version |
71+
|---------|-----------------|
72+
| MERGE statement | 15+ |
73+
| UNIQUE NULLS NOT DISTINCT | 15+ |
74+
75+
### PostgreSQL 16+ Features
76+
77+
| Feature | Minimum Version |
78+
|---------|-----------------|
79+
| REGEXP_LIKE function | 16+ |
80+
| random_normal() function | 16+ |
81+
82+
### PostgreSQL 17+ Features
83+
84+
| Feature | Minimum Version |
85+
|---------|-----------------|
86+
| Enhanced JSON_TABLE | 17+ |
87+
| MERGE with CTE | 17+ |
88+
| UPDATE RETURNING OLD | 17+ |
89+
4090
## Python Version Support
4191

4292
| Python Version | Support Status | Notes |

‎docs/zh_CN/introduction/supported_versions.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,56 @@
3737
| 存储生成列 | 12+ |
3838
| JSON 路径查询 | 12+ |
3939

40+
### 并行查询功能
41+
42+
| 功能 | 最低版本 |
43+
|-----|---------|
44+
| 并行顺序扫描 | 9.6+ |
45+
| 并行 Append | 10+ |
46+
| 并行索引扫描 | 10+ |
47+
| Gather Merge | 10+ |
48+
| 并行仅索引扫描 | 11+ |
49+
| 并行哈希连接 | 11+ |
50+
51+
### 存储过程
52+
53+
| 功能 | 最低版本 |
54+
|-----|---------|
55+
| CALL 语句 | 11+ |
56+
| 存储过程事务控制 | 11+ |
57+
| SQL 标准函数体 | 14+ |
58+
59+
### 扩展统计信息
60+
61+
| 功能 | 最低版本 |
62+
|-----|---------|
63+
| CREATE STATISTICS | 10+ |
64+
| 函数依赖 | 10+ |
65+
| NDistinct 统计 | 10+ |
66+
| MCV(最常见值) | 12+ |
67+
68+
### PostgreSQL 15+ 功能
69+
70+
| 功能 | 最低版本 |
71+
|-----|---------|
72+
| MERGE 语句 | 15+ |
73+
| UNIQUE NULLS NOT DISTINCT | 15+ |
74+
75+
### PostgreSQL 16+ 功能
76+
77+
| 功能 | 最低版本 |
78+
|-----|---------|
79+
| REGEXP_LIKE 函数 | 16+ |
80+
| random_normal() 函数 | 16+ |
81+
82+
### PostgreSQL 17+ 功能
83+
84+
| 功能 | 最低版本 |
85+
|-----|---------|
86+
| 增强的 JSON_TABLE | 17+ |
87+
| MERGE with CTE | 17+ |
88+
| UPDATE RETURNING OLD | 17+ |
89+
4090
## Python 版本支持
4191

4292
| Python 版本 | 支持状态 | 说明 |

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ pooling = [
4848
test = [
4949
"pytest>=7.0.0",
5050
"coverage>=7.0.0",
51+
"pytest-mock>=3.0.0",
5152
]
5253
dev = [
5354
"black>=23.0.0",

‎src/rhosocial/activerecord/backend/impl/postgres/adapters/base.py‎

Lines changed: 133 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
class PostgresListAdapter(SQLTypeAdapter):
1717
"""
1818
Adapts Python list to PostgreSQL array types.
19-
19+
2020
This adapter does not perform any conversion - psycopg handles
2121
Python lists natively for PostgreSQL array types.
2222
"""
@@ -37,11 +37,42 @@ def from_database(self, value: Any, target_type: Type, options: Optional[Dict[st
3737
return value
3838
raise TypeError(f"Cannot convert {type(value).__name__} to list")
3939

40+
def to_database_batch(self, values: List[Any], target_type: Type,
41+
options: Optional[Dict[str, Any]] = None) -> List[Any]:
42+
"""Optimized batch conversion - psycopg handles lists natively.
43+
44+
No per-element processing needed, just pass through.
45+
This avoids the overhead of calling to_database() for each element.
46+
47+
Args:
48+
values: List of values to convert
49+
target_type: Target type (not used for lists)
50+
options: Optional conversion options
51+
52+
Returns:
53+
The same list, psycopg handles conversion internally
54+
"""
55+
return values
56+
57+
def from_database_batch(self, values: List[Any], target_type: Type,
58+
options: Optional[Dict[str, Any]] = None) -> List[Any]:
59+
"""Optimized batch conversion from database.
60+
61+
Args:
62+
values: List of values from database
63+
target_type: Target Python type
64+
options: Optional conversion options
65+
66+
Returns:
67+
The same list, psycopg already returns Python lists
68+
"""
69+
return values
70+
4071

4172
class PostgresJSONBAdapter(SQLTypeAdapter):
4273
"""
4374
Adapts Python dict to PostgreSQL JSONB and vice-versa.
44-
75+
4576
Note: psycopg automatically serializes/deserializes JSON, so dict <-> JSON.
4677
When reading from DB, psycopg returns dict. When target type is str,
4778
we need to serialize back to JSON string.
@@ -55,7 +86,8 @@ def to_database(self, value: Union[dict, list], target_type: Type, options: Opti
5586
return None
5687
return Jsonb(value)
5788

58-
def from_database(self, value: Any, target_type: Type, options: Optional[Dict[str, Any]] = None) -> Union[dict, list]:
89+
def from_database(self, value: Any, target_type: Type,
90+
options: Optional[Dict[str, Any]] = None) -> Union[dict, list]:
5991
if value is None:
6092
return None
6193
# For string target type, serialize dict/list back to JSON string
@@ -73,6 +105,55 @@ def from_database(self, value: Any, target_type: Type, options: Optional[Dict[st
73105
# In case it's a string representation
74106
return json.loads(value)
75107

108+
def to_database_batch(self, values: List[Any], target_type: Type,
109+
options: Optional[Dict[str, Any]] = None) -> List[Any]:
110+
"""Optimized batch JSON conversion to database format.
111+
112+
Creates Jsonb wrappers for all values in a single pass,
113+
reducing function call overhead compared to individual calls.
114+
115+
Args:
116+
values: List of values to convert
117+
target_type: Target type (Jsonb)
118+
options: Optional conversion options
119+
120+
Returns:
121+
List of Jsonb-wrapped values
122+
"""
123+
result = []
124+
for value in values:
125+
if value is None:
126+
result.append(None)
127+
else:
128+
result.append(Jsonb(value))
129+
return result
130+
131+
def from_database_batch(self, values: List[Any], target_type: Type,
132+
options: Optional[Dict[str, Any]] = None) -> List[Any]:
133+
"""Optimized batch JSON deserialization from database.
134+
135+
Args:
136+
values: List of values from database
137+
target_type: Target Python type (dict, list, or str)
138+
options: Optional conversion options
139+
140+
Returns:
141+
List of converted values
142+
"""
143+
result = []
144+
for value in values:
145+
if value is None:
146+
result.append(None)
147+
elif target_type is str:
148+
if isinstance(value, (dict, list)):
149+
result.append(json.dumps(value))
150+
else:
151+
result.append(value)
152+
else:
153+
# For dict/list target types, return as-is
154+
result.append(value)
155+
return result
156+
76157

77158

78159
class PostgresNetworkAddressAdapter(SQLTypeAdapter):
@@ -111,6 +192,55 @@ def from_database(self, value: Any, target_type: Type, options: Optional[Dict[st
111192
except (ImportError, ValueError):
112193
return value
113194

195+
def to_database_batch(self, values: List[Any], target_type: Type,
196+
options: Optional[Dict[str, Any]] = None) -> List[Any]:
197+
"""Optimized batch network address conversion.
198+
199+
Converts all values to strings in a single pass.
200+
201+
Args:
202+
values: List of values to convert
203+
target_type: Target type
204+
options: Optional conversion options
205+
206+
Returns:
207+
List of string representations
208+
"""
209+
result = []
210+
for value in values:
211+
if value is None:
212+
result.append(None)
213+
else:
214+
result.append(str(value))
215+
return result
216+
217+
def from_database_batch(self, values: List[Any], target_type: Type,
218+
options: Optional[Dict[str, Any]] = None) -> List[Any]:
219+
"""Optimized batch network address parsing from database.
220+
221+
Args:
222+
values: List of string values from database
223+
target_type: Target Python type
224+
options: Optional conversion options
225+
226+
Returns:
227+
List of ipaddress objects or strings
228+
"""
229+
import ipaddress
230+
result = []
231+
for value in values:
232+
if value is None:
233+
result.append(None)
234+
else:
235+
try:
236+
result.append(ipaddress.ip_address(value))
237+
except (ImportError, ValueError):
238+
try:
239+
result.append(ipaddress.ip_network(value))
240+
except (ImportError, ValueError):
241+
result.append(value)
242+
return result
243+
114244

115245
class PostgresEnumAdapter(SQLTypeAdapter):
116246
"""PostgreSQL ENUM type adapter.

‎src/rhosocial/activerecord/backend/impl/postgres/backend/async_backend.py‎

Lines changed: 28 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@
5252
from ..adapters.object_identifier import PostgresOidAdapter, PostgresXidAdapter, PostgresTidAdapter
5353
from ..adapters.pg_lsn import PostgresLsnAdapter
5454
from ..adapters.text_search import PostgresTsVectorAdapter, PostgresTsQueryAdapter
55-
from ..config import PostgresConnectionConfig
55+
from ..config import PostgresConnectionConfig, RangeAdapterMode
5656
from ..dialect import PostgresDialect
5757
from .base import PostgresBackendMixin
5858
from ..protocols import PostgresExtensionInfo
@@ -126,12 +126,12 @@ def __init__(self, **kwargs):
126126
def _register_postgres_adapters(self):
127127
"""Register PostgreSQL-specific type adapters.
128128
129-
Note: PostgresRangeAdapter and PostgresMultirangeAdapter are NOT registered
130-
by default to allow users to choose:
131-
1. Use psycopg.types.range.Range directly (pass-through)
132-
2. Use PostgresRange by manually registering the adapter
133-
134-
This design follows the "don't make decisions for the user" principle.
129+
Range and Multirange adapter registration is controlled by configuration:
130+
- range_adapter_mode: Controls how Range types are handled
131+
- NATIVE (default): Use psycopg's native Range type (pass-through)
132+
- CUSTOM: Use PostgresRange custom type
133+
- BOTH: Register both adapters, with custom taking precedence
134+
- multirange_adapter_mode: Same for Multirange types (PG 14+ only)
135135
136136
Note: The following adapters are NOT registered by default due to str->str
137137
type mapping conflicts with existing string handling:
@@ -145,10 +145,7 @@ def _register_postgres_adapters(self):
145145
PostgresListAdapter(),
146146
PostgresJSONBAdapter(),
147147
PostgresNetworkAddressAdapter(),
148-
# PostgresRangeAdapter(), # Not registered by default - user choice
149-
# PostgresMultirangeAdapter(), # Not registered by default - user choice
150148
PostgresGeometryAdapter(),
151-
# PostgresBitStringAdapter(), # Not registered: str->str conflict
152149
PostgresEnumAdapter(),
153150
PostgresMoneyAdapter(),
154151
PostgresMacaddrAdapter(),
@@ -169,6 +166,27 @@ def _register_postgres_adapters(self):
169166
for db_type in db_types:
170167
self.adapter_registry.register(adapter, py_type, db_type)
171168

169+
# Register Range adapters based on configuration
170+
range_mode = getattr(self.config, 'range_adapter_mode', RangeAdapterMode.NATIVE)
171+
if range_mode in (RangeAdapterMode.CUSTOM, RangeAdapterMode.BOTH):
172+
range_adapter = PostgresRangeAdapter()
173+
for py_type, db_types in range_adapter.supported_types.items():
174+
for db_type in db_types:
175+
self.adapter_registry.register(range_adapter, py_type, db_type)
176+
self.log(logging.DEBUG, "Registered PostgresRangeAdapter (custom range type)")
177+
178+
# Register Multirange adapters based on configuration (PG 14+ only)
179+
# Note: Version check happens in dialect, but at this point we haven't
180+
# connected yet, so we register if configured. The adapter itself will
181+
# handle version-specific logic if needed.
182+
multirange_mode = getattr(self.config, 'multirange_adapter_mode', RangeAdapterMode.NATIVE)
183+
if multirange_mode in (RangeAdapterMode.CUSTOM, RangeAdapterMode.BOTH):
184+
multirange_adapter = PostgresMultirangeAdapter()
185+
for py_type, db_types in multirange_adapter.supported_types.items():
186+
for db_type in db_types:
187+
self.adapter_registry.register(multirange_adapter, py_type, db_type)
188+
self.log(logging.DEBUG, "Registered PostgresMultirangeAdapter (custom multirange type)")
189+
172190
self.log(logging.DEBUG, "Registered PostgreSQL-specific type adapters")
173191

174192
@property

0 commit comments

Comments
 (0)