Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 69 additions & 2 deletions docs/api/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ This is the **taxonomy** tree, not the term tree: nodes are taxonomies. Every `t
**Goal:** same structure, but also pull each taxonomy's terms in the same call.

```python
structure = norbix.api.database.find_taxonomy_tree(include_terms=True)
structure = norbix.api.database.find_taxonomy_tree(includeTerms=True)
```

```json
Expand Down Expand Up @@ -316,7 +316,7 @@ structure = norbix.api.database.find_taxonomy_tree(include_terms=True)

Now each taxonomy node's `terms` holds that taxonomy's full term tree (same shape as `find_term_tree`) — *Countries* carries its countries, *Cities* carries its cities.

> Every term-reading call also accepts an optional `database_integration_id` to target a non-default database.
> Every term-reading call also accepts an optional `databaseIntegrationId` to target a non-default database.

---

Expand All @@ -337,3 +337,70 @@ roots, and the terms of its child taxonomies nest under them.
```python
tree = norbix.api.database.find_merged_term_tree("services")
```

---

## Changing many records

`update_many` and `delete_many` change every record that matches `filter`. An
empty filter (`'{}'`) matches **every** record of the collection, so the
gateway refuses it with `CM-ERRORS-DATABASE-037` unless you also send
`allRecords=True`. For `update_many` a missing filter counts as `'{}'`.

```python
norbix.api.database.update_many("products", filter='{"status": "draft"}', update='{"status": "live"}')
norbix.api.database.delete_many("products", filter="{}", allRecords=True) # every record
```

Pass the flag with its wire name, `allRecords` (the SDK sends keyword
arguments as given).

- **Own-record rights are enough.** A caller with only `createAsUser`,
`updateOwn` or `deleteOwn` may call `insert_many`, `update_many` and
`delete_many`; the call changes only the caller's own records (before, it
was refused with 403).
- **No `$` operators in an update.** `update_one` / `update_many` take the new
values as a plain document (`'{"price": 10}'`). `$inc`, `$set` and the like
are refused with `CM-ERRORS-DATABASE-035`.
- **A broken record document** on `insert_one`, `insert_many` or `replace_one`
is refused with `CM-ERRORS-DATABASE-036` "Invalid record document" (before,
`005`). For `insert_many`, `err.errors[0].context["Index"]` is the position
of the bad document.
- **Soft-deleted records are not found** by update, replace and change owner;
a bulk update skips them.
- **Change owner** (`change_responsibility`) refuses a new owner who is not a
user of the project in the request environment, with
`CM-ERRORS-MEMBERSHIP-USERS-012`.

```python
from norbix_python import NorbixError

try:
norbix.api.database.delete_many("products", filter="{}")
except NorbixError as err:
assert err.error_code == "CM-ERRORS-DATABASE-037"
```

## Term reads: rights and errors

- Term reads by taxonomy name (`find_terms`, `find_terms_children`,
`find_term_tree`, `find_merged_term_tree`) check `database:read` on
`database:term:{taxonomy id}`. The merged tree checks it on every nested
taxonomy too.
- `find_terms` and `find_terms_children` refuse `$where`, `$function` and
`$accumulator` in `filter` with `CM-ERRORS-DATABASE-031`.
- `CM-ERRORS-TAXONOMIES-010`: the taxonomy name is unknown (the merged tree
used to answer `-003`).
- `CM-ERRORS-TAXONOMIES-011`: the term tree has more than 5000 terms (whole
taxonomy, merged tree, `find_taxonomy_tree(includeTerms=True)`).
- `CM-ERRORS-TAXONOMIES-005`: a taxonomy name longer than 40 characters.
- `find_taxonomy_tree(includeTerms=True)` fails when reading the terms fails.
Before, it returned the taxonomies without terms.

## Aggregates

`aggregate` and `execute_aggregate` refuse a pipeline that joins a collection
the caller cannot read (`CM-ERRORS-DATABASE-033`), a name that is not a schema
in the request environment (`032`), or a join into another database (`034`).
`execute_aggregate` also needs `database:read` on the collection, and refuses
a saved `$out` / `$merge` / JavaScript stage (`031`).
160 changes: 159 additions & 1 deletion docs/hub/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ Access with `norbix.hub.database`.
| `get_database_aggregates` | `GET` | `/{version}/database/aggregates` | `project` |
| `save_database_aggregate` | `POST` | `/{version}/database/aggregates` | `project` |
| `test_database_aggregate` | `POST` | `/{version}/database/aggregates/test` | `project` |
| `create_collection_import` | `POST` | `/{version}/database/imports` | `project` |
| `get_collection_imports` | `GET` | `/{version}/database/imports` | `project` |
| `get_collection_import` | `GET` | `/{version}/database/imports/{Id}` | `project` |
| `delete_collection_import` | `DELETE` | `/{version}/database/imports/{Id}` | `project` |
| `request_import_upload_url` | `POST` | `/{version}/database/imports/upload-url` | `project` |
| `analyze_import_file` | `POST` | `/{version}/database/imports/analyze` | `project` |
| `get_allowed_flex_tiers` | `GET` | `/{version}/database/integrations/flex-tiers` | `project` |
| `test_database_integration` | `POST` | `/{version}/database/integrations/test` | `project` |
| `reveal_managed_flex_connection_string` | `GET` | `/{version}/database/integrations/{Id}/connection-string` | `project` |
Expand Down Expand Up @@ -85,7 +91,7 @@ extended-JSON **strings**.
products = norbix.hub.database.find_records("products", filter='{"status": "active"}', pageSize=20)
one = norbix.hub.database.find_one_record("products", "rec_1")
created = norbix.hub.database.insert_record("products", document='{"title": "Lamp", "price": 12}')
norbix.hub.database.update_one_record("products", "rec_1", update='{"$set": {"price": 10}}')
norbix.hub.database.update_one_record("products", "rec_1", update='{"price": 10}')
norbix.hub.database.delete_record("products", "rec_1")
total = norbix.hub.database.count_records("products", filter='{"status": "active"}')
indexes = norbix.hub.database.get_collection_indexes("products")
Expand All @@ -105,6 +111,158 @@ update no longer matches.
test_schemas = Norbix(project_id="...", api_key="...", env="TEST").hub.database.get_database_schemas()
```

## Changing many records

`update_many_records` and `delete_many_records` change every record that
matches `filter`. An empty filter (`'{}'`) matches **every** record of the
collection, so the gateway refuses it with `CM-ERRORS-DATABASE-037` unless
you also send `allRecords=True`. For `update_many_records` a missing filter
counts as `'{}'`.

```python
# Only the matching records:
norbix.hub.database.update_many_records("products", filter='{"status": "draft"}', update='{"status": "live"}')
# Every record — the flag says you mean it:
norbix.hub.database.delete_many_records("products", filter="{}", allRecords=True)
```

Pass the flag with its wire name, `allRecords` (the SDK sends keyword
arguments as given; on `update_many_records` it goes in the JSON body, on
`delete_many_records` in the query). The marketplace built-ins `db.update` and
`db.delete` take the same flag as the `allRecords` argument.

Other rules for record writes:

- **Who may call the bulk methods.** A caller with only own-record rights
(`createAsUser`, `updateOwn`, `deleteOwn`) may now call `insert_many_records`,
`update_many_records` and `delete_many_records`. The call changes only the
caller's own records (before, it was refused with 403).
- **No `$` operators in an update.** `update_one_record` / `update_many_records`
take the new values as a plain document. An update with `$inc`, `$set` and
the like is refused with `CM-ERRORS-DATABASE-035` (before, MongoDB failed
at run time).
- **A broken record document** on insert one, insert many or replace is
refused with `CM-ERRORS-DATABASE-036` "Invalid record document" (before,
`005` "Invalid filter document"). For insert many,
`err.errors[0].context["Index"]` is the position of the bad document.
- **Soft-deleted records are not found.** Update, replace and change owner
no longer match a soft-deleted record: it answers "not found", and a bulk
update skips it.
- **Change owner** (`change_record_responsibility`) refuses a new owner who is
not a user of the project in the request environment, with
`CM-ERRORS-MEMBERSHIP-USERS-012`.

## Aggregates

- `get_database_aggregate(id)` returns `joinedCollections`: the schemas the
pipeline joins (`$lookup` / `$graphLookup` / `$unionWith`), next to the
start `schemaId`. The aggregate list rows do not carry it.
- `save_database_aggregate(...)` refuses `$out`, `$merge`, `$where`,
`$function` and `$accumulator` (`CM-ERRORS-DATABASE-031`) and a join it
cannot follow (`034`); a pipeline that does not parse is refused with
`CM-ERRORS-AGGREGATES-002`.
- A pipeline (run, test or saved-and-executed) that joins a collection the
caller cannot read is refused with `033`; a join to a name that is not a
schema in the request environment with `032`.
- `test_database_aggregate(...)` needs `database:create` **or**
`database:update` on `database:aggregate:{schemaId}`, plus read. A caller
with read rights only gets 403.

## Schemas: rename and delete

- `rename_database_schema(id, title="...")` sends only the new title. The old
`renameUniqueName` switch is gone: a name that another schema in the same
environment already uses is always refused with `CM-ERRORS-SCHEMA-002`
("Two schemas cannot share one collection"); the error context carries
`SchemaName`. The schema name is the MongoDB collection name, so two schemas
can never share it.
- `delete_database_schema(id)` is also refused while a saved aggregate starts
from the schema or joins it: `CM-ERRORS-SCHEMA-018`. The message names the
aggregates; `err.errors[0].context` has `BlockerAggregateIds` and
`BlockerAggregateNames`. (A schema that a trigger uses is refused with
`CM-ERRORS-SCHEMA-017`, `BlockerTriggerIds`.)
- `save_database_schema(viewId=...)` with an id that is not a schema of the
request environment answers "schema not found".
- `update_database_schema_list_settings(...)` refusals all use
`CM-ERRORS-SCHEMA-034`; the context's `Rule` says which rule failed.
- `apply_database_schema_bundle(...)`: a refused step keeps its own error code
and context.

## Database integrations per environment

`get_database_integrations()` lists only the integrations of the client's
environment (`PROD` when no `env` is set); each row has `env`. A saved
integration has `isSystemOwned` (read only). Saving into an environment the
project does not have is refused. An integration whose secret is empty in the
request environment answers `CM-ERRORS-DATABASE-038`.

## Schema triggers per environment

Schema triggers are kept per environment. Every schema-trigger call works on
the client's environment (the `norbix-env` header; `PROD` when none is set):

- `get_schema_triggers()` lists only that environment's triggers; each row
has `env`.
- `get_schema_trigger(id)` returns `env`, and `schemaId` is now the owning
schema's id (`sch_…`). Before, it wrongly held the trigger's own id
(`trg_…`).
- `enable_schema_trigger`, `disable_schema_trigger` and `delete_schema_trigger`
act on the copy in that environment. No copy there answers
`CM-ERRORS-TRIGGERS-002` (not found).
- `save_schema_trigger(id=..., schemaId=...)` with the id of a trigger that
belongs to another schema answers `CM-ERRORS-TRIGGERS-002`.

```python
test = Norbix(project_id="...", api_key="...", env="TEST")
test.hub.database.disable_schema_trigger("trg_1") # the TEST copy only
```

## Taxonomies

- `get_database_taxonomies()` rows have `dependencyRefs` in place of the old
`dependencyNames`: one `{"id": ..., "name": ...}` pair per entry of
`dependencies`, in the same order. A dependency that no longer exists keeps
its place with `"name": None`.

```python
for row in norbix.hub.database.get_database_taxonomies()["list"]["items"]:
names = [ref["name"] or f"(deleted {ref['id']})" for ref in row.get("dependencyRefs") or []]
```

- `save_database_taxonomy(viewId=...)` with the id of an existing taxonomy is
an update and needs `database:update` on `database:taxonomy:{viewId}`.
Without `viewId` (or with an unknown one) it is a create and needs
`database:create`. An update replaces the whole taxonomy: send every field
you want to keep.
- Term reads by taxonomy name (term tree, merged tree) check
`database:read` on `database:term:{taxonomy id}`; the merged tree checks it
on every nested taxonomy too.
- `get_database_taxonomy_tree(includeTerms=True)` now fails when reading the
terms fails. Before, it returned the taxonomies without terms.
- Errors on term reads: `CM-ERRORS-TAXONOMIES-010` the taxonomy name is
unknown (the merged tree used to answer `-003`); `-011` the term tree has
more than 5000 terms (whole taxonomy, merged tree, `includeTerms`); `-005`
a taxonomy name longer than 40 characters.

## Collection imports

Import CSV rows into a collection: ask for an upload URL, upload the file,
analyze it, then create the import and follow it.

```python
target = norbix.hub.database.request_import_upload_url(fileName="products.csv")["result"]
httpx.put(target["url"], content=csv_bytes, headers={"Content-Type": target["contentType"]})
preview = norbix.hub.database.analyze_import_file(file=target["file"], hasHeader=True)
created = norbix.hub.database.create_collection_import(file=target["file"], schemaId="sch_1", hasHeader=True)
status = norbix.hub.database.get_collection_import(created["id"])
imports = norbix.hub.database.get_collection_imports()
norbix.hub.database.delete_collection_import(created["id"])
```

The request fields are listed under `RequestImportUploadUrlRequest`,
`AnalyzeImportFileRequest` and `CreateCollectionImport` in
`references/hub_dtos.py`.

## Trees, list settings, embed and bundles

- `get_database_taxonomy_tree()` — the taxonomy structure as a tree.
Expand Down
80 changes: 80 additions & 0 deletions docs/tasks/database-last-wave.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Database last wave — Python SDK side of the gateway contract changes

## Goal

The Python SDK follows the Database contract changes now on gateway
`refactoringV2` (records, taxonomies, triggers, schemas, aggregates): fresh
DTO references, every Database method matches the gateway's routes, tests and
docs cover each change, the full suite is green, the branch is shipped.

Not in scope: typed request / response classes (the Python SDK is untyped by
design — every request field is a keyword argument that goes to the wire as
given); other modules than Database.

## Plan

1. chore(references): refresh `references/{hub,api}_dtos.py` and
`references/{hub,api}2.dtos.ts` from the running Community Hub
(`localhost:64964`) and Api (`localhost:64965`) built from
`origin/refactoringV2` — done
2. feat(database): add the 6 collection-import methods the Hub serves but the
SDK did not have (`/database/imports…`), sync and async, so every Database
route of both hosts has a method — done
3. test(database): tests with a capture transport for every contract change
that reaches the wire: `allRecords` on update/delete many (Hub + Api),
rename without `renameUniqueName`, schema triggers per env (`norbix-env`
header), the import routes, a route check of every Database method
against the hosts' routes, and the new error codes surfacing as
`NorbixError.error_code` — done
4. docs(database): Hub and Api database pages — bulk writes (empty filter,
`allRecords`, own-scope bulk calls, `$` operators refused, record
document errors, soft-deleted rows), change owner, aggregate test rights,
schema delete blockers, `joinedCollections`, rename, schema triggers per
env, taxonomy `dependencyRefs`, term-read rights and new error codes — done
5. Full suite (`uv run pytest`, `make typecheck`, `make lint`) green; ship
with `nbx-ship --wait-release --cleanup` — done (993 passed, mypy and ruff
clean)

## Changes

| file | what changed | plan step # |
|---|---|---|
| `references/hub_dtos.py`, `references/hub2.dtos.ts` | refreshed from Hub `/types/python` and `/types/typescript` | 1 |
| `references/api_dtos.py`, `references/api2.dtos.ts` | refreshed from Api; the `.ts` was last taken from ServiceStack 8.30, so its diff is large | 1 |
| `src/norbix_python/hub/database.py` | `create_collection_import`, `get_collection_imports`, `get_collection_import`, `delete_collection_import`, `request_import_upload_url`, `analyze_import_file` (sync + async) | 2 |
| `tests/hub/test_database_last_wave.py` | new: request shapes for the contract changes, import routes, route check against the hosts' routes | 3 |
| `tests/api/test_database_last_wave.py` | new: `allRecords` on Api update/delete many, change owner error, `$where` refused on terms | 3 |
| `docs/hub/database.md` | import methods in the table; new sections for every contract change | 4 |
| `docs/api/database.md` | bulk writes, change owner, term reads and the new error codes | 4 |

## Findings

- The Python SDK had no method for the 6 Hub collection-import routes
(`/database/imports`, `/imports/{Id}`, `/imports/upload-url`,
`/imports/analyze`); `norbix-js` has them. Fixed here (step 2).
- `references/api2.dtos.ts` was a year old (ServiceStack 8.30, 2025-10-22).
Refreshed here (step 1).
- The SDK forwards keyword arguments by their given name: `all_records=True`
would go out as `all_records`, which the gateway does not read. The docs
and tests use the wire name `allRecords=True`, the same way the rest of the
SDK uses `pageSize`. Left as is (this is the SDK's documented rule).
- `docs/api/database.md` showed `find_taxonomy_tree(include_terms=True)` and
"an optional `database_integration_id`": by the rule above those go out
under the snake_case name, which the gateway's request names
(`includeTerms`, `databaseIntegrationId`) do not match. Fixed here (step 4).
- `docs/hub/database.md` showed `update_one_record(..., update='{"$set": ...}')`,
which the gateway now refuses (`CM-ERRORS-DATABASE-035`). Fixed here (step 4).

## Rejected / moved out

- A local guard that refuses an empty filter before sending. The gateway owns
this rule (CM-ERRORS-DATABASE-037); a copy in the SDK would drift.

## Needs you

- Nothing. The import methods are new methods (a minor release), no existing
method changed its name, parameters or return type.

## Open questions

- None.
Loading
Loading