Skip to content

feat(entities): new entity APIs — cursor pages, count, aggregate, upsert - #287

Merged
arosenan merged 12 commits into
mainfrom
entities-scan-free-primitives
Sep 22, 2026
Merged

arosenan merged 12 commits into
mainfrom
entities-scan-free-primitives

Conversation

@arosenan

Copy link
Copy Markdown
Contributor

Why

Fifty apps with the deepest skip reads on the entities cluster were reviewed. None of them pages because a user asked for page N. Every deep-skip loop exists to get something the SDK cannot give:

Need Apps of 50 New API
A number: count, sum, group-by, avg, max-per-key, count-distinct ~37 aggregate(), count()
"Does this key already exist?" before insert; duplicate cleanup ~23 upsert(), aggregate({ having })
Walk a big table in order: export, sitemap, resumable job ~15 cursor on list() / filter()
Distinct values for dropdowns / autocomplete 5 distinct option on list() / filter()

Three apps built binary search with limit-1 probes as a substitute for count (skips of 10.2M, 900k and 64k). Builders persist lastSkip as a resume token because nothing else exists.

The server routes land in base44-dev/apper#24939. This PR is the SDK half, and its purpose right now is to settle the API shape. Version stays 0.8.x: everything is additive, so existing apps on ^0.8.x pick it up on their next install.

Proposed API

// Cursor pages: same list()/filter(), an options object instead of positional args.
// The token carries the query, sort and fields, so later pages need only the cursor.
let page = await base44.entities.Order.filter(
  { status: 'open' },
  { sort: '-created_date', limit: 1000 }
);                                        // -> { items, next_cursor, has_more }
while (page.has_more) {
  page = await base44.entities.Order.filter({ status: 'open' }, { cursor: page.next_cursor, limit: 1000 });
}

await base44.entities.Order.list('-created_date', 100);   // unchanged, -> Order[]

// A number, not rows.
const open = await base44.entities.Order.count({ status: 'open' });

// Distinct values of a field, for dropdowns and autocomplete: a page of values, not records.
const { items: brands } = await base44.entities.Product.filter({ active: true }, { distinct: 'brand' });
// items: ['Adidas', 'Nike', ...]  (ascending, cursor-paged like any other page)

// Group-by aggregates, computed server-side.
const { rows } = await base44.entities.Sale.aggregate({
  query: { sale_date: { $gte: '2026-09-01' } },
  groupBy: 'agent_id',
  sum: 'amount',
  sort: '-sum_amount',
});
// rows: [{ agent_id, count, sum_amount }, ...]; also dateBucket, avg/min/max, countDistinct, having, limit

// Create or update by a key of your own, up to 500 records per call.
const { created, updated, records } = await base44.entities.Contact.upsert(rows, { key: 'crm_id' });

Design notes:

  • The cursor token carries the query, sort and fields of the walk, like Wix Data's cursorPaging: a later page needs only cursor and limit, and a different query or sort with a cursor is a 400 rather than silently ignored. The server re-sanitizes the query and applies scope and RLS on every page, so the token adds no new trust.
  • distinct is an option on list()/filter(), like distinct() on a Wix Data or Mongo query, rather than a group-by spelled through aggregate(). It returns the same page envelope with values as items.
  • The cursor lives on list()/filter() rather than a new method, matching Stripe/Firestore/Prisma. Passing an options object is what switches the return type to a page envelope; positional calls keep returning arrays, so no existing code changes. skip stays and is documented as deprecated for loops. The options form defaults to 100 rows per page (Notion-style), the positional form keeps its 5,000 default.
  • Vocabulary follows the SDK, not Mongo: the aggregate filter is query like filter(query), spec keys are camelCase like the SDK's methods; server-produced fields stay snake_case (next_cursor, has_more, sum_amount) like created_date.
  • updateMany is unchanged here. Making it update every match, instead of batches the caller loops over, is a server-side change tracked in the backend PR.

What

  • list(options) / filter(query, options)GET /{entity}/v2/listEntityPage<T>; EntityListOptions<T, K> typed against the schema. With EntityDistinctOptions<T, K> ({ distinct, limit, cursor }) the same route returns EntityPage<T[K]>. Same route family as list, versioned like /conversations/v2/… in the agents module; the backend route takes this path once the API is closed.
  • count(query?)GET /{entity}/count?q=number.
  • aggregate(spec)POST /{entity}/aggregateEntityAggregateResult (server caps at 1000 rows); EntityAggregateSpec<T> typed against the schema. Spec keys are query, groupBy, dateBucket, count, sum, avg, min, max, countDistinct, having, sort, limit; the spec is the wire body.
  • upsert(records, { key })POST /{entity}/upsertEntityUpsertResult<T>.

New types are exported from the package entry and listed in types-to-expose.json for the docs pipeline. JSDoc follows the sdk-docs-writing skill: description, params, returns, examples on every method.

Tests

  • tests/unit/entities-primitives.test.ts: nock tests for each route, params and body; positional list/filter still hit the list route and return arrays.
  • tests/types/entities-primitives.types.ts: spec and option fields are tied to the entity type; unknown fields, units, sorts and keys fail to compile.
  • npm run build, npm run test:types, npm run test:unit (308 tests) and npm run lint pass.

No version bump, following the repo's separate chore: bump version commits.

🤖 Generated with Claude Code

arosenan and others added 11 commits September 16, 2026 12:50
…egate, upsert

Wraps the scan-free entity routes added in base44-dev/apper#24939. Fifty apps with
the deepest `skip` reads were reviewed; none paged because a user asked for page N.
Every loop existed to get a number, to check whether a key already exists, to walk a
table with a resume point that is not an offset, or to list a field's distinct values.

- list(options) / filter(query, options): pass an options object {sort, limit,
  cursor, fields} instead of positional args to read one cursor page; returns
  {items, next_cursor, has_more}. Positional calls are unchanged. skip is
  documented as deprecated for loops.
- count(query?): number of readable records matching a filter.
- distinct(field, query?): {values, truncated}, capped at 5000 values.
- aggregate(spec): group_by / date_bucket / count / sum / avg / min / max /
  count_distinct / having / sort / limit; returns {rows, truncated}.
- upsert(records, {key}): create or update by a natural key, up to 500 records.

New public types are exported and listed in types-to-expose.json.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rameter

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- aggregate spec: `query` (not `match`), camelCase keys (`groupBy`, `dateBucket`,
  `countDistinct`), and no rule against combining countDistinct with other measures
- cursor pages default to 100 rows; the maximum stays 5,000

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A cursor token now carries the query, sort and fields of the walk, so a
later page needs only cursor and limit (same model as Wix Data cursorPaging).
distinct moves out of aggregate-only usage into an option on list()/filter()
that returns a page of values, matching how query APIs usually expose it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
aggregate(stages[]) posts { pipeline } to the same route; field names are the
entity's own and the server translates them. The spec form stays the primary,
typed API; the pipeline is the escape hatch for what the spec cannot express.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

🚀 Package Preview Available!


Install this PR's preview build with npm:

npm i @base44-preview/sdk@0.8.48-pr.287.c4af256

Prefer not to change any import paths? Install using npm alias so your code still imports @base44/sdk:

npm i "@base44/sdk@npm:@base44-preview/sdk@0.8.48-pr.287.c4af256"

Or add it to your package.json dependencies:

{
  "dependencies": {
    "@base44/sdk": "npm:@base44-preview/sdk@0.8.48-pr.287.c4af256"
  }
}

Preview published to npm registry — try new features instantly!

@github-actions github-actions Bot added the docs-draft PR has auto-drafted documentation suggestions label Sep 18, 2026
The entities primitives (and every other route that raises ApiError) answer
with {"error": {"code", "message", "details"}}. The error mapper only read the
legacy top-level code/message, so a caller catching an invalid_cursor 400 saw
code undefined and axios's generic message. Found while running the preview
build of this branch inside a Base44 app.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@arosenan
arosenan merged commit dea1829 into main Sep 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-draft PR has auto-drafted documentation suggestions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants