Developer Guide

Developer Guide

How to work on API Dock itself: setup, tests, conventions, how the code is laid out, and how releases are made.

Setup

curl -fsSL https://pixi.sh/install.sh | bash        # once: install pixi
git clone https://github.com/SchmidtDSE/api_dock.git
cd api_dock
pixi install -e dev                                  # dev tools, psycopg, a local PostgreSQL
pixi run -e dev pytest -q                            # run the tests

The dev environment includes the optional postgres extra (psycopg and psycopg_pool) and conda-forge postgresql. The postgres_server fixture in tests/conftest.py starts a throwaway PostgreSQL server for the test session. PostgreSQL tests are skipped when PostgreSQL or psycopg isn't installed, so the rest of the suite runs anywhere.

Contributing

1. Branch from main.
2. Make the change, with tests for any new behaviour.
3. Run pixi run -e dev pytest -q; it must pass.
4. Commit file by file. Start each message with the file name, prefixed by a short task label when several files change for one reason: (task label) path/to/file.py: what changed.
5. Open a pull request against main.

Code conventions

• PEP 8, 100-character lines, type hints on every function, Google-style docstrings.
• Each .py file's module docstring ends with License: BSD 3-Clause.
• Modules are laid out in four sections, each headed by a three-line comment with two blank lines above and none below: IMPORTS, CONSTANTS, PUBLIC (names without a leading underscore) and INTERNAL (names with one).
• Names imported from typing are listed alphabetically.
• Two blank lines between module-level functions, one between methods, and no blank line after a docstring or section comment.
• Prefer clear, small helpers over abstraction for its own sake.

Codebase

api_dock/
  cli.py                 api-dock commands: init, start, describe, encrypt, decrypt, generate-key
  config.py              main and remote config loading, settings, route allow/deny, cookies
  config_discovery.py    finds configs by name; `init` copies the example config
  route_mapper.py        RouteMapper: request handling for remotes and databases, startup checks,
                         start()/aclose() for PostgreSQL pools
  fast_api.py            FastAPI app: routes, streaming proxy, lifespan, base_path middleware
  flask_api.py           Flask app (buffered; refuses PostgreSQL configs)
  database_config.py     database configs, the shared databases/config.yaml (tables, meta,
                         schemas, connections, slugs, shared routes, include/exclude),
                         version discovery, route matching, config checks
  sql_builder.py         builds SQL: conditional selection, [[table]] and union expansion,
                         bound values, query-param fragments, engine choice per route
  sql_template_check.py  refuses quoted/commented variables and templates ending in a comment
  database_backends.py   DatabaseBackend interface; DuckDBBackend (worker thread, duckdb
                         settings, storage credentials, schema views, attached PostgreSQL)
  postgres_config.py     PostgreSQL connection checks, env: resolution, connection strings
  postgres_backend.py    PostgresBackend and PostgresPools (psycopg pools, one per connection)
  storage_auth.py        DuckDB credentials for S3 (scoped per table), GCS, Azure, HTTP
  listings.py            the `expose` catalog endpoints
  lookups.py             lookups: definitions, templates and safety checks, SQL/HTTP runners,
                         LookupStore (rows, refresh), refresh endpoint settings
  auth.py, encryption.py authentication providers for database routes; value encryption
  types.py               ProxyResponse, PreparedRequest, ListingSpec, TableReference, SqlContext
  example_api_dock_config/  copied by `api-dock init`
tests/                   one file per feature area; conftest.py has the PostgreSQL fixture

How a database request is handled

RouteMapper.map_database_route does the following:

1. Resolves the version (latest, version files and slugs).
2. Loads the version config and merges the main config and the shared routes/query params that apply to it.
3. Checks authentication.
4. Matches the route and merges top-level query params.
5. Handles early responses (response:, required).
6. Chooses the engine (route_engine).
7. Builds the SQL with ? (DuckDB) or %s (psycopg) markers and the bound values (build_sql_query_with_tables).
8. Runs it on the chosen backend and converts the rows to JSON.

Remote requests go through prepare_remote_request, which validates and builds the upstream request. The FastAPI app streams it with httpx.

Releasing

Publishing a GitHub release triggers .github/workflows/publish_to_pypi.yml, which builds the package and uploads it to PyPI with trusted publishing. Nothing is built or uploaded locally.

export VERSION=0.9.1                         # the new version, no leading "v"
# set version = "$VERSION" in pyproject.toml, then:
pixi run -e dev pytest -q
export COMMIT_MESSAGE='short summary'        # the commit adds the "v$VERSION: " prefix
git add -A && git commit -m "v$VERSION: $COMMIT_MESSAGE"
git tag "v$VERSION" && git push origin main "v$VERSION"
gh release create "v$VERSION" --title "v$VERSION" --notes "..."   # not --draft

Check PyPI with https://pypi.org/pypi/api-dock/$VERSION/json; the main JSON page is cached for a few minutes. conda-forge follows through its bot's pull request on conda-forge/api_dock-feedstock, which a maintainer merges after checking the recipe's dependencies match pyproject.toml.

Documentation

• The README is a short overview; details live in this wiki.
• In wiki pages, keep API Dock's [[table]] syntax inside backticks or code blocks: GitHub wikis turn bare double brackets into wiki links.