diff --git a/docs/architecture/comparison.md b/docs/architecture/comparison.md index fde587d5..901b98f6 100644 --- a/docs/architecture/comparison.md +++ b/docs/architecture/comparison.md @@ -15,7 +15,7 @@ PgDog aims to be the de facto PostgreSQL proxy and pooler. Below is a feature co | [Read/write separation](../features/load-balancer/index.md) | No | Basic support | Advanced support handling edge cases | | [Failover](../features/load-balancer/healthchecks.md) | No | :material-check-circle-outline: | :material-check-circle-outline: | | [Health checks](../features/load-balancer/healthchecks.md) | No | :material-check-circle-outline: | :material-check-circle-outline: | -| [Authentication](../features/authentication.md) | :material-check-circle-outline: | `md5`, `plain` | `scram-sha-256`, `md5`, `plain` | +| [Authentication](../features/auth/index.md) | :material-check-circle-outline: | `md5`, `plain` | `scram-sha-256`, `md5`, `plain` | | [Metrics](../features/metrics.md) | Admin database only | OpenMetrics & admin database | OpenMetrics & admin database | | [Mirroring](../features/mirroring.md) | No | Partial support | :material-check-circle-outline: | | TLS | :material-check-circle-outline: | :material-check-circle-outline: | :material-check-circle-outline: | diff --git a/docs/configuration/pgdog.toml/general.md b/docs/configuration/pgdog.toml/general.md index 6f63c31c..07bd4e10 100644 --- a/docs/configuration/pgdog.toml/general.md +++ b/docs/configuration/pgdog.toml/general.md @@ -11,6 +11,7 @@ General settings are relevant to the operations of the pooler itself, or apply t The IP address of the local network interface PgDog will bind to listen for connections. !!! note "Requires restart" + This setting cannot be changed at runtime. Default: **`0.0.0.0`** (all interfaces) @@ -22,6 +23,7 @@ The TCP port PgDog will bind to listen for connections. Default: **`6432`** !!! note "Requires restart" + This setting cannot be changed at runtime. ### `listen_backlog` @@ -29,6 +31,7 @@ Default: **`6432`** Maximum number of pending client connections waiting to be accepted on the TCP socket. Increase this to accommodate many clients reconnecting at once. On Linux, the effective value is capped by `net.core.somaxconn`, so increase that limit as well. !!! note "Requires restart" + This setting cannot be changed at runtime. Default: **`1_024`** @@ -41,6 +44,7 @@ virtual CPU. The value `0` means to spawn no threads and use the current thread Default: **`2`** !!! note "Requires restart" + This setting cannot be changed at runtime. ### `background_workers` @@ -48,6 +52,7 @@ Default: **`2`** Maximum number of background threads used to offload blocking or CPU-intensive tasks, like SCRAM authentication. Set to `0` (default) to run this on the runtime workers instead. !!! note "Requires restart" + The background thread limit is set at startup. Default: **`0`** (offloading disabled) @@ -57,6 +62,7 @@ Default: **`0`** (offloading disabled) Default maximum number of server connections per database pool. The pooler will not open more than this many PostgreSQL database connections when serving clients. !!! note "Recommendation" + It's recommended to keep this value below the supported connections of the backend database(s) to allow connections for maintenance in high load scenarios. Default: **`10`** @@ -364,6 +370,7 @@ Default: **none** (disabled) IP address of the local interface on which the OpenMetrics HTTP endpoint listens. The endpoint is enabled by setting [`openmetrics_port`](#openmetrics_port). !!! note "Requires restart" + This setting cannot be changed at runtime. Default: **`0.0.0.0`** (all interfaces) @@ -384,7 +391,7 @@ Default: **none** ### `auth_type` -What kind of [authentication](../../features/authentication.md) mechanism to use for client connections. +What kind of [authentication](../../features/auth/index.md) mechanism to use for client connections. Currently supported: @@ -398,7 +405,7 @@ Default: **`scram`** ### `passthrough_auth` -Toggle automatic creation of connection pools given the user name, database and password. See [passthrough authentication](../../features/authentication.md#passthrough-authentication). +Toggle automatic creation of connection pools given the user name, database and password. See [passthrough authentication](../../features/auth/password.md#passthrough-authentication). Available options are: @@ -408,6 +415,18 @@ Available options are: Default: **`disabled`** +### `passthrough_auth_debounce_delay` + +How long, in milliseconds, to reuse a passthrough authentication check result for the same credentials. + +Default: **`1_000`** (1s) + +### `auth_token_cache_size` + +Maximum number of entries in the client token passthrough authentication cache. + +Default: **`1_000`** + ## Prepared statements ### `prepared_statements` @@ -512,6 +531,7 @@ Default: **`true`** (enabled) Directory where the [two-phase commit](../../features/sharding/2pc/index.md) write-ahead log is stored. !!! note "Requires restart" + This setting cannot be changed at runtime. Default: **`./pgdog_wal`** @@ -543,6 +563,7 @@ Default: **`1_000`** ### `query_parser_enabled` !!! warning "Deprecated setting" + This setting is deprecated. Use [`query_parser`](#query_parser) instead. Force-enable query parsing to take advantage of its features in non-sharded databases, like [advisory locks](../../features/connection-pooler/transaction-mode.md#advisory-locks) or managing [session state](../../features/connection-pooler/transaction-mode.md#session-state). @@ -625,6 +646,12 @@ How many parallel `COPY` workers to launch during [resharding](../../features/sh Default: **`1`** +### `resharding_parallel_within_table_copies` + +Number of parallel source reads for each table copy during resharding. Increasing this can speed up copies of TOAST-heavy tables. + +Default: **`1`** + ### `resharding_copy_retry_max_attempts` Maximum number of retries for a failed table copy during resharding (per-table). Retries use exponential backoff starting at [`resharding_copy_retry_min_delay`](#resharding_copy_retry_min_delay), doubling each attempt and capped at 32×. @@ -652,6 +679,7 @@ Default: **`1_000`** (1s) ### `reload_schema_on_ddl` !!! warning + This setting requires [PgDog Enterprise Edition](../../enterprise_edition/index.md) to work as expected. If using the open source edition, it will only work with single-node PgDog deployments, e.g., in local development or CI. @@ -773,6 +801,12 @@ Number of identical log messages allowed within [`log_dedup_window`](#log_dedup_ Default: **`0`** (disabled) +### `query_log` + +Path to a file where all queries are logged. Logging every query is slow; avoid using this in production. + +Default: **none** (disabled) + ### `query_log_stdout` Log client SQL at `INFO` level. Query text is limited by [`log_query_sample_length`](#log_query_sample_length), and control characters are sanitized to keep each entry on one line. @@ -800,6 +834,7 @@ Default: **`1_000`** characters Maximum size, in bytes, of a query message (`Query` or `Parse`) received from a client. When a message exceeds this size, the action taken depends on [`query_size_limit_action`](#query_size_limit_action). Other protocol messages (e.g. `Bind`, `CopyData`) are not affected. !!! note + This setting is useful for protecting the query parser from very large SQL texts that could cause excessive CPU or memory usage. Default: **none** (disabled) diff --git a/docs/features/auth/rds-iam.md b/docs/features/auth/rds-iam.md index 6ed19e0f..fa023bf0 100644 --- a/docs/features/auth/rds-iam.md +++ b/docs/features/auth/rds-iam.md @@ -81,6 +81,74 @@ For each user in `users.toml`, you can specify its IAM role (and optionally IAM In order for this to work correctly, make sure the IAM role used to deploy PgDog has the correct Trust Policy to assume all roles specified in the configuration. +## IAM passthrough + +!!! note "Experimental feature" + + This feature is new and experimental. Please make sure to test it before deploying to production. + +PgDog can authenticate applications using RDS IAM authentication directly against databases in RDS. This allows apps to _not_ use password auth to connect to PgDog, while maintaining its own connection pool (also using IAM) to the database. + +### How it works + +Applications can use the RDS SDK to generate temporary tokens to connect to RDS. PgDog can attempt a connection to RDS, and if successful, mark that token as valid for its duration, allowing clients to connect. + +To make sure this doesn't cause database connection storms (defeating the purpose of a connection pooler), PgDog caches tokens it receives from clients for 15 minutes. If an application uses temporary tokens correctly, i.e., by caching them for their validity period, this mechanism can work well at scale. + +### Configuration + +RDS IAM passthrough auth can be configured in `users.toml`, for example: + +=== "pgdog.toml" + + ```toml + [[users]] + name = "pgdog" + database = "prod" + auth_type = "external_token" + server_auth = "rds_iam" + ``` + +=== "Helm chart" + + ```yaml + users: + - name: pgdog + database: prod + authType: external_token + serverAuth: rds_iam + ``` + +When IAM passthrough is enabled, there are no passwords anywhere in the stack: apps, PgDog and Postgres use temporary credentials. + +Additionally, you can configure the size of the token cache in `pgdog.toml`: + +=== "pgdog.toml" + + ```toml + [general] + auth_token_cache_size = 1_000 + ``` + +=== "Helm chart" + + ```yaml + authTokenCacheSize: 1000 + ``` + +### Monitoring + +For this feature to work well, it's important for the token cache in PgDog to have a high hit rate. Otherwise, it would have to create connections to Postgres almost every time a new client connects to the pooler. + +The cache metrics are exported via [OpenMetrics](../metrics.md) and [OTEL](../metrics.md#otel): + +| Metric | Description | +| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| `token_cache_entries` | Number of tokens in the cache. | +| `token_cache_evictions` | Number of tokens evicted from the cache. If this is high, the cache is too small or applications are not using IAM authentication correctly. | +| `token_cache_hits` | Number of times the token an application provided was found in the cache. If this is high, the cache is performing well. | +| `token_cache_misses` | Number of times PgDog had to connect to RDS to validate a token. | + ## Read more {{ next_steps_links([ diff --git a/docs/features/index.md b/docs/features/index.md index 7a9a8dba..c9cbdf67 100644 --- a/docs/features/index.md +++ b/docs/features/index.md @@ -19,7 +19,7 @@ All features are configurable to fit your environment and can be toggled on/off. | [Sharding](sharding/index.md) | Query routing, data migration and schema management to scale PostgreSQL horizontally. | | [Prepared statements](connection-pooler/prepared-statements.md) | Support for Postgres named prepared statements in transaction mode. | | [Plugins](plugins/index.md) | Pluggable libraries to add functionality to PgDog at runtime, without recompiling code. | -| [Authentication](authentication.md) | Support for various PostgreSQL user authentication mechanisms, like SCRAM. | +| [Authentication](auth/index.md) | Support for various PostgreSQL user authentication mechanisms, like SCRAM. | | [Session mode](connection-pooler/session-mode.md) | Compatibility mode with direct PostgreSQL connections. | | [Metrics](metrics.md) | Real time reporting, including Prometheus/OpenMetrics and an admin database. | | [Mirroring](mirroring.md) | Copy queries from one database to another in the background. | diff --git a/docs/features/plugins/index.md b/docs/features/plugins/index.md index af5da925..c9c93121 100644 --- a/docs/features/plugins/index.md +++ b/docs/features/plugins/index.md @@ -1,14 +1,15 @@ --- icon: material/power-plug --- + # Plugins overview PgDog comes with a powerful plugin system that allows you to customize the query routing behavior. Plugins are written in Rust, compiled into shared libraries, and loaded at runtime. - ## Getting started #### Rust compiler + Our plugins use the newest features of the Rust compiler. Before proceeding, make sure to update yours to the latest version: ``` @@ -38,7 +39,7 @@ pgdog-plugin = "0.2.0" # make sure to use the version compatible with your PgDog This turns the crate into a shared library, exposing its functions using the C ABI, which PgDog will call at runtime. !!! note - The `pgdog-plugin` crate is published on [crates.io](https://crates.io/crates/pgdog-plugin) and is fully documented. You can find our [Rust docs here](https://docsrs.pgdog.dev), including all dependencies like [`pg_query`](https://docsrs.pgdog.dev/pg_query/index.html). +The `pgdog-plugin` crate is published on [crates.io](https://crates.io/crates/pgdog-plugin) and is fully documented. You can find our [Rust docs here](https://docsrs.pgdog.dev), including all dependencies like [`pg_query`](https://docsrs.pgdog.dev/pg_query/index.html). ### Writing plugins @@ -64,7 +65,6 @@ This ensures the following requirements are followed: See [Safety](#safety) section for more info. - ## Functions ### `init` @@ -73,7 +73,6 @@ This function is executed once at startup, when PgDog loads the plugin. It allow kind of internal plugin state. Execution of this function is synchronized, so it's safe to include any thread-unsafe functions or initialize synchronization primitives, like mutexes. - This function has the following signature: ```rust @@ -83,7 +82,6 @@ fn init() { } ``` - ### `route` This function is called every time the query router processes a query and needs to figure out @@ -119,7 +117,6 @@ use pgdog_plugin::pg_query; use pgdog_plugin::prelude::*; ``` - #### Outputs The plugin is expected to return a [`Route`](https://docsrs.pgdog.dev/pgdog_plugin/context/struct.Route.html). It can pass the following information back to PgDog: @@ -130,8 +127,6 @@ The plugin is expected to return a [`Route`](https://docsrs.pgdog.dev/pgdog_plug All of these are optional. If you don't return any of these, the plugin doesn't influence the routing decision at all and can be used for logging queries, or some other purpose. - - ### `fini` This function is called before PgDog is shut down. It allows plugins to perform any cleanup tasks, like saving @@ -154,17 +149,21 @@ Plugins need to be compiled and placed into a folder on your machine where PgDog 2. Export the plugin's parent directory into the `LD_LIBRARY_PATH` environment variable, provided to PgDog at runtime 3. Pass the absolute (or relative) path to the plugin in [`pgdog.toml`](../../configuration/pgdog.toml/plugins.md) -!!! note +!!! note "Performance" + Make sure to compile plugins in release mode for good performance: `cargo build --release`. The plugin's shared library will be in the `target/release/` folder of your Cargo project, e.g., `target/release/libmy_plugin.so`. You then need to specify which plugins you'd like PgDog to load at runtime: === "pgdog.toml" + ```toml [[plugins]] name = "my_plugin" ``` + === "Helm chart" + ```yaml plugins: - name: my_plugin @@ -173,11 +172,14 @@ You then need to specify which plugins you'd like PgDog to load at runtime: This can be the name of the library (without the `lib` prefix or the `.so`/`.dylib` extension) or a relative/absolute path to the shared library, for example: === "pgdog.toml" + ```toml [[plugins]] name = "/usr/lib/libmy_plugin.so" ``` + === "Helm chart" + ```yaml plugins: - name: /usr/lib/libmy_plugin.so @@ -206,7 +208,7 @@ Whatever Rust compiler version is used to build PgDog itself needs to be used to PgDog provides the compiler version used to build it at startup: ``` -INFO pgdog: 🐕 PgDog v0.1.29 [main@ff3fe3e, pgdog-plugin 0.2.0, rustc 1.93.0 (254b59607 2026-01-19)] +INFO: 🐕 PgDog Enterprise v0.1.61-v2026-10-09-1459 [main@7fca29d, pgdog-plugin 0.5.0, rustc 1.96.0 (ac68faa20 2026-05-25)] ``` #### `pgdog-plugin` compatibility @@ -216,5 +218,5 @@ To ensure your plugin works correctly with PgDog, the `pgdog-plugin` version use You can verify the `pgdog-plugin` version of your plugin by checking the PgDog startup logs: ``` -INFO pgdog: 🐕 PgDog v0.1.29 [main@ff3fe3e, pgdog-plugin 0.2.0, rustc 1.93.0 (254b59607 2026-01-19)] +INFO: 🐕 PgDog Enterprise v0.1.61-v2026-10-09-1459 [main@7fca29d, pgdog-plugin 0.5.0, rustc 1.96.0 (ac68faa20 2026-05-25)] ``` diff --git a/docs/index.md b/docs/index.md index e7e57ed7..fb55c489 100644 --- a/docs/index.md +++ b/docs/index.md @@ -14,7 +14,7 @@ PgDog is an open source project. You can download its code from our [repository] Every commit in the `main` branch, and weekly tagged releases, have corresponding images in our [Docker](https://github.com/orgs/pgdogdev/packages/container/package/pgdog) repository, for example: ```bash -docker run ghcr.io/pgdogdev/pgdog:v0.1.48 +docker run ghcr.io/pgdogdev/pgdog:{{ open_source_tag }} ``` You can read more about how to deploy PgDog [here](installation.md). diff --git a/docs/installation.md b/docs/installation.md index ca7066f1..82e1e416 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -22,10 +22,10 @@ docker run ghcr.io/pgdogdev/pgdog:main ### SemVer -PgDog follows SemVer, and for each tagged release, a corresponding tag will be available in the Docker repository. For example, you can run `v0.1.44` like so: +PgDog follows SemVer, and for each tagged release, a corresponding tag will be available in the Docker repository. For example, you can run `{{ open_source_tag }}` like so: ``` -docker run ghcr.io/pgdogdev/pgdog:v0.1.44 +docker run ghcr.io/pgdogdev/pgdog:{{ open_source_tag }} ``` ### AWS ECS @@ -155,7 +155,7 @@ Most configuration options have sensible defaults. This makes single-database co #### [users.toml](configuration/users.toml/users.md) -This config file contains a mapping between databases, users, and passwords. Unless you configured [passthrough authentication](features/authentication.md#passthrough-authentication), users not specified in this file will not be able to connect: +This config file contains a mapping between databases, users, and passwords. Unless you configured [passthrough authentication](features/auth/password.md#passthrough-authentication), users not specified in this file will not be able to connect: === "users.toml" ```toml diff --git a/docs/migrating-to-pgdog/from-pgbouncer.md b/docs/migrating-to-pgdog/from-pgbouncer.md index 2dc756a6..aff526af 100644 --- a/docs/migrating-to-pgdog/from-pgbouncer.md +++ b/docs/migrating-to-pgdog/from-pgbouncer.md @@ -188,7 +188,7 @@ Settings that control how clients and server connections authenticate. | PgBouncer | PgDog | Notes | |-|-|-| -| [`auth_type`](https://www.pgbouncer.org/config.html#auth_type) | [`auth_type`](../configuration/pgdog.toml/general.md#auth_type) | PgDog supports only a subset of [authentication](../features/authentication.md) mechanisms. +| [`auth_type`](https://www.pgbouncer.org/config.html#auth_type) | [`auth_type`](../configuration/pgdog.toml/general.md#auth_type) | PgDog supports only a subset of [authentication](../features/auth/index.md) mechanisms. | [`auth_file`](https://www.pgbouncer.org/config.html#auth_file) | N/A | The path to `users.toml` can be passed in as a CLI argument on startup: `--users `. | | [`auth_hba_file`](https://www.pgbouncer.org/config.html#auth_hba_file) | N/A | HBA authentication is not currently supported. | | [`auth_ident_file`](https://www.pgbouncer.org/config.html#auth_ident_file) | N/A | Same as above. | diff --git a/docs/migrating-to-pgdog/index.md b/docs/migrating-to-pgdog/index.md index f19b2589..b15ed5bc 100644 --- a/docs/migrating-to-pgdog/index.md +++ b/docs/migrating-to-pgdog/index.md @@ -8,7 +8,7 @@ PgDog attempts to make the migration from other connection poolers as smooth as ## Configuration -PgDog uses the **TOML** language for its [configuration](../configuration/index.md) files. If you're coming from PgBouncer, you'll need to rewrite your configs. We separate user [authentication](../features/authentication.md) (usernames, passwords) from the main settings, so you'll still be able to encrypt passwords in production. +PgDog uses the **TOML** language for its [configuration](../configuration/index.md) files. If you're coming from PgBouncer, you'll need to rewrite your configs. We separate user [authentication](../features/auth/index.md) (usernames, passwords) from the main settings, so you'll still be able to encrypt passwords in production. See [migrating from PgBouncer](from-pgbouncer.md) for more info. diff --git a/docs/roadmap.md b/docs/roadmap.md index 28dabdd1..99101a0f 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -31,7 +31,7 @@ These features are required for PgDog to act as a replacement for PgBouncer and/ | [Prepared statements](features/connection-pooler/prepared-statements.md) | :material-check-circle-outline: | | | [Metrics](features/metrics.md) | :material-check-circle-outline: | Admin database views contain more columns than PgBouncer. | | [Encryption](features/tls.md) | :material-check-circle-outline: | | -| [Authentication](features/authentication.md) | :material-wrench: | Password authentication only. `scram-sha-256`, `md5` are supported. | +| [Authentication](features/auth/index.md) | :material-wrench: | Password authentication only. `scram-sha-256`, `md5` are supported. | ## Sharding diff --git a/main.py b/main.py index 3ede452b..f6192c3e 100644 --- a/main.py +++ b/main.py @@ -7,14 +7,20 @@ log = logging.getLogger("mkdocs.plugins.macros") +# Released tag for the open source Docker image. Update this in one place; +# reference it in docs with {{ open_source_tag }}. Can be overridden at build +# time with the OPEN_SOURCE_TAG environment variable. +OPEN_SOURCE_TAG = os.environ.get("OPEN_SOURCE_TAG", "v0.1.61") + # Latest released tag for the Enterprise Docker images. Update this in one # place; reference it in docs with {{ enterprise_tag }}. Can be overridden at # build time with the ENTERPRISE_TAG environment variable. -ENTERPRISE_TAG = os.environ.get("ENTERPRISE_TAG", "v2026-09-03-2023") +ENTERPRISE_TAG = os.environ.get("ENTERPRISE_TAG", "v2026-10-09-1459") def define_env(env): + env.variables["open_source_tag"] = OPEN_SOURCE_TAG env.variables["enterprise_tag"] = ENTERPRISE_TAG def _validate_link(href, page):