Improve secrets handling

Cargo.toml

@@ -60,6 +60,10 @@ chrono = "0.4"
 # stays on native-tls with no rustls/ring/aws-lc in the tree.
 reqwest = { version = "0.13", default-features = false, features = ["json", "native-tls", "multipart"] }
 
+# For redacting credential-bearing URLs before they reach a log or an error.
+# Already in the graph via reqwest; declared directly so redact.rs can use it.
+url = "2.5"
+
 # For df.http_multipart() — base64-decodes part payloads carried in the config JSON.
 base64 = "0.22"
 

USER_GUIDE.md

@@ -1042,6 +1042,10 @@ Durable function variables allow you to configure durable functions with externa
 
 > **Important**: `df.setvar()`, `df.unsetvar()`, and `df.clearvars()` cannot be called from within a running durable function. They are for configuration only.
 
+> **Variables are not secrets.** `df.vars` stores values as plaintext, and the value is copied
+> into durable execution history when `df.start()` runs. See
+> [Variables and secrets](#variables-and-secrets) before putting a credential in one.
+
 ### System Variables
 
 These read-only variables are automatically available during durable function execution:
@@ -1051,6 +1055,39 @@ These read-only variables are automatically available during durable function ex
 | `{sys_instance_id}` | Current durable function instance ID |
 | `{sys_label}` | Durable function label (if provided) |
 
+### Variables and secrets
+
+Using `df.setvar()` to hold an API key is a natural move, and it genuinely helps — but only in
+one place. Here is exactly what it does and does not protect, so you can make an informed
+choice.
+
+Given this pattern:
+
+```sql
+SELECT df.setvar('api_key', '<credential>');
+SELECT df.start(df.http('https://api.example.com/users', 'GET', NULL,
+                        '{"Authorization": "Bearer {api_key}"}'::jsonb), 'fetch-users');
+```
+
+| Location | Holds the credential? | Notes |
+|----------|----------------------|-------|
+| `df.nodes.query` | No — stores the literal `{api_key}` | This is the one thing `{var}` genuinely buys you. Substitution happens at execution time, not when the node is created. |
+| `df.vars.value` | **Yes, plaintext** | RLS isolates it from other users. It is *not* encrypted, and it is not hidden from a superuser or the table owner. |
+| Durable execution history | **Yes, plaintext** | `df.start()` snapshots every variable you own into the orchestration input, and the substituted header is recorded as the HTTP step's input. Parallel branches and each loop iteration re-record it. |
+| WAL, backups, replicas | **Yes** | Follows every write above, typically with a longer retention than `pg_durable.retention_days`. |
+| Server log | URLs are redacted; SQL text depends on `pg_durable.log_workflow_sql` | See [What reaches the server log](#what-reaches-the-server-log). |
+| `pg_stat_activity`, `pg_stat_statements` | Only if you inline the literal | `SELECT df.setvar('api_key', '<credential>')` is itself a statement. Bind it as a parameter, or use `\getenv` in psql, rather than typing the value into SQL. |
+
+Writing the credential directly into `df.http(...)` instead of using a variable is strictly
+worse: it adds `df.nodes.query` to that list without removing anything from it.
+
+Practical guidance:
+
+- Prefer credentials that are short-lived and narrowly scoped, so history exposure is bounded.
+- Treat `df.vars` as configuration — hostnames, table names, batch sizes, API versions.
+- Treat a database backup as containing every credential any workflow has used.
+- Set `pg_durable.retention_days` to the shortest value your operations allow.
+
 ### Variable Substitution
 
 > **Security note**: All `{...}` substitutions — including `{varname}`, `{sys_label}`, and `{sys_instance_id}` — perform **raw text substitution**. The value is inserted directly into the SQL string without escaping or parameterization. This is by design so that variables can hold SQL fragments like table names or expressions. Since you control both the variable value and the query template, and SQL executes under your own role, this is safe for configuration values you set yourself. Do **not** store untrusted external input in variables that get substituted into SQL. For passing query *results* between steps, use `$name` (via `|=>`), which applies proper SQL escaping.
@@ -1060,11 +1097,11 @@ Use `{varname}` in SQL queries to substitute variable values:
 ```sql
 -- Set up configuration
 SELECT df.setvar('api_base', 'https://api.example.com');
-SELECT df.setvar('api_key', 'secret123');
+SELECT df.setvar('page_size', '50');
 
 -- Start durable function using variables
 SELECT df.start(
-    df.http('{api_base}/users', 'GET', NULL, '{"Authorization": "Bearer {api_key}"}'::jsonb)
+    df.http('{api_base}/users?per_page={page_size}', 'GET')
     ~> 'INSERT INTO playground.logs (msg) VALUES (''Fetched users'')',
     'fetch-users'
 );
@@ -2060,10 +2097,38 @@ Row-level security (RLS) restricts each user to their own instances and nodes:
 - `df.cancel()` and `df.signal()` check ownership before acting — attempts on other users' instances return "Instance not found or access denied"
 - Superusers bypass RLS and can see all instances (standard PostgreSQL behavior)
 
+### What reaches the server log
+
+The background worker writes an execution trace to the PostgreSQL server log. That log is a
+different security boundary from the database: RLS does not apply to it, it is not bound by
+`pg_durable.retention_days`, and it is often shipped and backed up separately.
+
+| Trace | Contents |
+|-------|----------|
+| HTTP and multipart requests | Method, scheme, host, port and path. **Query-string values, userinfo and the URL fragment are redacted**, so an Azure SAS token or `?api-key=` does not reach the log. Parameter *names* are kept so the line stays diagnosable. |
+| HTTP request errors | Same redaction, applied to the message text as well, since the HTTP client interpolates the request URL into its own errors. |
+| HTTP and multipart request headers and bodies | Never logged. |
+| SQL nodes | The submitting role and target database, plus the fully-substituted SQL text when `pg_durable.log_workflow_sql` is on. |
+| Workflow result | The final return value is logged on completion. For a workflow ending in an HTTP step this includes the response body. |
+
+`pg_durable.log_workflow_sql` (default `on`) is what gates the SQL text. SQL cannot be
+redacted heuristically — a variable value spliced into a query is indistinguishable from the
+query itself — so this is an on/off switch rather than a masking rule. It is read by the
+background worker, so like the other worker GUCs it is Postmaster-context and needs a restart:
+
+```ini
+# postgresql.conf — omit workflow SQL text from the log
+pg_durable.log_workflow_sql = off
+```
+
+Turning it off also removes the primary record of what workflows actually executed, which is the
+first thing an incident investigation looks for. Leave it on unless the server log is less well
+protected than the database, and prefer keeping credentials out of SQL in the first place.
+
 ### Security Best Practices
 
 1. **Worker role must be superuser** — The background worker role (`pg_durable.worker_role`) must be a superuser to bypass RLS and manage all instances
-2. **Review df.vars usage** — Variables are scoped per-user via RLS, but avoid storing secrets in plain text
+2. **Do not put credentials in `df.vars`** — Variables are scoped per-user via RLS, but they are stored as plaintext and are copied into durable execution history. See [Variables and secrets](#variables-and-secrets)
 3. **Use labels carefully** — Instance labels are visible only to the submitting user (RLS-filtered) and superusers
 4. **Monitor instances** — Superusers can use `df.list_instances()` to see all users' instances; regular users see only their own
 5. **Avoid unsafe `SECURITY DEFINER` wrappers around `df.start()`** — Never allow untrusted callers to supply SQL or futures to `df.start()` from a `SECURITY DEFINER` context unless definer-level execution is intentional.
@@ -2166,7 +2231,7 @@ GRANT pg_durable_user TO app_backend, etl_service;
 
 Users get `SELECT` and `INSERT` on `df.instances` and `df.nodes` (required for `df.start()`, `df.status()`, `df.result()`). Column-level `UPDATE` on `(status, updated_at)` allows `df.cancel()` to set status. No full `UPDATE` or `DELETE` — the identity column (`submitted_by`) and structural columns are protected.
 
-> **Note:** `df.vars` uses per-user scoping via an `owner` column and RLS — each user can only read and write their own variables. Superusers bypass RLS but the DSL functions (`df.setvar()`, `df.getvar()`, etc.) still scope to the calling user via explicit filters. Avoid storing secrets in plain text.
+> **Note:** `df.vars` uses per-user scoping via an `owner` column and RLS — each user can only read and write their own variables. Superusers bypass RLS but the DSL functions (`df.setvar()`, `df.getvar()`, etc.) still scope to the calling user via explicit filters. Values are stored as plaintext and are copied into durable execution history — see [Variables and secrets](#variables-and-secrets).
 
 ### Revoking Privileges
 
@@ -2247,7 +2312,7 @@ pg_durable.max_new_transaction_starts = 2
 pg_durable.new_transaction_start_timeout = 5
 ```
 
-> **Other GUCs:** `pg_durable.list_instances_max_limit` (SUSET context, default `1000`) caps the per-call page size of `df.list_instances()`. Unlike the connection-limit GUCs above, it is superuser-settable at runtime (no restart) and is not loaded from `postgresql.conf` at startup only. See [docs/api-reference.md](docs/api-reference.md#pg_durablelist_instances_max_limit).
+> **Other GUCs:** `pg_durable.list_instances_max_limit` (SUSET context, default `1000`) caps the per-call page size of `df.list_instances()`. Unlike the connection-limit GUCs above, it is superuser-settable at runtime (no restart) and is not loaded from `postgresql.conf` at startup only. See [docs/api-reference.md](docs/api-reference.md#pg_durablelist_instances_max_limit). `pg_durable.log_workflow_sql` (Postmaster context, default `on`) controls whether workflow SQL text is written to the server log — see [What reaches the server log](#what-reaches-the-server-log).
 
 ### Connection Budget Formula
 

docs/api-reference.md

@@ -588,6 +588,11 @@ Sets a workflow variable for the current user (before `df.start()`). Each user h
 SELECT df.setvar('api_url', 'https://api.example.com');
 ```
 
+> **Not for credentials.** Values are stored as plaintext in `df.vars`, and `df.start()` copies
+> every variable you own into durable execution history. Using `{varname}` keeps the value out of
+> `df.nodes.query` but not out of history. See
+> [Variables and secrets](../USER_GUIDE.md#variables-and-secrets).
+
 ---
 
 ### df.getvar(name)
@@ -684,7 +689,7 @@ SELECT df.revoke_usage('app_role');
 
 ## Server Configuration (GUCs)
 
-These settings are configured via `ALTER SYSTEM SET` or `postgresql.conf` and take effect after `SELECT pg_reload_conf()` (no restart required).
+These settings are configured via `ALTER SYSTEM SET` or `postgresql.conf`. Each one lists its context: `SUSET` settings take effect after `SELECT pg_reload_conf()`, while `POSTMASTER` settings require a restart. Every GUC read by the background worker is `POSTMASTER`, because the worker does not process a configuration reload.
 
 ---
 
@@ -749,3 +754,28 @@ SELECT pg_reload_conf();
 ```
 
 > **Behavior change (v0.2.4):** prior to v0.2.4, `df.list_instances()` silently truncated `limit_count` to 10000. It now raises an error when `limit_count` exceeds this GUC (default 1000). Callers that previously requested very large pages should lower `limit_count` or use the paginated overload (`after_cursor`/`next_cursor`).
+
+---
+
+### pg_durable.log_workflow_sql
+
+Controls whether the background worker writes the SQL text of an executed workflow node to the PostgreSQL server log.
+
+| Property | Value |
+|----------|-------|
+| Type | `boolean` |
+| Default | `on` |
+| Context | `POSTMASTER` (set in `postgresql.conf`; requires restart) |
+
+The SQL is logged *after* variable substitution, so a credential held in a `df.vars` variable and spliced into a query reaches the server log in cleartext. Unlike a URL query string — which pg_durable redacts unconditionally — SQL cannot be masked heuristically, because a substituted value is indistinguishable from the surrounding statement. This GUC is therefore an on/off switch.
+
+```ini
+# postgresql.conf
+pg_durable.log_workflow_sql = off
+```
+
+The value is read by the background worker, which does not process a configuration reload, so a restart is required — the same as `pg_durable.retention_days` and the connection-limit GUCs.
+
+Turning it off keeps the submitting role and target database in the log but drops the statement text. That also removes the primary record of what workflows executed, which is usually the first thing an incident investigation looks for — prefer keeping credentials out of SQL over disabling the log. See [Variables and secrets](../USER_GUIDE.md#variables-and-secrets).
+
+This trace is written by the background worker's own logging, not by PostgreSQL statement logging, so `log_statement` neither enables nor suppresses it.

docs/http-security.md

@@ -166,22 +166,25 @@ REVOKE EXECUTE ON FUNCTION df.http(text, text, text, jsonb, integer) FROM PUBLIC
 ```
 
 When `df.grant_usage(role, include_http => false)` is called and the role still
-has effective HTTP access via the PUBLIC grant (or another inherited grant), a
-`WARNING` is emitted to signal that the revocation had no net effect.
+has effective HTTP access via the PUBLIC grant (or another inherited grant),
+that grant is left intact — `df.grant_usage()` is purely additive and never
+issues a `REVOKE`. Call `df.revoke_usage()` first to downgrade a role.
 
 ### 3.4 Admin function protection
 
 `df.grant_usage()` and `df.revoke_usage()` are admin-only functions.
 `EXECUTE` is revoked from `PUBLIC` at `CREATE EXTENSION` time, so only
-superusers can call them.
-
-> **Caution:** `df.grant_usage()` internally runs
-> `GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA df`, which temporarily includes
-> `df.grant_usage()` and `df.revoke_usage()` themselves before the function
-> immediately revokes them from the target role.  If an admin replicates the
-> blanket `GRANT` manually without the matching `REVOKE`s, the target role
-> will gain access to these admin helpers.  Always use `df.grant_usage()`
-> rather than hand-crafting the equivalent `GRANT` statements.
+superusers can call them — plus any role an admin delegated to with
+`df.grant_usage(role, with_grant => true)`.
+
+`df.grant_usage()` issues a specific set of grants: `USAGE ON SCHEMA df`,
+column-scoped table privileges, `EXECUTE` on `df.http()` / `df.http_multipart()`
+when `include_http => true`, and `EXECUTE` on `df.grant_usage()`,
+`df.revoke_usage()` and `df.metrics()` when `with_grant => true`. Those five
+functions are the only ones with `PUBLIC` `EXECUTE` revoked at install; every
+other `df.*` function keeps PostgreSQL's default, so **schema `USAGE` is the
+real access gate**. Granting `USAGE ON SCHEMA df` by hand therefore exposes the
+whole DSL surface at once; prefer `df.grant_usage()`.
 
 ### 3.5 Feature-flag interaction
 
@@ -315,12 +318,47 @@ Every HTTP attempt (allowed or blocked) is logged via `ctx.trace_info` with:
 
 - `submitted_by` — the role that called `df.start()` at the time the node was
   created (captured as `current_user` in the DSL and stored in `FunctionNode`)
-- `url` — the requested URL
+- `url` — the requested URL, **redacted** (see below)
 - Block reason tag — `(scheme)`, `(allowlist)`, or `(ip)` in the log prefix
 
 Resolved IP addresses are **not** included in error messages or logs to avoid
 leaking internal network topology to potentially malicious users.
 
+### 7.1 URL redaction
+
+A URL is a credential carrier: an Azure SAS token lives entirely in the query
+string, and `?api-key=` / `?code=` are common elsewhere. The server log is a
+weaker boundary than the database — no RLS, no `pg_durable.retention_days`, and
+frequently a separate shipping and backup path — so `crate::redact::redact_url`
+is applied before any URL reaches a log line or an error string.
+
+| Component | Treatment |
+|-----------|-----------|
+| Scheme, host, port, path | Preserved. The URL is reparsed, so a logged line may be normalized (host lowercased, default port dropped) relative to what the workflow supplied. |
+| Query parameter *names* | Preserved — `sig` and `code` are not themselves secret, and keeping them makes a redacted line diagnosable |
+| Query parameter *values* | Replaced with `<redacted>`, except an allowlist of benign parameters (`api-version`, `apiversion`, `comp`, `restype`) |
+| Bare query token with no `=` | Replaced whole — indistinguishable from a name |
+| `userinfo@` | Replaced with `<redacted>@`, keeping the host |
+| Fragment | Replaced whole |
+| Unparseable input | Replaced whole — redaction fails closed and never echoes back a string it could not parse |
+
+Parsing uses the `url` crate, so IPv6 authorities, percent-encoding, default
+ports and userinfo follow the spec rather than ad-hoc string splitting.
+
+The same redaction is applied to the HTTP client's own error text, which
+interpolates the request URL into messages such as
+`error sending request for url (...)`. This matters because activity errors are
+persisted to `df.nodes.error` and recorded in durable execution history, not
+just written to the log.
+
+Request headers and bodies are never logged.
+
+> **Not covered:** the response body. A workflow's final result is traced on
+> completion, and the full response envelope is stored in `df.nodes.result`. A
+> request whose *response* is a credential — reading a secret from Key Vault, or
+> an OAuth token endpoint — still persists that value. Response-side redaction
+> is tracked separately.
+
 ---
 
 ## 8. Error Messages

scripts/test-e2e-local.sh

@@ -268,21 +268,41 @@ DATA_DIR="$PGRX_HOME/data-$PG_VERSION"
 LOG_FILE="$PGRX_HOME/$PG_VERSION.log"
 CONF_FILE="$DATA_DIR/postgresql.conf"
 
-shopt -s nullglob
-PGRX_CANDIDATES=("$PGRX_HOME"/"$PG_VERSION".*/pgrx-install/bin)
-shopt -u nullglob
-if [ "${#PGRX_CANDIDATES[@]}" -eq 0 ]; then
+# config.toml is authoritative: it is what cargo pgrx builds the extension
+# against, and it may point at an externally installed PostgreSQL (pgenv, a
+# distro package) that has no directory under ~/.pgrx. Fall back to the
+# pgrx-built layout when the entry is missing.
+PGRX_BIN=""
+if [ -f "$PGRX_HOME/config.toml" ]; then
+    PG_CONFIG_PATH=$(grep -E "^pg${PG_VERSION}[[:space:]]*=[[:space:]]*\"" "$PGRX_HOME/config.toml" | head -1 | cut -d'"' -f2)
+    if [ -n "$PG_CONFIG_PATH" ]; then
+        PGRX_BIN="$(dirname "$PG_CONFIG_PATH")"
+    fi
+fi
+
+if [ -z "$PGRX_BIN" ]; then
+    shopt -s nullglob
+    PGRX_CANDIDATES=("$PGRX_HOME"/"$PG_VERSION".*/pgrx-install/bin)
+    shopt -u nullglob
+    if [ "${#PGRX_CANDIDATES[@]}" -gt 0 ]; then
+        PGRX_BIN="${PGRX_CANDIDATES[0]}"
+    fi
+fi
+
+if [ -z "$PGRX_BIN" ] || [ ! -x "$PGRX_BIN/pg_ctl" ]; then
     echo "Error: pgrx PostgreSQL $PG_VERSION not installed"
     echo "Run: cargo pgrx init"
     exit 1
 fi
 
-PGRX_BIN="${PGRX_CANDIDATES[0]}"
 PSQL="$PGRX_BIN/psql"
 PG_CTL="$PGRX_BIN/pg_ctl"
 PG_ISREADY="$PGRX_BIN/pg_isready"
 PG_CONFIG="$PGRX_BIN/pg_config"
 
+# Keep a developer ~/.psqlrc out of test output, so local runs match CI.
+export PSQLRC=/dev/null
+
 stop_server() {
     if [ -d "$DATA_DIR" ] && "$PG_CTL" status -D "$DATA_DIR" >/dev/null 2>&1; then
         echo -e "${YELLOW}Stopping PostgreSQL...${NC}"