Repository navigation
feat(query): sandbox Python UDF workers and bound script UDF resources - #20830
Conversation
Python script UDF workers ran user code with the query node's environment, files, network and processes in reach. Harden and confine them: - Workers start with a cleared environment (allowlist only), no_new_privs and RLIMIT_CORE=0, and a private TMPDIR. - Before user code runs, Linux workers enter an unprivileged Landlock + seccomp sandbox: read-only Python/system libraries, read-write only the private directory, no sockets, no execve, no signals outside the worker, no ptrace/io_uring/bpf/namespaces/mounts. Failure to enter it is fatal. macOS workers run unconfined for local development and log a warning. - pip only accepts `name[extras]<specifiers>` requirements after `--`, runs with an environment allowlist and installs wheels only by default (`python_udf_pip_only_binary`). - Python aggregate UDFs, which still run in-process, are validated like scalar ones and rejected when `enable_udf_sandbox` is on. - JavaScript and WebAssembly UDFs get memory limits and JavaScript an execution timeout (`udf_script_memory_limit_mb`, `udf_script_execution_timeout_secs`).
github-actions
Bot
added
the
pr-feature
fork/vfork and clone without CLONE_THREAD are rejected, so user code cannot escape the per-worker process accounting or fork-bomb the node.
…on UDF sandbox The query binary runs jemalloc with background_thread:true, so workers had purge threads when entering the sandbox and refused to run. Stop them first; the worker process test now uses the query allocator to cover this.
⏱ CI timing · Devrun 40439332445 · 40.1 min (main p50 35.6 min, +4.5 min
|
| stage | this run | main p50 | Δ |
|---|---|---|---|
| total | 40.1 min | 35.6 min | +4.5 min |
| build (aarch64, ci profile) | 26.1 min | 16.6 min | +9.5 min |
| ├ deps + crates (parallel) | 13.4 min | — | — |
| ├ databend-query codegen | 2.7 min | — | — |
| ├ databend-binaries link | 7.6 min | — | — |
| └ sccache hit rate | 94.5 % | — | — |
| test_unit | 19.1 min | 15.9 min | +3.2 min |
| mac_check | — | 13.2 min | — |
| tests after build (49 jobs) | 11.4 min | 12.7 min | −1.3 min |
| ├ queue p50 / max | 2.8 min / 6.4 min | 2.1 min / 6.6 min | +0.7 min |
| ├ sqllogic standalone (8 jobs, max) | 4.3 min | 4.1 min | +0.2 min |
| ├ sqllogic cluster (8 jobs, max) | 6.9 min | 6.7 min | +0.3 min |
| ├ sqllogic stage (2 jobs, max) | 4.7 min | 4.9 min | −0.2 min |
| ├ sqllogic other (10 jobs, max) | 6.2 min | 5.5 min | +0.7 min |
| ├ stateless (2 jobs, max) | 6.6 min | 6.3 min | +0.3 min |
| ├ stateful (4 jobs, max) | 4.8 min | 4.8 min | +0.0 min |
| ├ compat (6 jobs, max) | 5.3 min | 4.9 min | +0.4 min |
| ├ meta / logs / lineage / udf (5 jobs, max) | 4.5 min | 4.4 min | +0.0 min |
| ├ ee (2 jobs, max) | 1.4 min | 1.3 min | +0.1 min |
| ├ private_tasks (1 jobs, max) | 9.0 min | 8.8 min | +0.2 min |
| └ fuzz (1 jobs, max) | 1.1 min | 1.0 min | +0.1 min |
Minutes unless stated; durations exclude queue time unless stated; "max" = longest job in the group. Baseline = median of the last 10 successful runs (Dev: Main on main; Cloud: recent Cloud runs). Build breakdown has no baseline. Updated on each push.
Python UDAFs ran user code inside the query process. They now use the same query-owned worker pool, quotas and Landlock/seccomp sandbox as scalar Python UDFs: - Workers load create_state/accumulate/merge/finish for an aggregate and serve stateless requests; states stay in the query process. - accumulate_keys and merge_serialized send one batched request per block (distinct states + row-to-state index), applied in row order in the worker. - The initial state is created lazily so other Python runtimes of the query can still register while the pipeline is built. - Assignments replace AggrState::write on existing states, so replaced states are dropped instead of leaked.
Partial and final aggregation build separate UDAF instances; each registered its own Python runtime and spent the per-query worker minimum, so a UDAF plus a scalar Python UDF exceeded python_udf_max_workers_per_query = 2. Stateless aggregate workers can be shared, so reuse the query's runtime for an identical aggregate function.
sundy-li
left a comment
sundy-li
left a comment
There was a problem hiding this comment.
AI-assisted author-side review of 78b5a17. I recommend addressing the two inline findings before merge; this is not a human approval.
Validation: cargo test -p databend-query-script-udf-support --features python-udf --locked passed all 37 unit tests and the custom worker-process regression suite with Python 3.12; targeted cargo clippy -p databend-query-script-udf-support --features python-udf --all-targets --locked -- -D warnings, git diff --check, and bash -n on both new shell tests passed. An independent, bounded Arrow 58.4.0 probe confirmed the state-buffer retention described inline. The JS finding is based on tracing the pinned dependency, not an adversarial reproduction. SQL service/cluster suites were not rerun locally.
The authenticated account is also the PR author, so this is a comment review rather than a formal approval/request-changes review. A responsible human still needs to read the full diff and complete the PR's unchecked acknowledgement before merge.
- JavaScript: arrow-udf-runtime arms its deadline only around the user function call, so argument/result conversion (struct getters, toJSON) and async continuations ran unbounded. LimitedJsRuntime owns the interrupt handler and arms one deadline per whole operation, also covering top-level module code during initialization. - Python UDAF: per-group states were 1-row slices of the worker response, so every state of a block retained the whole response buffer. Copy each group's row into its own buffers.
sundy-li
left a comment
sundy-li
left a comment
There was a problem hiding this comment.
AI-assisted follow-up review of 5da805c, focusing on the delta since the previous review.
The JS deadline finding is addressed: LimitedJsRuntime installs an independent handler and scopes the deadline across initialization and every scalar/aggregate engine operation, including output conversion. The new conversion regression passes.
The Python aggregate retention finding is only partially addressed: take copies ordinary Utf8/numeric buffers but still shares Utf8View payload buffers, and Utf8View is the production mapping for STRING state fields. One remaining P1 is detailed inline, so I do not recommend approval yet.
Validation with Python 3.12: all 39 crate unit tests and the custom python_worker_process suite passed; targeted clippy with --all-targets and -D warnings, git diff --check, and bash -n for both added shell tests passed. A bounded standalone probe against the exact pinned Arrow source confirms the remaining view-buffer sharing. The latest visible checks show 61 successful and 2 skipped checks. SQL service/cluster suites were not rerun locally. No repository files were changed.
This is an author-account comment review, not a formal independent approval or request-changes review.
|
I think the runtime-level changes in this PR would be better implemented in the In particular, the operation-wide JavaScript deadline belongs in Databend should retain the application-specific pieces: configuration and defaults, SQL validation, query admission and scheduling, pipeline lifecycle integration, and aggregate-state ownership. The worker launch interface would also need to accommodate host-specific initialization rather than depending directly on the Databend executable or allocator setup. Could we split the change along this boundary, implementing the reusable runtime/worker capabilities in the existing fork and keeping the Databend-side integration here? This would give those capabilities a clearer API and independent test coverage, while avoiding a substantial runtime implementation inside the query engine. |
…fork Per review, reusable execution-layer code now lives in datafuse-extras/arrow-udf (datafuse-extras/arrow-udf#3), pinned at ffd6f943: - arrow-udf-python-worker: worker launch, IPC protocol, process lifecycle, environment allowlist, pre-exec hardening and the Landlock/seccomp sandbox. Removes python_sandbox.rs and the protocol/process code from python_worker.rs. - arrow-udf-runtime: operation-wide JavaScript deadline and the set_memory_limit(None) fix. Removes LimitedJsRuntime; Databend sets the limits before adding functions. Databend keeps configuration, SQL validation, admission/quotas/pooling, pipeline lifecycle, aggregate-state ownership, dependency installation and the worker entrypoint, which stops jemalloc background threads through WorkerHooks::before_sandbox. Tests: the worker-process suite checks that the worker installs its own seccomp filter (containers already filter every process) and allows 5s for startup in the timeout scenario (debug binary exec alone takes ~1s).
|
@forsaken628 Thanks — split along that boundary. datafuse-extras/arrow-udf#3 (pinned here at
This PR keeps configuration and defaults, SQL validation, admission/quotas/pooling, pipeline lifecycle, aggregate-state ownership, dependency installation and the worker entrypoint ( Order: once arrow-udf#3 is merged I'll re-pin to its merge commit on the fork's |
83f98e3 is the merge commit on the fork's main; same tree as the previously pinned PR branch plus CI-only fixes.
|
datafuse-extras/arrow-udf#3 is merged; |
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Python script UDF workers run arbitrary user code. Before this PR they inherited the query node's environment (cloud credentials), could read its config/data files and
/proc/<query pid>/environ, connect to meta/storage/metadata endpoints, spawn programs, andpip installaccepted option-like requirements (--index-url=...,-r /path) and ran sdist build scripts on the node. JavaScript/WebAssembly UDFs were isolated but unbounded (awhile(true){}or a large allocation could hang or OOM the node).Python worker sandbox (
python_sandbox.rs)After the interpreter starts and before any user code runs, each Linux worker enters an unprivileged sandbox. There is no switch: if it cannot be entered the worker fails with
Python UDF sandbox is unavailable, refusing to run user code.sys.path, stdlib, site-packages), system libraries and a few device/proc entries; read-write only the worker's private directory (cwd,TMPDIR, deps, imports). On newer ABIs also TCP and signal/abstract-socket scoping.socket,execve,kill/tgkilloutside the worker,ptrace,process_vm_*,io_uring_*,bpf, namespaces (clonewithCLONE_NEW*,clone3→ ENOSYS), mounts, kernel modules, path-basedchmod/chown/xattr, x32 syscalls.Requirements: Linux 5.13+ with Landlock enabled, x86_64/aarch64, container seccomp profile permitting
landlock_*.Hardening
env_clear()+ allowlist,no_new_privs,RLIMIT_CORE=0.name[extras]<specifiers>, passed after--, allowlisted environment (keepsPIP_*/proxy/CA for mirrors),--only-binary=:all:by default (python_udf_pip_only_binary = true).enable_udf_python_scriptat create/bind/execute, rejected whenenable_udf_sandbox = true(that mode routes Python to an external sandbox with no aggregate API). WASM UDAF returns an error instead of panicking.Python UDAF in worker processes
Python aggregates no longer run in the query process. They use the same query-owned worker pool, quotas and sandbox as scalar Python UDFs. Workers are stateless: states stay in the query process (serialized for spill/exchange as before) and travel with each request.
accumulate_keysandmerge_serializedsend one batched request per block (distinct states + row-to-state index, applied in row order);merge_resultis still one request per group. The initial state is created lazily so later Python runtimes of the same query can still register during pipeline build.udf_script_memory_limit_mb = 1024(JS heap, WASM linear memory) andudf_script_execution_timeout_secs = 300(JS per call and top-level module code).Not covered (documented in
script_udf_support/README.md)Python worker memory/CPU limits, WASM execution timeout (needs
arrow-udf-runtimechange), query cancellation inside a JS/WASM call, batched finalization for Python UDAF,packages/importsfor Python UDAF.Tests
Unit Test
Logic Test
Benchmark Test
No Test - Explain why
python_worker_process: escape attempts (read secret file,/etc/passwd, parent environ, list/, write/unlink/chmod outside, TCP to a listening port, Unix socket, subprocess, signal parent,kill(-1)) all raisePermissionError; private temp files, threads, stdlib still work;Seccomp: 2,NoNewPrivs: 1, no env leak, pip injection rejected. Negative control: removing the sandbox call makes the test fail.seccomp program evaluated by a BPF interpreter unit test; requirement validator, JS timeout/memory, config mapping unit tests.
sqltype-check tests for UDAF language validation;system.configsgolden updated.python_worker_processUDAF cases: GROUP BY through batchedaccumulate_keyswith interleaved places and NULLs, single-state accumulate, serialize +merge_serializedwith a repeated target place,merge_states, empty state, scalar UDF registered after the UDAF, UDAF pid differs from the query process,open('/etc/passwd')in a UDAF raisesPermissionError.tests/suites/5_ee/08_udf_python/08_0004_hardening.sh,08_0005_udaf.sh.Manually: numpy + pandas import and compute inside the sandbox.
Type of change
Breaking for Python UDFs that use
subprocess, network, files outside their directory,multiprocessinglocks, sdist-only dependencies, or run on Linux without Landlock; and for Python UDAFs whenenable_udf_sandbox = true.AI assistance
src/query/script_udf_support, the UDF validator and config; the responsible human reviews the final diff.This change is