Repository navigation
feat: operation-wide JS deadlines and a sandboxed Python worker crate - #3
Merged
Merged
Conversation
arrow-udf-runtime (JavaScript): - `set_timeout` now bounds whole operations instead of only the user function call: argument conversion, async continuations and Arrow conversion of results (getters, toJSON) were unbounded. Operations are call, each table-function chunk, create_state, accumulate, accumulate_or_retract, merge and finish; top-level module code evaluated by add_function/add_aggregate is bounded too. The deadline is armed under the runtime lock and cleared on drop, including cancellation. - `set_memory_limit(None)` set a QuickJS limit of 0 bytes instead of removing the limit. arrow-udf-python-worker (new): - Out-of-process execution of Python Arrow UDFs: `WorkerProcess` starts a host-provided worker binary (`WorkerCommand`) in its own process group with a cleared environment, no_new_privs and no core dumps, exchanges Arrow IPC frames over a private Unix socket, and kills/reaps the group on drop, keeping a host `keep_alive` until the process exited. - `run_worker` serves one scalar or aggregate `WorkerFunction`; aggregate requests carry their states and batch many groups per request. - Linux workers enter a Landlock + seccomp sandbox before user code runs (read-only Python/system files, read-write working directory, no sockets, execve, new processes, foreign signals, ptrace, io_uring, namespaces or mounts). `WorkerHooks::before_sandbox` lets hosts stop their own threads first. macOS runs workers unconfined for local development.
Author
|
CI note:
Neither job builds code changed here. Happy to fix them in a separate PR if wanted. |
sundy-li
added a commit
to sundy-li/databend
that referenced
this pull request
Oct 26, 2026
…fork Per review, reusable execution-layer code now lives in datafuse-extras/arrow-udf (datafuse-extras/arrow-udf#3), pinned at ffd6f943: - arrow-udf-python-worker: worker launch, IPC protocol, process lifecycle, environment allowlist, pre-exec hardening and the Landlock/seccomp sandbox. Removes python_sandbox.rs and the protocol/process code from python_worker.rs. - arrow-udf-runtime: operation-wide JavaScript deadline and the set_memory_limit(None) fix. Removes LimitedJsRuntime; Databend sets the limits before adding functions. Databend keeps configuration, SQL validation, admission/quotas/pooling, pipeline lifecycle, aggregate-state ownership, dependency installation and the worker entrypoint, which stops jemalloc background threads through WorkerHooks::before_sandbox. Tests: the worker-process suite checks that the worker installs its own seccomp filter (containers already filter every process) and allows 5s for startup in the timeout scenario (debug binary exec alone takes ~1s).
Merged
5 of 12 tasks
Both jobs float (Cargo.lock is not committed) and broke without code changes since main's last CI run: - docs: current nightlies reject rquickjs-core 0.6.2 (never-type fallback). Pin nightly-2025-12-11 until rquickjs is upgraded. - test-duckdb: duckdb 1.20000 moved to arrow 60 while arrow-udf uses arrow 58, so the example mixed two RecordBatch types. Pin the last release line on arrow 58 (~1.10506).
The extension test installs the latest duckdb from PyPI; duckdb 2.0 removed relation methods (fetchone) the sqllogictest runner uses. Test with the release matching the pinned duckdb crate.
Author
|
CI drift fixed in
|
sundy-li
added a commit
to sundy-li/databend
that referenced
this pull request
Oct 27, 2026
83f98e3 is the merge commit on the fork's main; same tree as the previously pinned PR branch plus CI-only fixes.
TCeason
pushed a commit
to TCeason/datafuse
that referenced
this pull request
Oct 28, 2026
databendlabs#20830) * feat(query): sandbox Python UDF workers and bound script UDF resources Python script UDF workers ran user code with the query node's environment, files, network and processes in reach. Harden and confine them: - Workers start with a cleared environment (allowlist only), no_new_privs and RLIMIT_CORE=0, and a private TMPDIR. - Before user code runs, Linux workers enter an unprivileged Landlock + seccomp sandbox: read-only Python/system libraries, read-write only the private directory, no sockets, no execve, no signals outside the worker, no ptrace/io_uring/bpf/namespaces/mounts. Failure to enter it is fatal. macOS workers run unconfined for local development and log a warning. - pip only accepts `name[extras]<specifiers>` requirements after `--`, runs with an environment allowlist and installs wheels only by default (`python_udf_pip_only_binary`). - Python aggregate UDFs, which still run in-process, are validated like scalar ones and rejected when `enable_udf_sandbox` is on. - JavaScript and WebAssembly UDFs get memory limits and JavaScript an execution timeout (`udf_script_memory_limit_mb`, `udf_script_execution_timeout_secs`). * feat(query): deny process creation in the Python UDF sandbox fork/vfork and clone without CLONE_THREAD are rejected, so user code cannot escape the per-worker process accounting or fork-bomb the node. * fix(query): stop jemalloc background threads before entering the Python UDF sandbox The query binary runs jemalloc with background_thread:true, so workers had purge threads when entering the sandbox and refused to run. Stop them first; the worker process test now uses the query allocator to cover this. * feat(query): run Python aggregate UDFs in sandboxed worker processes Python UDAFs ran user code inside the query process. They now use the same query-owned worker pool, quotas and Landlock/seccomp sandbox as scalar Python UDFs: - Workers load create_state/accumulate/merge/finish for an aggregate and serve stateless requests; states stay in the query process. - accumulate_keys and merge_serialized send one batched request per block (distinct states + row-to-state index), applied in row order in the worker. - The initial state is created lazily so other Python runtimes of the query can still register while the pipeline is built. - Assignments replace AggrState::write on existing states, so replaced states are dropped instead of leaked. * fix(query): share one Python runtime per aggregate UDF within a query Partial and final aggregation build separate UDAF instances; each registered its own Python runtime and spent the per-query worker minimum, so a UDAF plus a scalar Python UDF exceeded python_udf_max_workers_per_query = 2. Stateless aggregate workers can be shared, so reuse the query's runtime for an identical aggregate function. * fix(query): bound whole JS UDF operations and compact Python UDAF states - JavaScript: arrow-udf-runtime arms its deadline only around the user function call, so argument/result conversion (struct getters, toJSON) and async continuations ran unbounded. LimitedJsRuntime owns the interrupt handler and arms one deadline per whole operation, also covering top-level module code during initialization. - Python UDAF: per-group states were 1-row slices of the worker response, so every state of a block retained the whole response buffer. Copy each group's row into its own buffers. * refactor(query): move Python worker and JS limits into the arrow-udf fork Per review, reusable execution-layer code now lives in datafuse-extras/arrow-udf (datafuse-extras/arrow-udf#3), pinned at ffd6f943: - arrow-udf-python-worker: worker launch, IPC protocol, process lifecycle, environment allowlist, pre-exec hardening and the Landlock/seccomp sandbox. Removes python_sandbox.rs and the protocol/process code from python_worker.rs. - arrow-udf-runtime: operation-wide JavaScript deadline and the set_memory_limit(None) fix. Removes LimitedJsRuntime; Databend sets the limits before adding functions. Databend keeps configuration, SQL validation, admission/quotas/pooling, pipeline lifecycle, aggregate-state ownership, dependency installation and the worker entrypoint, which stops jemalloc background threads through WorkerHooks::before_sandbox. Tests: the worker-process suite checks that the worker installs its own seccomp filter (containers already filter every process) and allows 5s for startup in the timeout scenario (debug binary exec alone takes ~1s). * chore(query): pin arrow-udf fork to merged datafuse-extras/arrow-udf#3 83f98e3 is the merge commit on the fork's main; same tree as the previously pinned PR branch plus CI-only fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the reusable execution-layer pieces of databendlabs/databend#20830 into this fork, as requested in review (comment). Databend keeps configuration, SQL validation, admission/scheduling, pipeline integration and aggregate-state ownership.
arrow-udf-runtime: operation-wide JavaScript deadline
set_timeoutarmed the deadline only around the user function call. Argument conversion, continuations of async functions, and Arrow conversion of results (which runs user getters andtoJSON) ran unbounded. Now one deadline spans each operation (call, each table-function chunk,create_state,accumulate,accumulate_or_retract,merge,finish), and top-level module code is bounded whileadd_function/add_aggregateevaluate it. The deadline is armed under the runtime lock and cleared on drop (also on cancellation).Also fixes
set_memory_limit(None), which set a QuickJS limit of 0 bytes (no allocation possible) instead of removing the limit.Behavior change: the timeout is per operation (batch) rather than per row call.
arrow-udf-python-worker (new crate, unix)
WorkerProcess::spawn(command, function, working_dir, keep_alive, reaper): starts a host-provided worker binary in its own process group with a cleared environment (allowlist),no_new_privs, no core dumps, cwd/TMPDIR=working_dir; Arrow IPC over a private Unix socket; drop kills and reaps the group and dropskeep_alive(host quota, files) only after the process exited.run_worker(socket, WorkerHooks): worker entrypoint serving one scalar or aggregateWorkerFunction. Aggregate requests carry states and batch many groups per request, so workers are stateless. Workers exit when the host disconnects.execve, new processes, foreign signals, ptrace, io_uring, bpf, namespaces, mounts, kernel modules, path-based chmod/chown/xattr, x32). Entered after interpreter init and before user code; must be single-threaded (WorkerHooks::before_sandboxlets hosts stop their threads); failure is fatal. macOS runs unconfined (SANDBOX_ENFORCED = false).Tests
arrow-udf-runtime/tests/javascript.rs: result conversion (getter), async continuation, aggregate state conversion, module evaluation, per-operation reset,set_memory_limit(None). The four timeout tests hang on the previous implementation.arrow-udf-python-worker: unit tests (protocol, group updates, seccomp program via a BPF interpreter) andtests/worker.rs(the test binary is its own worker): scalar, prelude, init and request errors, aggregate create/accumulate/merge/finish, environment, rlimit,no_new_privs, own seccomp filter, kill/reap withkeep_alive, sandbox escape attempts (Linux), and refusal to run a multi-threaded worker. Removing the sandbox call makes the suite fail.cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features -D warnings.AI assistance
Drafted by an AI coding agent; responsible human: @sundy-li, who reviews the full diff.