Skip to content

feat(query): add admin Python UDF network exception - #20874

Closed
sundy-li wants to merge 2 commits into
databendlabs:mainfrom
sundy-li:feat/python-udf-network-policy
Closed

sundy-li wants to merge 2 commits into
databendlabs:mainfrom
sundy-li:feat/python-udf-network-policy

Conversation

@sundy-li

@sundy-li sundy-li commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Add [query].python_udf_allow_network = false (CLI: --python-udf-allow-network=true) as an administrator-only compatibility exception for existing network-dependent Python script UDFs following feat(query): sandbox Python UDF workers and bound script UDF resources #20830. It is not a SQL setting or UDF option, and does not change enable_udf_sandbox routing.
  • Wire the node policy through scalar/aggregate worker pools and pin arrow-udf to cc08c3e538f8347545540020dc289cc498c7d3e5. Dependency PR: feat(python-worker): add host-controlled outbound network policy datafuse-extras/arrow-udf#4 (must land before this PR).
  • Keep the Linux sandbox mandatory. The exception permits IPv4/IPv6 stream/datagram clients and resolver/public CA reads, while retaining filesystem/process isolation and denying Unix/raw sockets and bind/listen. Enabling logs a warning. It applies to every Python UDF on the node, is not a destination allowlist, and also exposes internal/loopback destinations subject to deployment egress policy.
  • Rollout: enable only on a dedicated warehouse, consistently across all nodes, before upgrading existing network-dependent customers. Restart nodes to enable/revoke; existing UDF definitions follow the policy without recreation. Track owner/expiry operationally; give advance notice and at least one month for migration before revoking an existing customer's exception, honoring any longer promised window.
  • Lock in existing error semantics: uncaught Python/network errors fail SQL rather than silently inserting NULL. Deliberate Python NULL or caught-error JSON remains a normal result; ingestion must validate actual events/business status.
  • Tests passed: 36 UDF unit tests and the real Python worker process suite; 2 configuration tests; test_configs_table_basic golden; affected config/UDF/service crates cargo clippy --tests -- -D warnings; query binary build; cargo fmt --all --check, diff and shell syntax checks. New regression: 08_0006_network_policy.sh verifies denied INSERT leaves no row, deliberate NULL/error JSON, and absence from SQL settings.
  • Isolated SQL/HTTPS validation passed: default denial, existing definition works after enabling/restart, DNS + verified TLS, requests.get() returns JSON events, ingestion stores actual events, and restarting with the exception disabled revokes access. Temporary services were stopped. Full workspace tests were not run.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: An AI coding agent drafted the network policy integration, dependency changes, regression tests, documentation and PR description, and ran local validation; the responsible human reviewed the complete diff before submission.
  • Responsible human: @sundy-li
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

⏱ CI timing · Dev

run 40843837115 · attempt 2 (wall time includes re-run gap) · 44.0 min (main p50 43.3 min, +0.7 min)

stage this run main p50 Δ
total 44.0 min 43.3 min +0.7 min
build (aarch64, ci profile) 18.1 min 19.4 min −1.3 min
├ deps + crates (parallel) 5.0 min — —
├ databend-query codegen 3.1 min — —
├ databend-binaries link 7.8 min — —
└ sccache hit rate 97.4 % — —
test_unit 16.4 min 17.1 min −0.7 min
mac_check — 13.8 min —
tests after build (49 jobs) 25.3 min 13.4 min +12.0 min ⚠️
├ queue p50 / max 1.6 min / 17.8 min 2.4 min / 7.7 min −0.8 min
├ sqllogic standalone (8 jobs, max) 4.5 min 4.0 min +0.5 min
├ sqllogic cluster (8 jobs, max) 7.8 min 6.8 min +1.0 min
├ sqllogic stage (2 jobs, max) 5.1 min 5.1 min +0.0 min
├ sqllogic other (10 jobs, max) 5.9 min 5.6 min +0.3 min
├ stateless (2 jobs, max) 6.5 min 6.5 min +0.0 min
├ stateful (4 jobs, max) 5.2 min 5.0 min +0.3 min
├ compat (6 jobs, max) 5.7 min 5.7 min −0.0 min
├ meta / logs / lineage / udf (5 jobs, max) 4.5 min 4.5 min +0.0 min
├ ee (2 jobs, max) 1.6 min 1.4 min +0.2 min
├ private_tasks (1 jobs, max) 8.9 min 8.8 min +0.1 min
└ fuzz (1 jobs, max) 1.1 min 1.0 min +0.1 min

Minutes unless stated; durations exclude queue time unless stated; "max" = longest job in the group. Baseline = median of the last 10 successful runs (Dev: Main on main; Cloud: recent Cloud runs). Build breakdown has no baseline. Updated on each push.

@sundy-li
sundy-li marked this pull request as ready for review October 29, 2026 08:40
@sundy-li

Copy link
Copy Markdown
Member Author

Consolidated into #20898 at the author’s request. That branch already contains all prerequisite Databend changes and the final integration-bound HTTP/CONNECT implementation, with no python_udf_allow_network config/CLI or blanket worker override. #20898 targets main directly; this PR must not be merged separately. The worker transport remains in the separate dependency PR datafuse-extras/arrow-udf#5. No code changes or history rewrites were needed for this consolidation.

@sundy-li sundy-li closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant