Add changes for compatibility with WASM components and collocated UDF servers

[kesmit13]

This PR makes several changes to allow the singlestoredb work in WASM environments. Many of these changes benefit standard installations as well such as lazy loading of numpy, pandas, polars, and pyarrow. Others move imports that are only needed in certain environments, but not within WASM.

A new collocated UDF server implementation is also included that uses a high-performance loop in the C extension to parse and call Python functions on each row. This function is used both by standard collocated servers as well as WASM-based UDF handlers.

---

Note

High Risk
High risk due to substantial additions in the C extension (new rowdat_1 type handling, mmap/socket syscalls, and call_function_accel) plus a new Unix-socket UDF server that supports dynamic registration via exec(). Failures here can impact UDF correctness/performance and introduces new security/compatibility considerations (FD passing, process forking, WASI stubs).

Overview
Adds a new plugin-mode UDF execution path: a Unix-socket server (functions/ext/plugin) with a CLI entry point (python-udf-server / python -m singlestoredb.functions.ext.plugin) supporting thread or pre-fork process pools, control signals (@@health, @@functions, @@register), and shared registry reloading.

Extends the native accelerator (accel.c) with a combined call_function_accel hot path (parse ROWDAT_1 → call Python → serialize ROWDAT_1), plus fd-level helpers (mmap_read, mmap_write, recv_exact) and broader ROWDAT_1 support for DECIMAL, DATE, TIME, and DATETIME/TIMESTAMP (also mirrored in Python rowdat_1.py and JSON encoding).

Improves WASM/optional-dependency compatibility by lazy-loading numeric/DF backends (switching to get_numpy_type_map/get_polars_type_map/get_pyarrow_type_map and get_numpy()), making jwt an on-demand import, and generalizing internal result normalization in Connection._iquery across pandas/polars/arrow/numpy and iterable outputs. Documentation is updated to describe the new plugin server mode and its env/CLI options.

^{Reviewed by Cursor Bugbot for commit 25c023e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR introduces WASM-compatibility improvements (primarily by lazy-loading heavyweight optional dependencies and moving environment-specific imports into call sites) and adds a new collocated Python UDF server implementation, including a new C-extension hot path to accelerate rowdat_1 decode → Python call → rowdat_1 encode.

Changes:

• Added a WIT interface definition and a WASM build helper script for external UDF component workflows.
• Refactored optional dependency handling (numpy/pandas/polars/pyarrow, IPython, JWT) to be more robust in constrained/WASM-like environments.
• Added a new collocated UDF server (socket + mmap protocol, thread/process modes, dynamic registration) and a C-extension accelerator entry point (call_function_accel).

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
wit/udf.wit	Defines the external UDF WIT interface and exported world.
singlestoredb/utils/_lazy_import.py	Adds cached lazy imports for heavy optional deps.
singlestoredb/utils/dtypes.py	Converts dtype maps to lazily-evaluated, cached getters.
singlestoredb/utils/results.py	Switches result formatting to lazy imports + cached type maps.
singlestoredb/utils/events.py	Broadens IPython import failure handling.
singlestoredb/converters.py	Uses lazy numpy import in vector converters.
singlestoredb/connection.py	Adjusts internal result-to-dict conversion to avoid importing pandas.
singlestoredb/mysql/connection.py	Adds WASM-friendly DEFAULT_USER detection (handles OSError).
singlestoredb/auth.py	Moves jwt import into call site.
singlestoredb/management/utils.py	Moves jwt import into call sites for WASM-friendliness.
singlestoredb/management/manager.py	Moves jwt import into is_jwt call site.
singlestoredb/functions/dtypes.py	Updates exports to use dtype-map getter functions.
singlestoredb/functions/ext/rowdat_1.py	Replaces eager dtype maps with lazy getter functions.
singlestoredb/functions/ext/json.py	Replaces eager dtype maps with lazy getter functions.
singlestoredb/functions/ext/collocated/*	Adds collocated server, protocol handling, registry, control signals, and WASM adapter.
singlestoredb/tests/test_connection.py	Makes pandas string dtype assertions version-tolerant.
resources/build_wasm.sh	Adds a build helper for wasm32-wasip2 wheels.
pyproject.toml	Adds python-udf-server CLI entry point.
accel.c	Adds call_function_accel C hot path and exports it from the extension module.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 11 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 10 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Same issue as _load(): _load_vectors() attempts decimal.Decimal(...) even when is_null is true and the encoded length is 0, which will raise decimal.InvalidOperation. Guard the decimal parsing on not is_null (or slen > 0) so NULL decimal columns can be loaded.

Suggested change

	val = decimal.Decimal(
	data_io.read(slen).decode('utf-8'),
	)
	sval = data_io.read(slen).decode('utf-8')
	val = decimal.Decimal(sval) if not is_null else sval

[Copilot]

datetime.time is normalized to 'time' here, but ROWDAT_1 TIME serialization/deserialization uses datetime.timedelta (see rowdat_1._pack_time/_unpack_time). A UDF annotated with datetime.time (or returning it) will be serialized with _pack_time() and crash because datetime.time has no total_seconds(). Either convert datetime.time to a timedelta during dump/call paths, or keep datetime.time unsupported and raise a clear error.

Suggested change

	return 'time'
	raise TypeError(
	'unsupported type annotation: datetime.time; '
	'use datetime.timedelta for TIME values, or use `args`/`returns` '
	'on the @udf/@tvf decorator to specify the data type',
	)

[Copilot]

VECTOR() validates element_type.upper() but then emits the SQL using the original element_type string. If callers pass a lowercase value (e.g. "f32"), validation passes but the SQL becomes VECTOR(n, f32) which is likely invalid. Normalize element_type = element_type.upper() before formatting the SQL string.

Suggested change

	if element_type.upper() not in (F16, F32, F64, I8, I16, I32, I64):
	element_type = element_type.upper()
	if element_type not in (F16, F32, F64, I8, I16, I32, I64):

[Copilot]

_bind_socket() unconditionally unlinks any existing path. If sock_path points to a regular file, this will delete it. Consider verifying the existing path is actually a Unix domain socket (e.g., via stat.S_ISSOCK(os.stat(...).st_mode)) before unlinking, and raise a clear error otherwise.

[Copilot]

PyDict_SetItem(py_objs, PyLong_FromUnsignedLongLong(u64), py_dec) leaks the temporary key object because PyDict_SetItem does not steal references. Create the key PyObject* separately, pass it to PyDict_SetItem, then Py_DECREF it. This matters on the hot path because it will leak one Python int per non-null DECIMAL/DATE/TIME/DATETIME cell.

[Copilot]

string_types includes 246 (MySQL NEWDECIMAL), which will cause decimal columns to be treated as strings (and binary_types will include -246). This will mis-parse/mis-dump DECIMAL/NEWDECIMAL data; remove 246 from string_types and rely on decimal_types for DECIMAL handling.

Suggested change

	string_types = set([15, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254])
	string_types = set([15, 245, 247, 248, 249, 250, 251, 252, 253, 254])

[Copilot]

Decimal decoding happens even when the value is NULL. For NULL decimals _dump() writes length=0, but _load() always calls decimal.Decimal(''), which raises decimal.InvalidOperation. Skip parsing when is_null (or when slen == 0), and set val to a safe placeholder so NULLs round-trip correctly.

Suggested change

	val = decimal.Decimal(
	data_io.read(slen).decode('utf-8'),
	)
	dec_bytes = data_io.read(slen)
	if is_null or slen == 0:
	val = None
	else:
	val = decimal.Decimal(dec_bytes.decode('utf-8'))

[Copilot]

New/expanded dtype support for decimal.Decimal (and now datetime.time) isn’t covered by existing tests. Add unit tests (e.g., in singlestoredb/tests/test_udf.py::test_datetimes / a new decimal test) to assert the generated SQL/type mapping for decimal.Decimal and to define/validate expected behavior for datetime.time.

[Copilot]

Handshake reads the function name using a single recvmsg(namelen, ...) call. On a stream socket, recvmsg is allowed to return fewer than namelen bytes, which would truncate function_name and desync the protocol. Validate len(msg) == namelen (and/or handle MSG_TRUNC) and if short, read the remaining bytes (without expecting additional FDs) before decoding.

Suggested change

	if flags & getattr(socket, 'MSG_TRUNC', 0):
	logger.warning('Function name data was truncated (MSG_TRUNC)')
	return
	if len(msg) < namelen:
	remaining = _recv_exact_py(conn, namelen - len(msg))
	if remaining is None:
	logger.warning(
	'Connection closed while receiving function name',
	)
	return
	msg += remaining

[Copilot]

Dynamic function registration executes generated code inside the __main__ module. This can clobber the server’s own __main__ namespace and cause collisions between registrations/reloads. Use a dedicated/private module name for generated functions (e.g. singlestoredb.functions.ext.plugin._dynamic_udfs_<counter>), or create a fresh module per registration and keep it referenced as needed.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

These PyDict_SetItem calls pass a freshly-created PyLong key (PyLong_FromUnsignedLongLong(...)) without DECREF'ing it afterward. Since PyDict_SetItem increments references to both key and value, the temporary key object leaks here (and similarly for other inserted objects). Store the key in a local PyObject*, call PyDict_SetItem, then Py_DECREF(key), or use PyDict_SetItemString with a stable key strategy if possible.

[Copilot]

Handshake reads the function name using a single recvmsg(namelen, ...), but recvmsg is not guaranteed to return all requested bytes. If it returns a partial name, msg.decode() will succeed with truncated data and the protocol will desync. Treat short reads as an error (e.g., verify len(msg) == namelen and abort/close), or loop until the full name is received (while ensuring SCM_RIGHTS fds are only accepted once).

Suggested change

	if len(msg) != namelen:
	logger.warning(
	f'Expected function name of {namelen} bytes, got {len(msg)}',
	)
	return

[Copilot]

_bind_socket unconditionally unlinks any existing path. If the socket path is user-configurable (often under /tmp), this can be exploited to delete arbitrary files (e.g., via symlink or replacing the path with a regular file) when the server starts. Before unlinking, validate that the existing path is a Unix domain socket (and ideally owned by the current user), or use a safer strategy like binding to a uniquely generated path and refusing to overwrite non-sockets.

[Copilot]

create_function executes generated code into the 'main' module when it exists (which it typically will for any running process). This can pollute/overwrite the server's own main globals and makes dynamic registrations interfere with unrelated code in the process. Use a dedicated module namespace for dynamic UDFs (e.g., a unique module name per registration batch or a private module like 'singlestoredb.functions.ext.plugin._dynamic'), and avoid reusing the process' real main module.

Suggested change

	name = '__main__'
	compiled = compile(full_code, f'<{name}>', 'exec')
	if name in sys.modules:
	module = sys.modules[name]
	else:
	module = types.ModuleType(name)
	module.__file__ = f'<{name}>'
	sys.modules[name] = module
	name = (
	'singlestoredb.functions.ext.plugin._dynamic.'
	f'{func_name}_{os.urandom(8).hex()}'
	)
	compiled = compile(full_code, f'<{name}>', 'exec')
	module = types.ModuleType(name)
	module.__file__ = f'<{name}>'
	module.__package__ = 'singlestoredb.functions.ext.plugin._dynamic'
	sys.modules[name] = module

[Copilot]

The lazy import helpers only catch ImportError. In constrained environments (e.g., WASM/WASI, missing shared library loaders, binary wheels built for the wrong platform), importing optional deps often raises OSError (or other exceptions) instead. To preserve the intended "optional" behavior, broaden the exception handling (at least ImportError + OSError) and consider storing/logging the exception for debugging when a module is present but fails to import.

Suggested change

	from functools import lru_cache
	from typing import Any
	from typing import Optional
	@lru_cache(maxsize=None)
	def get_numpy() -> Optional[Any]:
	"""Return numpy module or None if not installed."""
	try:
	return importlib.import_module('numpy')
	except ImportError:
	return None
	@lru_cache(maxsize=None)
	def get_pandas() -> Optional[Any]:
	"""Return pandas module or None if not installed."""
	try:
	return importlib.import_module('pandas')
	except ImportError:
	return None
	@lru_cache(maxsize=None)
	def get_polars() -> Optional[Any]:
	"""Return polars module or None if not installed."""
	try:
	return importlib.import_module('polars')
	except ImportError:
	return None
	@lru_cache(maxsize=None)
	def get_pyarrow() -> Optional[Any]:
	"""Return pyarrow module or None if not installed."""
	try:
	return importlib.import_module('pyarrow')
	except ImportError:
	return None
	import logging
	from functools import lru_cache
	from typing import Any
	from typing import Optional
	logger = logging.getLogger(__name__)
	def _import_optional_module(name: str) -> Optional[Any]:
	"""Return imported optional module or None if it cannot be imported."""
	try:
	return importlib.import_module(name)
	except (ImportError, OSError) as exc:
	logger.debug('Optional dependency %r could not be imported: %s', name, exc)
	return None
	@lru_cache(maxsize=None)
	def get_numpy() -> Optional[Any]:
	"""Return numpy module or None if not installed."""
	return _import_optional_module('numpy')
	@lru_cache(maxsize=None)
	def get_pandas() -> Optional[Any]:
	"""Return pandas module or None if not installed."""
	return _import_optional_module('pandas')
	@lru_cache(maxsize=None)
	def get_polars() -> Optional[Any]:
	"""Return polars module or None if not installed."""
	return _import_optional_module('polars')
	@lru_cache(maxsize=None)
	def get_pyarrow() -> Optional[Any]:
	"""Return pyarrow module or None if not installed."""
	return _import_optional_module('pyarrow')

[Copilot]

_pack_time uses td.total_seconds() (float) to derive microseconds, which can lose precision for large timedeltas and round-trip incorrectly. Prefer integer arithmetic based on td.days / td.seconds / td.microseconds to compute total microseconds exactly (and keep sign handling consistent).

[cursor]

WASM build script exits on startup

Medium Severity

The script uses set -eou pipefail, which parses -o incorrectly and treats u as an option name. Bash exits before running the build steps, so resources/build_wasm.sh cannot execute its WASM packaging workflow.

 

^{Reviewed by Cursor Bugbot for commit 5f84e84. Configure here.}

[cursor]

Missing bounds check on decimal payload length

High Severity

The new DECIMAL decoding paths consume i64 bytes from data without validating remaining buffer size. Truncated or malformed rowdat_1 input can advance past the end of the byte buffer, causing out-of-bounds reads and potential crashes in load_rowdat_1 and load_rowdat_1_numpy.

Additional Locations (1)

• accel.c#L2673-L2681

 

^{Reviewed by Cursor Bugbot for commit 5f84e84. Configure here.}

[cursor]

Handshake assumes single recvmsg reads full name

Medium Severity

The handshake reads the function name with one conn.recvmsg(...) and treats any short read as protocol failure. On SOCK_STREAM, partial reads are valid, so legitimate requests can be rejected intermittently when the name payload arrives in multiple segments.

 

^{Reviewed by Cursor Bugbot for commit 5f84e84. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit aeab5ff. Configure here.}