Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 12 additions & 14 deletions PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,20 +136,18 @@ SDKs via the binding's `extensions` + `suppress_config_extensions` /
still loaded by the core directly. Parity restored across Python and
TypeScript; Rust stays file-path-only by design (epic #227 carve-out).

**Where the planning lives — temporary drift
([#1111](https://github.com/thekevinscott/dirsql/issues/1111)).** The decisions
inside that helper — does any entry name a package, what does each literal path
resolve to against its own config's directory, which installed file is *the*
loadable — are language-neutral, and since
[#1119](https://github.com/thekevinscott/dirsql/issues/1119) they live once in
`dirsql::extension_resolution` (`#[doc(hidden)]`, not a public Rust API).
Python reached through the binding in
[#1120](https://github.com/thekevinscott/dirsql/issues/1120)
(`dirsql._dirsql.plan_config_extensions` / `select_loadable` / `is_bare_name`);
TypeScript still carries its own port and is tracked by
[#1121](https://github.com/thekevinscott/dirsql/issues/1121). Each host keeps
only what it alone can do: reading the config files, and locating an installed
package (`importlib.util.find_spec` / `require.resolve`).
**Where the planning lives.** The decisions inside that helper — does any entry
name a package, what does each literal path resolve to against its own config's
directory, which installed file is *the* loadable — are language-neutral, and
since [#1119](https://github.com/thekevinscott/dirsql/issues/1119) they live
once in `dirsql::extension_resolution` (`#[doc(hidden)]`, not a public Rust
API). Both SDKs reach it through their binding —
[#1120](https://github.com/thekevinscott/dirsql/issues/1120) for Python
(`dirsql._dirsql.plan_config_extensions` / `select_loadable` / `is_bare_name`),
[#1121](https://github.com/thekevinscott/dirsql/issues/1121) for TypeScript
(the addon's `planConfigExtensions` / `selectLoadable` / `isBareName`). Each
host keeps only what it alone can do: reading the config files, and locating an
installed package (`importlib.util.find_spec` / `require.resolve`).

**Config-flag scanning breadth
([#754](https://github.com/thekevinscott/dirsql/issues/754) /
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
**Changed**

- **Extension resolution now plans in the Rust core.** `resolveConfigsExtensionSpecs` reads each config off disk and hands `{ path, contents }` pairs to the addon's `planConfigExtensions`; `resolvePackage` hands the located package directory and its files to `selectLoadable`; `resolveExtensionPath` classifies through `isBareName`. TypeScript keeps only the two host-specific halves — reading the files, and `require.resolve`. Six internal modules go away (`is-bare-name`, `has-bare-name`, `load-extension-entries`, `resolve-entries`, `platform-suffixes`, `resolve-config-extension-specs`) against two added (`read-config`, `plan-entry-path`), and with them the `smol-toml` dependency and the lazy-load guard it needed (#720). No change to the published API. (#1121)

**Fixed**

- The error raised when a config's `[[dirsql.extension]]` names an installed package with no loadable file now writes the searched suffixes as glob patterns — `no loadable extension file (*.so / *.node) found in package 'x' (searched /dir)` — matching the Python SDK. (#1121)
7 changes: 7 additions & 0 deletions packages/ts/e2e-attestations/claude-1121-ts-ext.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "pnpm test:e2e",
"ran_at": 1789683288,
"exit_code": 0,
"commit": "3cab5ee29ace70626e8042059bb065888302650b",
"branch": "claude/1121-ts-ext"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
### Missing-loadable error wording is now glob-shaped (#1121)

#### Summary

Extension resolution moved onto the Rust core's plan contract
(`dirsql::extension_resolution`), so the "no loadable extension file" error is
now rendered by the core rather than by the TypeScript port of it. The core
writes each searched suffix as a glob, which changes the message text.

#### Required changes

_None._ No exported symbol, argument, or config key changed.

#### Deprecations removed

_None._

#### Behavior changes without code changes

- A config `[[dirsql.extension]]` naming an installed package that ships no
loadable file rejects with `no loadable extension file (*.so / *.node) found
in package 'x' (searched /dir)`. It previously read `(.so / .node)`. Code
matching that string exactly must be updated; the Python SDK already emitted
the glob form, so the two now agree.
- `smol-toml` is no longer a runtime dependency. Nothing imported it from
outside the package, but a consumer relying on it being hoisted into
`node_modules` by `dirsql` must depend on it directly.

#### Verification

```bash
mkdir -p node_modules/empty-ext
echo '{"name":"empty-ext","version":"1.0.0"}' > node_modules/empty-ext/package.json
printf '[[dirsql.extension]]\npath = "empty-ext"\n' > .dirsql.toml
npx dirsql query "SELECT 1" --config .dirsql.toml
# -> no loadable extension file (*.so / *.node) found in package 'empty-ext' (searched …/node_modules/empty-ext)
```
184 changes: 183 additions & 1 deletion packages/ts/napi/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
//! the TS wrapper exposes a matching overloaded constructor so callers can
//! write either `new DirSQL(configPath)` or `new DirSQL({ root, tables, ... })`.

use dirsql::extension_resolution::{self, ConfigSource as CoreConfigSource, PlanEntry};
use dirsql::{
DirSQL as CoreDirSQL, Extension, PreparedBuild, RawFileEvent, Row, RowEvent as CoreRowEvent,
Table, Value,
Expand All @@ -27,7 +28,7 @@ use napi::bindgen_prelude::*;
use napi_derive::napi;
use std::cell::RefCell;
use std::collections::HashMap;
use std::path::PathBuf;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;

Expand All @@ -54,6 +55,106 @@ pub fn config_paths_from_argv(argv: Vec<String>) -> Vec<String> {
dirsql::launcher::config_paths_from_argv(&argv)
}

/// The loadable-file suffixes a Node host recognizes.
const NODE_SUFFIXES: &[&str] = &[".so", ".dylib", ".dll", ".node"];

/// The subset this platform actually builds, used when picking a package's
/// loadable out of everything installed beside it.
#[cfg(target_os = "macos")]
const PLATFORM_SUFFIXES: &[&str] = &[".dylib", ".node"];
#[cfg(target_os = "windows")]
const PLATFORM_SUFFIXES: &[&str] = &[".dll", ".node"];
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
const PLATFORM_SUFFIXES: &[&str] = &[".so", ".node"];

/// One config file for the extension planner: its absolute path, and its
/// contents or `null` when the launcher could not read it.
#[napi(object)]
pub struct ConfigSource {
pub path: String,
pub contents: Option<String>,
}

/// One planned `[[dirsql.extension]]` entry.
///
/// Exactly one of `path` and `package` is set: a `path` is ready to load, a
/// `package` must be located with `require.resolve` first — unless `shadow`
/// names an existing file, which takes precedence over the package.
#[napi(object, object_from_js = false)]
pub struct ExtensionPlanEntry {
pub path: Option<String>,
pub package: Option<String>,
pub shadow: Option<String>,
pub entrypoint: Option<String>,
}

fn plan_entry_to_js(entry: PlanEntry) -> ExtensionPlanEntry {
match entry {
PlanEntry::Literal { path, entrypoint } => ExtensionPlanEntry {
path: Some(display(&path)),
package: None,
shadow: None,
entrypoint,
},
PlanEntry::Package {
name,
shadow,
entrypoint,
} => ExtensionPlanEntry {
path: None,
package: Some(name),
shadow: Some(display(&shadow)),
entrypoint,
},
}
}

fn display(path: &Path) -> String {
path.to_string_lossy().into_owned()
}

/// Plan several configs' `[[dirsql.extension]]` entries.
///
/// `null` back means no entry names a package, so the core's own config
/// loading handles them all.
#[napi(js_name = "planConfigExtensions")]
pub fn plan_config_extensions(configs: Vec<ConfigSource>) -> Option<Vec<ExtensionPlanEntry>> {
let sources: Vec<CoreConfigSource<'_>> = configs
.iter()
.map(|source| CoreConfigSource {
path: Path::new(&source.path),
contents: source.contents.as_deref(),
})
.collect();
let plan = extension_resolution::plan_config_extensions(&sources, NODE_SUFFIXES)?;
Some(plan.into_iter().map(plan_entry_to_js).collect())
}

/// Pick package `name`'s single loadable file out of `candidates`.
#[napi(js_name = "selectLoadable")]
pub fn select_loadable(name: String, dirs: Vec<String>, candidates: Vec<String>) -> Result<String> {
selected_loadable(&name, dirs, candidates).map_err(Error::from_reason)
}

/// The JS-free half of [`select_loadable`], so it stays unit-testable.
fn selected_loadable(
name: &str,
dirs: Vec<String>,
candidates: Vec<String>,
) -> std::result::Result<String, String> {
let dirs: Vec<PathBuf> = dirs.into_iter().map(PathBuf::from).collect();
let candidates: Vec<PathBuf> = candidates.into_iter().map(PathBuf::from).collect();
extension_resolution::select_loadable(name, &dirs, &candidates, PLATFORM_SUFFIXES)
.map(|path| display(&path))
.map_err(|err| err.to_string())
}

/// Whether an extension entry names a package rather than a file.
#[napi(js_name = "isBareName")]
pub fn is_bare_name(path: String) -> bool {
extension_resolution::is_bare_name(&path, NODE_SUFFIXES)
}

/// A row-level event emitted by the file watcher.
///
/// `table` is nullable because error events may occur before a file has
Expand Down Expand Up @@ -764,6 +865,87 @@ fn row_event_to_js(event: &CoreRowEvent) -> Result<RowEvent> {
mod tests {
use super::*;

fn config(path: &str, contents: Option<&str>) -> ConfigSource {
ConfigSource {
path: path.to_string(),
contents: contents.map(str::to_string),
}
}

#[test]
fn plan_config_extensions_is_none_without_a_package_name() {
assert!(
plan_config_extensions(vec![config(
"/cfg/.dirsql.toml",
Some("[[dirsql.extension]]\npath = \"./vec0.so\"\n"),
)])
.is_none()
);
}

#[test]
fn plan_config_extensions_resolves_literals_against_their_config_dir() {
let plan = plan_config_extensions(vec![config(
"/cfg/.dirsql.toml",
Some(
"[[dirsql.extension]]\npath = \"./vec0.so\"\n\n[[dirsql.extension]]\npath = \"sqlite_vec\"\n",
),
)])
.expect("a package name forces a plan");
assert_eq!(plan[0].path.as_deref(), Some("/cfg/./vec0.so"));
assert_eq!(plan[1].package.as_deref(), Some("sqlite_vec"));
assert_eq!(plan[1].shadow.as_deref(), Some("/cfg/sqlite_vec"));
}

#[test]
fn plan_config_extensions_skips_an_unreadable_config() {
assert!(plan_config_extensions(vec![config("/cfg/.dirsql.toml", None)]).is_none());
}

#[test]
fn plan_config_extensions_carries_the_entrypoint() {
let plan = plan_config_extensions(vec![config(
"/cfg/.dirsql.toml",
Some("[[dirsql.extension]]\npath = \"sqlite_vec\"\nentrypoint = \"init\"\n"),
)])
.expect("a package name forces a plan");
assert_eq!(plan[0].entrypoint.as_deref(), Some("init"));
}

#[test]
fn select_loadable_returns_the_single_match() {
let found = selected_loadable(
"vec",
vec!["/pkg".into()],
vec![
"/pkg/README.md".into(),
format!("/pkg/vec0{}", PLATFORM_SUFFIXES[0]),
],
)
.expect("one loadable file");
assert_eq!(found, format!("/pkg/vec0{}", PLATFORM_SUFFIXES[0]));
}

#[test]
fn select_loadable_carries_the_cores_message_verbatim() {
let err =
selected_loadable("vec", vec!["/pkg".into()], vec![]).expect_err("no loadable file");
assert!(err.starts_with("no loadable extension file (*"), "{err}");
assert!(
err.ends_with("found in package 'vec' (searched /pkg)"),
"{err}"
);
}

#[test]
fn is_bare_name_applies_the_node_suffix_list() {
assert!(is_bare_name("sqlite-vec".into()));
assert!(!is_bare_name("vec0.node".into()));
assert!(!is_bare_name("vec0.so".into()));
assert!(is_bare_name("vec0.pyd".into()));
assert!(!is_bare_name("ext/vec0".into()));
}

fn one_row() -> HashMap<String, Value> {
HashMap::from([("k".to_string(), Value::Integer(7))])
}
Expand Down
3 changes: 1 addition & 2 deletions packages/ts/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -159,8 +159,7 @@
}
},
"dependencies": {
"bin-shim": "^0.2.0",
"smol-toml": "^1.6.1"
"bin-shim": "^0.2.0"
},
"devDependencies": {
"@biomejs/biome": "^1.9.0",
Expand Down
9 changes: 0 additions & 9 deletions packages/ts/pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

23 changes: 5 additions & 18 deletions packages/ts/src/cli/resolve-config-extensions.test.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
import { existsSync } from "node:fs";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { getCore } from "../core.js";
import { resolveConfigsExtensionSpecs } from "../resolve-config-extensions.js";
import { withResolvedExtensions } from "./resolve-config-extensions.js";

vi.mock("node:fs");
vi.mock("../core.js");
vi.mock("../resolve-config-extensions.js", async () => ({
...(await vi.importActual<typeof import("../resolve-config-extensions.js")>(
Expand All @@ -18,7 +16,6 @@ describe("withResolvedExtensions", () => {

beforeEach(() => {
vi.clearAllMocks();
vi.mocked(existsSync).mockReturnValue(true);
vi.mocked(getCore).mockReturnValue({ configPathsFromArgv } as never);
configPathsFromArgv.mockReturnValue(["/x/.dirsql.toml"]);
});
Expand All @@ -45,9 +42,9 @@ describe("withResolvedExtensions", () => {
"cfg.cjs",
]);
const argv = ["--config", "dirsql.config.mjs"];
vi.mocked(resolveConfigsExtensionSpecs).mockReturnValue(null);
expect(await withResolvedExtensions(argv)).toBe(argv);
expect(existsSync).not.toHaveBeenCalled();
expect(resolveConfigsExtensionSpecs).not.toHaveBeenCalled();
expect(resolveConfigsExtensionSpecs).toHaveBeenCalledWith([]);
});

it("drops native configs but resolves the TOML ones", async () => {
Expand All @@ -63,19 +60,9 @@ describe("withResolvedExtensions", () => {
]);
});

it("passes through without loading the parser when no config file exists", async () => {
vi.mocked(existsSync).mockReturnValue(false);
configPathsFromArgv.mockReturnValue(["/gone/.dirsql.toml"]);
const argv = ["--config", "/gone/.dirsql.toml"];
expect(await withResolvedExtensions(argv)).toBe(argv);
expect(existsSync).toHaveBeenCalledWith("/gone/.dirsql.toml");
expect(resolveConfigsExtensionSpecs).not.toHaveBeenCalled();
});

it("consults the resolver with every path when at least one config exists", async () => {
// The resolver skips missing configs itself; the existsSync guard only
// keeps the TOML parser off the launch path when NO config exists.
vi.mocked(existsSync).mockImplementation((p) => p === "/b/.dirsql.toml");
it("consults the resolver with every path, missing configs included", async () => {
// The resolver hands each config to the core, which skips the ones it
// could not read.
configPathsFromArgv.mockReturnValue(["/a/.dirsql.toml", "/b/.dirsql.toml"]);
vi.mocked(resolveConfigsExtensionSpecs).mockReturnValue(null);
await withResolvedExtensions([
Expand Down
Loading
Loading