Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ repository = "https://github.com/thekevinscott/dirsql"
[workspace.dependencies]
dirsql = { path = "packages/rust" }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
# `preserve_order`: a serialized row must keep the projection order the SELECT
# list asked for, which the default BTreeMap-backed object discards.
serde_json = { version = "1", features = ["preserve_order"] }
regex = "1"
thiserror = "2"
tokio = { version = "1", features = ["rt", "sync"] }
Expand Down
1 change: 1 addition & 0 deletions PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ core, so all three SDKs change at once. **No drift.**
|----------------------------|------------------------------------------------|------------------------------------------------------|---------------------------------------------------------|
| Constructor | `DirSQL(root=None, *, tables=None, ignore=None, no_ignore=False, config=None, persist=False, persist_path=None, extensions=None)` | `DirSQL::builder().root(..).tables(..).ignore(..).config(..).persist(Option<path>).extensions(..).build()` (also `DirSQL::new`/`with_ignore` shortcuts) | `new DirSQL(configPath)` or `new DirSQL({ root?, tables?, ignore?, config?, persist?, persistPath?, extensions?, noIgnore? })` + `await db.ready` |
| Query (read-only; rejects non-SELECT) | `db.query(sql) -> list[dict]` | `db.query(sql) -> Result<Vec<Row>>` | `await db.query(sql) -> Record[]` (runs on libuv threadpool) |
| Query carrying the projection order | — (drift, #1173) | `db.query_ordered(sql) -> Result<QueryResult>` (`columns` + `rows`) | — (drift, #1174) |
| Start watcher | `db._start_watcher()` | `db.start_watching()` | `await db.startWatcher()` (runs on libuv threadpool) |
| Poll events | `db._poll_events(ms)` | `db.poll_events(duration)` | `await db.pollEvents(ms)` (runs on libuv threadpool) |
| Watch (channel/stream) | `async for event in db.watch()` (via `_async.py`) | `db.watch() -> WatchStream` (channel) | `for await (const ev of db.watch())` |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"command": "just test-e2e",
"ran_at": 1790181270,
"exit_code": 0,
"commit": "010d9772867fcb1e1b56ce4a77eb8b0223b0870a",
"branch": "claude/1170-projection-order"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
**Fixed** Columns now render in the order the SELECT list names them, in both
the table and JSON renderings. `SELECT path, ctime FROM './'` printed
`ctime, path` because the row maps are unordered and every renderer re-derived
its columns alphabetically. `Db::query_ordered` / `DirSQL::query_ordered`
return the projection order alongside the rows, and the CLI serializes each row
against it.
35 changes: 35 additions & 0 deletions packages/rust/migrations.d/2026-09-23-projection-column-order.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
### Result columns follow the projection order

**Summary**

Query results are rendered in the order the SELECT list names the columns,
where they were previously alphabetical. This affects the Rust core's CLI
(`--format table` and `--format json`) and the HTTP `POST /query` body. No
signature changed; `DirSQL::query` still returns `Vec<Row>`, and the new
`DirSQL::query_ordered` is additive.

**Required changes**

_None._

**Deprecations removed**

_None._

**Behavior changes without code changes**

| Surface | Before | After |
|---|---|---|
| `dirsql "SELECT path, ctime FROM './'" --format table` | header `ctime path` | header `path ctime` |
| `dirsql "SELECT path, ctime FROM './'" --format json` | `{"ctime":…,"path":…}` | `{"path":…,"ctime":…}` |
| `POST /query` response rows | keys sorted alphabetically | keys in projection order |

A consumer that relied on alphabetical key order (e.g. comparing serialized
JSON byte-for-byte) must key off the column names instead.

**Verification**

```bash
dirsql "SELECT path, ctime FROM './' LIMIT 2" --format json
# [{"path":"a.md","ctime":1790181167},…] -- path first, as written
```
10 changes: 7 additions & 3 deletions packages/rust/src/cli/execute.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ use serde::Deserialize;
use serde_json::Value;

use super::AppState;
use super::serialize::rows_to_json;
use super::serialize::result_to_json;
use crate::{DirSQL, DirSqlError};

/// Why a query failed, classified independently of transport. The HTTP
Expand Down Expand Up @@ -66,10 +66,14 @@ pub async fn execute_query(

let db = require_ready(state)?;

let join = run_bounded(timeout, tokio::task::spawn_blocking(move || db.query(&sql))).await?;
let join = run_bounded(
timeout,
tokio::task::spawn_blocking(move || db.query_ordered(&sql)),
)
.await?;

match join {
Ok(Ok(rows)) => Ok(Value::Array(rows_to_json(&rows))),
Ok(Ok(result)) => Ok(Value::Array(result_to_json(&result))),
Ok(Err(err)) => Err(classify_query_error(err)),
Err(join_err) => Err(QueryFailure::Internal(join_err.to_string())),
}
Expand Down
51 changes: 48 additions & 3 deletions packages/rust/src/cli/serialize.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,29 @@

use serde_json::{Map, Value};

use crate::Row;
use crate::db::Value as CellValue;
use crate::differ::RowEvent;
use crate::row_event_flat::flatten_row_event;
use crate::{QueryResult, Row};

/// Serialize a query result, each object keyed in projection order. Row events
/// carry no projection, so they keep going through [`row_to_json`].
pub(super) fn result_to_json(result: &QueryResult) -> Vec<Value> {
result
.rows
.iter()
.map(|row| ordered_row_to_json(&result.columns, row))
.collect()
}

pub(super) fn rows_to_json(rows: &[Row]) -> Vec<Value> {
rows.iter().map(row_to_json).collect()
fn ordered_row_to_json(columns: &[String], row: &Row) -> Value {
let mut map = Map::with_capacity(columns.len());
for column in columns {
if let Some(value) = row.get(column) {
map.insert(column.clone(), cell_to_json(value));
}
}
Value::Object(map)
}

pub(super) fn row_to_json(row: &Row) -> Value {
Expand Down Expand Up @@ -75,6 +91,35 @@ mod tests {
assert_eq!(json.get("count").and_then(Value::as_i64), Some(3));
}

#[test]
fn a_result_serializes_its_rows_in_projection_order() {
let mut row: Row = HashMap::new();
row.insert("title".into(), CellValue::Text("Hello".into()));
row.insert("count".into(), CellValue::Integer(3));
let result = QueryResult {
columns: vec!["count".into(), "title".into()],
rows: vec![row],
};

let json = result_to_json(&result);

assert_eq!(json[0].to_string(), r#"{"count":3,"title":"Hello"}"#);
}

#[test]
fn a_column_no_row_key_matches_is_left_out() {
let mut row: Row = HashMap::new();
row.insert("title".into(), CellValue::Text("Hello".into()));
let result = QueryResult {
columns: vec!["title".into(), "title".into()],
rows: vec![row],
};

let json = result_to_json(&result);

assert_eq!(json[0].to_string(), r#"{"title":"Hello"}"#);
}

#[test]
fn insert_event_emits_expected_shape() {
let mut row: Row = HashMap::new();
Expand Down
37 changes: 31 additions & 6 deletions packages/rust/src/cli/table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
//! swallowing the table; laying the table out to the terminal's width, and
//! paging a long result, are both out of scope.

use std::collections::BTreeSet;
use std::collections::HashSet;

use serde_json::Value;

Expand Down Expand Up @@ -53,16 +53,22 @@ pub(super) fn render(rows: &[Value]) -> String {
out
}

/// Every column any row carries. Rows are JSON objects, so the keys arrive
/// sorted and stay in step with the JSON rendering of the same result.
/// Every column any row carries, first seen first. Rows are JSON objects
/// serialized in projection order, so that is the order the SELECT list asked
/// for -- and the same order the JSON rendering of this result prints.
fn columns(rows: &[Value]) -> Vec<String> {
let mut names = BTreeSet::new();
let mut seen = HashSet::new();
let mut names = Vec::new();
for row in rows {
if let Value::Object(map) = row {
names.extend(map.keys().cloned());
for key in map.keys() {
if seen.insert(key.clone()) {
names.push(key.clone());
}
}
}
}
names.into_iter().collect()
names
}

/// One row's cells, in column order. A row missing a column renders it as
Expand Down Expand Up @@ -242,6 +248,25 @@ mod tests {
}
}

#[test]
fn the_header_keeps_the_order_the_row_carries() {
// Rows arrive serialized in projection order; re-sorting them here is
// what made `SELECT size, basename` print `basename size`.
let table = rendered([row([
("size", Value::from(6)),
("basename", Value::from("a.md")),
])]);

assert!(table.starts_with("size basename\n"), "{table:?}");
}

#[test]
fn a_later_rows_new_columns_come_after_the_earlier_ones() {
let table = rendered([row([("b", Value::from(1))]), row([("a", Value::from(2))])]);

assert!(table.starts_with("b a\n"), "{table:?}");
}

#[test]
fn every_column_any_row_carries_is_shown() {
// A ragged result must not silently drop the columns only some rows
Expand Down
54 changes: 53 additions & 1 deletion packages/rust/src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,13 @@ pub fn ensure_internal_rows_table(conn: &Connection) -> rusqlite::Result<()> {
)
}

/// One query's rows plus the column order the SELECT list asked for.
#[derive(Debug, Clone, PartialEq)]
pub struct QueryResult {
pub columns: Vec<String>,
pub rows: Vec<HashMap<String, Value>>,
}

pub struct Db {
conn: Connection,
/// The directory a path-table's glob is resolved against. `None` disables
Expand Down Expand Up @@ -714,6 +721,14 @@ impl Db {
/// internal writes (`insert_row`, delete-by-file, persist), which never go
/// through `query()`, are unaffected.
pub fn query(&self, sql: &str) -> Result<Vec<HashMap<String, Value>>> {
self.query_ordered(sql).map(|result| result.rows)
}

/// [`query`](Self::query), plus the projection order. A row is a
/// `HashMap`, so the order the SELECT list asked for survives only in
/// [`QueryResult::columns`]; a renderer that re-derives its columns from
/// row keys prints them alphabetically instead.
pub fn query_ordered(&self, sql: &str) -> Result<QueryResult> {
// Bracket the whole statement, prepare included: a worker-backed
// function called on every row makes this the slowest thing dirsql
// does, and the guard is what erases the counter before the caller
Expand Down Expand Up @@ -757,7 +772,10 @@ impl Db {
for row in rows {
results.push(row?);
}
Ok(results)
Ok(QueryResult {
columns: column_names,
rows: results,
})
}

/// Prepare `sql` with the internal-table / ATTACH authorizer installed for
Expand Down Expand Up @@ -1026,6 +1044,40 @@ mod tests {
assert!(!rows[0].contains_key("_dirsql_row_index"));
}

#[test]
fn query_ordered_reports_the_select_list_order() {
let db = Db::new().unwrap();
db.create_table("docs", "CREATE TABLE docs (title TEXT, draft INTEGER)")
.unwrap();

let result = db.query_ordered("SELECT draft, title FROM docs").unwrap();

assert_eq!(
result.columns,
vec!["draft".to_string(), "title".to_string()]
);
}

#[test]
fn query_ordered_carries_the_same_rows_query_returns() {
let db = Db::new().unwrap();
db.create_table("docs", "CREATE TABLE docs (title TEXT, draft INTEGER)")
.unwrap();
let row = HashMap::from([
("title".into(), Value::Text("Hello".into())),
("draft".into(), Value::Integer(0)),
]);
db.insert_row("docs", &row, "docs/hello.md", 0).unwrap();

let result = db.query_ordered("SELECT title, draft FROM docs").unwrap();

assert_eq!(
result.rows,
db.query("SELECT title, draft FROM docs").unwrap()
);
assert_eq!(result.rows[0]["title"], Value::Text("Hello".into()));
}

#[test]
fn pragma_table_info_matches_user_ddl_exactly() {
let db = Db::new().unwrap();
Expand Down
11 changes: 9 additions & 2 deletions packages/rust/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ use std::time::Duration;
use thiserror::Error;

pub use crate::config::ExtensionSpec as Extension;
pub use crate::db::{DbError, Value};
pub use crate::db::{DbError, QueryResult, Value};
pub use crate::differ::RowEvent;
#[doc(hidden)]
pub use crate::row_event_flat::{FlatRowEvent, flatten_row_event};
Expand Down Expand Up @@ -404,8 +404,15 @@ impl DirSQL {
/// keeps the ephemeral index consistent with the on-disk files that back
/// it: mutations only happen through the watcher/indexer pipeline.
pub fn query(&self, sql: &str) -> Result<Vec<Row>> {
self.query_ordered(sql).map(|result| result.rows)
}

/// [`query`](Self::query), plus the projection order in
/// [`QueryResult::columns`]. A [`Row`] is a `HashMap`, so a caller that
/// renders columns in the order the user asked for needs this instead.
pub fn query_ordered(&self, sql: &str) -> Result<QueryResult> {
let db = self.inner.db.lock().map_err(DirSqlError::lock)?;
db.query(sql).map_err(map_db_error)
db.query_ordered(sql).map_err(map_db_error)
}

/// Lazily create the filesystem watcher. Idempotent; subsequent calls are
Expand Down
Loading
Loading