Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions src/dialect/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -535,10 +535,37 @@ pub trait Dialect: Debug + Any {
/// ```sql
/// SELECT transform(array(1, 2, 3), x -> x + 1); -- returns [2,3,4]
/// ```
///
/// This enables both the `->` spelling above and the `LAMBDA` keyword
/// spelling gated by [`Self::supports_lambda_keyword_syntax`]. A dialect
/// that uses `->` as a binary operator should override only the latter.
fn supports_lambda_functions(&self) -> bool {
false
}

/// Returns true if the dialect supports the `LAMBDA` keyword spelling of
/// lambda functions, for example:
///
/// ```sql
/// SELECT list_transform([1, 2, 3], lambda x : x + 1); -- returns [2, 3, 4]
/// ```
///
/// This spelling does not claim the `->` token, so it can be enabled by
/// dialects that already give `->` a different meaning — for example JSON
/// member access. DuckDB uses `->` for both, resolving the ambiguity from
/// the function signature at bind time rather than while parsing, and
/// deprecated the arrow lambda form in v1.3 in favour of this one; v2.0
/// disables the arrow form by default.
///
/// Defaults to [`Self::supports_lambda_functions`], so dialects supporting
/// the `->` spelling accept the `LAMBDA` spelling too unless they say
/// otherwise.
///
/// See <https://duckdb.org/docs/stable/sql/functions/lambda>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im not sure I understood the problem being solved for - the description mentions pg as an example, suggesting there is some ambiguous grammar in play but pg doesnt have lambda syntax to my knowledge?

is there an example syntax that issupported by a dialect and the parser doesnt cover?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a fair point. I replied in #2458 (comment) but reflecting a bit more I think there's an even stronger framing. Your hesitation comes from PostgreSQL being a bad example. It has no lambda syntax, so the new flag
would be unreachable there and the motivation is then speculative. I've reframed the
description to focus on DuckDB and custom dialects.

For DuckDB this enables fixing a live bug rather than a hypothetical.
DuckDbDialect already sets supports_lambda_functions() == true, so DuckDB's own
documented JSON example misparses today:

// DuckDbDialect
"SELECT j -> 'field' FROM t"    // => Expr::Lambda { params: [j], body: 'field' }  ❌
"SELECT t.j -> 'field' FROM t"  // => BinaryOp { op: Arrow }                       ✅

A bare column becomes a lambda, a qualified one stays JSON access. Both print back as
the same SQL, which is why no round-trip test catches it.

DuckDB itself resolves -> from the function signature at bind time, which a parser
can't do — and that's exactly why they deprecated the arrow form in v1.3 in favor of
lambda x : x + 1, with v2.0 disabling it by default and SET lambda_syntax to
choose in between. Following that requires the two spellings to be separable, which is
all this PR does.

I've deliberately not flipped DuckDbDialect here: it fixes the case above but breaks
x -> x > 1 in the other direction, so it's worth its own PR. I did add an assertion
pinning the current arrow behavior because I found the whole suite stays green when
you flip it. This way if we do flip it we can verify the change in behavior against tests.

This PR also pplies to custom dialects wanting JSON accessors and lambdas at once, which
is what the derive_dialect! test now covers. This could be a postgres based dialect that wants to add support for lambda functions (our case, using derive_dialect!) or a completely custom dialect.

fn supports_lambda_keyword_syntax(&self) -> bool {
Comment thread
adriangb marked this conversation as resolved.
self.supports_lambda_functions()
}
Comment on lines +565 to +567

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One could argue for adding fn supports_lambda_arrow_syntax() as well, but I'd hold off until there is a concrete use case for enabling arrow syntax but not lambda syntax.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow-only is already expressible by overriding supports_lambda_keyword_syntax to false, so I believe there is no need.


/// Returns true if the dialect supports multiple variable assignment
/// using parentheses in a `SET` variable declaration.
///
Expand Down
2 changes: 1 addition & 1 deletion src/parser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1635,7 +1635,7 @@ impl<'a> Parser<'a> {
Keyword::MAP if *self.peek_token_ref() == Token::LBrace && self.dialect.support_map_literal_syntax() => {
Ok(Some(self.parse_duckdb_map_literal()?))
}
Keyword::LAMBDA if self.dialect.supports_lambda_functions() => {
Keyword::LAMBDA if self.dialect.supports_lambda_keyword_syntax() => {
Ok(Some(self.parse_lambda_expr()?))
}
_ if self.dialect.supports_geometric_types() => match w.keyword {
Expand Down
117 changes: 117 additions & 0 deletions tests/sqlparser_custom_dialect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ use sqlparser::{
dialect::Dialect,
keywords::Keyword,
parser::{Parser, ParserError},
test_utils::{expr_from_projection, only},
tokenizer::Token,
};

Expand Down Expand Up @@ -167,3 +168,119 @@ fn is_identifier_part(ch: char) -> bool {
|| ch == '$'
|| ch == '_'
}

#[test]
fn custom_dialect_lambda_keyword_syntax_without_arrow() {
// A dialect that gives `->` its own meaning can still support lambdas
// through the `LAMBDA` keyword spelling.
#[derive(Debug)]
struct MyDialect {}

impl Dialect for MyDialect {
fn is_identifier_start(&self, ch: char) -> bool {
is_identifier_start(ch)
}

fn is_identifier_part(&self, ch: char) -> bool {
is_identifier_part(ch)
}

fn supports_lambda_keyword_syntax(&self) -> bool {
true
}
}

let dialect = MyDialect {};

// The `LAMBDA` spelling parses.
let sql = "SELECT transform(xs, lambda x : x + 1)";
assert_eq!(
sql,
&format!("{}", Parser::parse_sql(&dialect, sql).unwrap()[0])
);

// `->` keeps whatever meaning the dialect gives it, rather than
// introducing a lambda parameter.
let sql = "SELECT a -> 'b'";
let ast = Parser::parse_sql(&dialect, sql).unwrap();
match &ast[0] {
Statement::Query(query) => {
let Expr::BinaryOp { op, .. } =
expr_from_projection(only(&query.body.as_select().unwrap().projection))
else {
panic!("expected `->` to stay a binary operator");
};
assert_eq!(&BinaryOperator::Arrow, op);
}
stmt => panic!("unexpected statement {stmt}"),
}
}

#[test]
fn custom_dialect_lambda_keyword_defaults_to_arrow_support() {
// Dialects that opt into the `->` spelling get the `LAMBDA` spelling too,
// so the new capability does not change any existing dialect.
#[derive(Debug)]
struct MyDialect {}

impl Dialect for MyDialect {
fn is_identifier_start(&self, ch: char) -> bool {
is_identifier_start(ch)
}

fn is_identifier_part(&self, ch: char) -> bool {
is_identifier_part(ch)
}

fn supports_lambda_functions(&self) -> bool {
true
}
}

let dialect = MyDialect {};
assert!(dialect.supports_lambda_keyword_syntax());
for sql in [
"SELECT transform(xs, lambda x : x + 1)",
"SELECT transform(xs, x -> x + 1)",
] {
assert_eq!(
sql,
&format!("{}", Parser::parse_sql(&dialect, sql).unwrap()[0])
);
}
}

#[test]
fn custom_dialect_lambda_arrow_syntax_without_keyword() {
// Arrow lambdas stay on while the `LAMBDA` keyword spelling is off,
// as in engines like Spark and Snowflake.
#[derive(Debug)]
struct MyDialect {}

impl Dialect for MyDialect {
fn is_identifier_start(&self, ch: char) -> bool {
is_identifier_start(ch)
}

fn is_identifier_part(&self, ch: char) -> bool {
is_identifier_part(ch)
}

fn supports_lambda_functions(&self) -> bool {
true
}

fn supports_lambda_keyword_syntax(&self) -> bool {
false
}
}

let dialect = MyDialect {};

let sql = "SELECT transform(xs, x -> x + 1)";
assert_eq!(
sql,
&format!("{}", Parser::parse_sql(&dialect, sql).unwrap()[0])
);
assert!(Parser::parse_sql(&dialect, "SELECT transform(xs, lambda x : x + 1)").is_err());
}
69 changes: 69 additions & 0 deletions tests/sqlparser_derive_dialect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,13 @@

//! Tests for the `derive_dialect!` macro.

use sqlparser::ast::{
BinaryOperator, Expr, FunctionArg, FunctionArgExpr, FunctionArguments, LambdaSyntax, Statement,
};
use sqlparser::derive_dialect;
use sqlparser::dialect::{Dialect, GenericDialect, MySqlDialect, PostgreSqlDialect};
use sqlparser::parser::Parser;
use sqlparser::test_utils::{expr_from_projection, only};

#[test]
fn test_method_overrides() {
Expand Down Expand Up @@ -121,3 +125,68 @@ fn test_identifier_quote_style_overrides() {
None
);
}

#[test]
fn test_lambda_keyword_syntax_with_json_arrow_operator() {
// A custom dialect can opt into the `LAMBDA` keyword spelling of lambda
// functions without giving up `->` as JSON member access. The two meet in
// a single expression below: a lambda whose body is a JSON access.
//
// PostgreSqlDialect is used only as a convenient base that already gives
// `->` its JSON meaning; nothing here is specific to PostgreSQL.
derive_dialect!(
LambdaPostgreSqlDialect,
PostgreSqlDialect,
overrides = { supports_lambda_keyword_syntax = true }
);
let dialect = LambdaPostgreSqlDialect::new();

// Only the keyword spelling is enabled; the arrow spelling stays off.
assert!(dialect.supports_lambda_keyword_syntax());
assert!(!dialect.supports_lambda_functions());

let sql = "SELECT transform(xs, lambda x : (x -> 'a')::INT + 1)";
let ast = Parser::parse_sql(&dialect, sql).unwrap();
assert_eq!(sql, ast[0].to_string());

// Round-tripping alone would not distinguish a JSON access from a nested
// lambda, since both print as `x -> 'a'`, so check the parsed shape.
let Statement::Query(query) = &ast[0] else {
panic!("unexpected statement {}", ast[0]);
};
let Expr::Function(func) =
expr_from_projection(only(&query.body.as_select().unwrap().projection))
else {
panic!("expected a function call");
};
let FunctionArguments::List(args) = &func.args else {
panic!("expected an argument list");
};
let [_, FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Lambda(lambda)))] = &args.args[..]
else {
panic!("expected the second argument to be a lambda");
};

// The lambda came from the `LAMBDA` keyword, not from `->`.
assert_eq!(LambdaSyntax::LambdaKeyword, lambda.syntax);

// And the `->` in its body is still JSON member access.
let Expr::BinaryOp {
left,
op: BinaryOperator::Plus,
..
} = lambda.body.as_ref()
else {
panic!("expected the lambda body to be an addition");
};
let Expr::Cast { expr, .. } = left.as_ref() else {
panic!("expected the left operand to be a cast");
};
let Expr::Nested(json_access) = expr.as_ref() else {
panic!("expected the cast operand to be parenthesized");
};
let Expr::BinaryOp { op, .. } = json_access.as_ref() else {
panic!("expected `->` to stay a binary operator");
};
assert_eq!(&BinaryOperator::Arrow, op);
}
Comment thread
adriangb marked this conversation as resolved.
18 changes: 18 additions & 0 deletions tests/sqlparser_duckdb.rs
Original file line number Diff line number Diff line change
Expand Up @@ -902,6 +902,24 @@ fn test_duckdb_lambda_function() {
let sql_arrow = "SELECT list_filter([1, 2, 3], x -> x > 1)";
duckdb().verified_stmt(sql_arrow);

// `->` is ambiguous in DuckDB: it is both the arrow lambda spelling and
// JSON member access, and DuckDB resolves it from the function signature
// at bind time. `DuckDbDialect` currently resolves it to a lambda. Both
// readings print identically, so round-tripping cannot tell them apart —
// assert the shape so any future change to that choice is visible here.
let select = duckdb().verified_only_select(sql_arrow);
let Expr::Function(func) = expr_from_projection(only(&select.projection)) else {
panic!("expected a function call");
};
let FunctionArguments::List(args) = &func.args else {
panic!("expected an argument list");
};
let [_, FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Lambda(lambda)))] = &args.args[..]
else {
panic!("expected the second argument to be a lambda");
};
assert_eq!(LambdaSyntax::Arrow, lambda.syntax);

// Test lambda with multiple parameters (with index)
let sql_multi = "SELECT list_filter([1, 3, 1, 5], lambda x, i : x > i)";
duckdb().verified_stmt(sql_multi);
Expand Down
Loading