Separate the LAMBDA keyword syntax from the -> lambda syntax by adriangb · Pull Request #2458 · apache/datafusion-sqlparser-rs

adriangb · 2026-08-25T23:29:27Z

Motivation

supports_lambda_functions() gates two different spellings of the same feature:

the arrow form, x -> x + 1
the LAMBDA keyword form, LAMBDA x : x + 1

Because they share one flag, a dialect cannot have one without the other. That is
a problem for any dialect where -> already means something else.

DuckDB

DuckDB uses -> for both JSON member access (j -> '$.family') and the arrow
lambda form, and resolves the ambiguity from the function signature at bind time —
which a parser cannot do. DuckDB's own answer was to add the lambda keyword and
deprecate the arrow:

DuckDB v1.3 deprecated the old lambda single arrow syntax (x -> x + 1) in
favor of the Python-style syntax (lambda x : x + 1). [...] DuckDB v2.0 will
disable the single arrow syntax by default.

DuckDB exposes the choice as SET lambda_syntax = 'ENABLE_SINGLE_ARROW' | 'DISABLE_SINGLE_ARROW'. Today this crate can only express the first; the two flags
here are what it takes to express the second.

The consequence is visible now. DuckDbDialect sets supports_lambda_functions() == true, so DuckDB's own documented JSON example misparses:

SQL (DuckDB dialect)	parses as
`SELECT j -> 'field' FROM t`	`Expr::Lambda { params: [j], body: 'field' }`
`SELECT t.j -> 'field' FROM t`	`BinaryOp { op: Arrow }`

A bare column becomes a lambda; a qualified column stays JSON access. Both round-trip
to identical SQL, which is why no test catches it.

This PR does not change DuckDbDialect. Flipping it to the keyword-only spelling
fixes the table above but introduces the mirror-image misparse on x -> x > 1, and
that tradeoff deserves its own discussion. This change makes the correct behaviour
expressible, which is the prerequisite either way.

Custom dialects

The same conflict is unavoidable for anyone building a custom dialect — derived from
PostgreSQL or not — that wants JSON accessors and lambda functions at once. Today,
enabling lambdas silently reinterprets every ->; there is no way to have both.
Overriding only supports_lambda_keyword_syntax gives them lambdas with -> left
alone. Unlike DuckDB there is no tradeoff here, since the keyword form is unambiguous.

Change

Adds Dialect::supports_lambda_keyword_syntax(), which gates only the LAMBDA
keyword form and defaults to supports_lambda_functions().

No existing dialect changes behaviour. Dialects that support the arrow form keep both
spellings; dialects that support neither still get neither. A dialect that uses ->
for something else can now override just the new method to get lambdas without
disturbing its operator.

Tests

tests/sqlparser_custom_dialect.rs:

a dialect enabling only supports_lambda_keyword_syntax parses lambda x : x + 1
while -> stays a BinaryOperator::Arrow
a dialect enabling only supports_lambda_functions still accepts both spellings,
pinning the defaulting behaviour
a dialect enabling the arrow form but disabling the keyword form accepts only the
arrow spelling

tests/sqlparser_derive_dialect.rs:

a derived dialect overriding only supports_lambda_keyword_syntax parses a lambda
whose body is a JSON access, lambda x : (x -> 'a')::INT + 1, asserting the outer
expression is a LambdaSyntax::LambdaKeyword lambda and the inner -> is still
BinaryOperator::Arrow

tests/sqlparser_duckdb.rs:

pins x -> x > 1 as a LambdaSyntax::Arrow lambda. Both readings of -> print
identically, so the pre-existing round-trip check passes under either one — without
this assertion, switching DuckDbDialect to the keyword-only spelling leaves the
entire suite green.

Full suite, cargo fmt --check, and cargo clippy --all-targets --all-features all
pass.

adriangb · 2026-08-26T12:13:01Z

+    fn supports_lambda_keyword_syntax(&self) -> bool {
+        self.supports_lambda_functions()
+    }


One could argue for adding fn supports_lambda_arrow_syntax() as well, but I'd hold off until there is a concrete use case for enabling arrow syntax but not lambda syntax.

Arrow-only is already expressible by overriding supports_lambda_keyword_syntax to false, so I believe there is no need.

adriangb · 2026-08-26T12:35:21Z

+#[test]
+fn test_lambda_keyword_syntax_on_postgres_derivative() {


I'm split between this (the real regression test I want) and another test using a MyDialect in sqlparser_custom_dialect.rs that enables the two flags. Open to input.

Both seem worth keeping.

adriangb · 2026-08-30T15:02:02Z

@LucaCappelletti94 could I ask you to take a look at this change? Thanks!

LucaCappelletti94

Generally ok, just a missing Snowflake syntax test, and the missing documentation to lambda, and the wrong syntax in the example.

LucaCappelletti94 · 2026-08-30T15:29:10Z

+    /// lambda functions, for example:
+    ///
+    /// ```sql
+    /// SELECT transform(array(1, 2, 3), LAMBDA x : x + 1); -- returns [2,3,4]


I believe transform(array(...), LAMBDA ...) is the wrong spelling, if you meant the DuckDB syntax it would be more like:

Suggested change

/// SELECT transform(array(1, 2, 3), LAMBDA x : x + 1); -- returns [2,3,4]

/// SELECT list_transform([1, 2, 3], lambda x : x + 1); -- returns [2, 3, 4]

Thanks, applied!

LucaCappelletti94 · 2026-08-30T15:30:41Z

+    fn supports_lambda_keyword_syntax(&self) -> bool {
+        self.supports_lambda_functions()
+    }


Arrow-only is already expressible by overriding supports_lambda_keyword_syntax to false, so I believe there is no need.

LucaCappelletti94 · 2026-08-30T15:31:26Z

+#[test]
+fn test_lambda_keyword_syntax_on_postgres_derivative() {


Both seem worth keeping.

adriangb · 2026-08-31T11:36:17Z

Thanks for the review @LucaCappelletti94!

@iffyio or @yoavcloud could one of you take a look at this PR that @LucaCappelletti94 has already reviewed and approved? Thanks!

iffyio

left a comment, its not clear to me what the PR is looking to solve for

iffyio · 2026-09-02T07:54:47Z

+    /// [`Self::supports_lambda_functions`], so dialects supporting the `->`
+    /// spelling accept the `LAMBDA` spelling too unless they say otherwise.
+    ///
+    /// See <https://duckdb.org/docs/stable/sql/functions/lambda>


Im not sure I understood the problem being solved for - the description mentions pg as an example, suggesting there is some ambiguous grammar in play but pg doesnt have lambda syntax to my knowledge?

is there an example syntax that issupported by a dialect and the parser doesnt cover?

This is a fair point. I replied in #2458 (comment) but reflecting a bit more I think there's an even stronger framing. Your hesitation comes from PostgreSQL being a bad example. It has no lambda syntax, so the new flag
would be unreachable there and the motivation is then speculative. I've reframed the
description to focus on DuckDB and custom dialects.

For DuckDB this enables fixing a live bug rather than a hypothetical.
DuckDbDialect already sets supports_lambda_functions() == true, so DuckDB's own
documented JSON example misparses today:

// DuckDbDialect "SELECT j -> 'field' FROM t" // => Expr::Lambda { params: [j], body: 'field' } ❌ "SELECT t.j -> 'field' FROM t" // => BinaryOp { op: Arrow } ✅

A bare column becomes a lambda, a qualified one stays JSON access. Both print back as
the same SQL, which is why no round-trip test catches it.

DuckDB itself resolves -> from the function signature at bind time, which a parser
can't do — and that's exactly why they deprecated the arrow form in v1.3 in favor of
lambda x : x + 1, with v2.0 disabling it by default and SET lambda_syntax to
choose in between. Following that requires the two spellings to be separable, which is
all this PR does.

I've deliberately not flipped DuckDbDialect here: it fixes the case above but breaks
x -> x > 1 in the other direction, so it's worth its own PR. I did add an assertion
pinning the current arrow behavior because I found the whole suite stays green when
you flip it. This way if we do flip it we can verify the change in behavior against tests.

This PR also pplies to custom dialects wanting JSON accessors and lambdas at once, which
is what the derive_dialect! test now covers. This could be a postgres based dialect that wants to add support for lambda functions (our case, using derive_dialect!) or a completely custom dialect.

adriangb · 2026-09-02T11:21:53Z

you cannot currently have a grammar that supports both postgres style json operators and lambda transforms because -> conflicts

Exercises the capability the way a downstream crate would: derive a dialect from PostgreSqlDialect with `supports_lambda_keyword_syntax` overridden, then check that `LAMBDA x : x + 1` parses while `->` and `->>` keep parsing as JSON member access rather than lambda parameters. PostgreSqlDialect is used here only as a convenient base that already gives `->` its JSON meaning; the case being covered is any custom dialect that wants JSON accessors and lambda functions at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Address review feedback: rather than parsing the `LAMBDA` spelling and the `->` operator as separate statements, parse one expression that uses both — a lambda whose body is a JSON access — which is the shape a PostgreSQL derivative actually cares about. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

All three changes are applied review suggestions from Luca Cappelletti. Co-Authored-By: Luca Cappelletti <7738570+LucaCappelletti94@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

PostgreSQL was a poor example: it has no lambda functions, so the capability was unreachable there and the motivation read as speculative. The real case is DuckDB, which uses `->` for both JSON member access and the arrow lambda form and resolves the ambiguity from the function signature at bind time. A parser cannot, which is why DuckDB deprecated the arrow form in v1.3 (`SET lambda_syntax`) and disables it by default in v2.0. Expressing that requires the two spellings to be separable. Also pin the shape of DuckDB's `x -> x > 1` as a lambda. Both readings of `->` print identically, so the existing round-trip check passes under either one; without this assertion, changing `DuckDbDialect` to the keyword-only spelling leaves the whole suite green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb commented Aug 26, 2026

View reviewed changes

Comment thread tests/sqlparser_derive_dialect.rs Outdated

LucaCappelletti94 suggested changes Aug 30, 2026

View reviewed changes

adriangb force-pushed the split-lambda-keyword-syntax branch from 2d11f84 to d994419 Compare August 30, 2026 16:08

adriangb requested a review from LucaCappelletti94 August 30, 2026 16:08

adriangb force-pushed the split-lambda-keyword-syntax branch from d994419 to 1d89c66 Compare August 30, 2026 16:24

LucaCappelletti94 approved these changes Aug 31, 2026

View reviewed changes

iffyio reviewed Sep 2, 2026

View reviewed changes

adriangb and others added 6 commits September 6, 2026 15:06

Allow dialects that use -> as an operator to support LAMBDA syntax

17ee034

Remove redundant borrows in format! args to satisfy clippy 1.98

571b7ed

Address review: DuckDB doc example and link, arrow-only dialect test

752b6e7

All three changes are applied review suggestions from Luca Cappelletti. Co-Authored-By: Luca Cappelletti <7738570+LucaCappelletti94@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb force-pushed the split-lambda-keyword-syntax branch from 1d89c66 to 7e7bfc7 Compare September 6, 2026 22:07

adriangb changed the title ~~Allow dialects that use -> as an operator to support LAMBDA syntax~~ Separate the LAMBDA keyword syntax from the -> lambda syntax Sep 6, 2026

		#[test]
		fn test_lambda_keyword_syntax_on_postgres_derivative() {

	/// SELECT transform(array(1, 2, 3), LAMBDA x : x + 1); -- returns [2,3,4]
	/// SELECT list_transform([1, 2, 3], lambda x : x + 1); -- returns [2, 3, 4]

Uh oh!

Conversation

adriangb commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

DuckDB

Custom dialects

Change

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adriangb commented Aug 30, 2026

Uh oh!

LucaCappelletti94 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Aug 31, 2026

Uh oh!

iffyio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adriangb commented Aug 25, 2026 •

edited

Loading