Skip to content

Commit 4f601d2

Browse files
committed
perf: reuse cached tokens when probing PostgreSQL typed literals
The typed-literal lookahead copied the tokenizer and scanned the token after an identifier, then scanned it again on the normal expression path. Ordinary column references and function names paid this cost even when the following token could not form a type-prefixed literal. Consume the leading name once and reuse the real tokenizer's cached next token. Only copy tokenizer state when a following string, modifier list, qualification or multiword type prefix warrants a type probe. Let the type parser accept an already-consumed leading token, retaining the original source span without rescanning it. Failed probes resume the ordinary name or function parser with the same token boundary and quoting information. This changes allocation-free lookahead, not grammar, AST shape or ownership. It reduces the observed simple SELECT, JOIN and complex SELECT costs relative to the first expression implementation. The broader grammar still has a measurable cost relative to the earlier selective parser; performance claims remain scoped to measured workloads and allocation modes. Validation: the forced full C++ build passes 1,370 active tests with 37 backend-dependent skips and builds the corpus harness. All 42 focused expression/AST tests pass under AddressSanitizer and UndefinedBehaviorSanitizer. The direct Rust API benchmark validates complete AST parsing and canonical SQL equivalence before timing. Benchmark sources, results and build artifacts remain uncommitted.
1 parent 24cd1f3 commit 4f601d2

2 files changed

Lines changed: 11 additions & 8 deletions

File tree

include/sql_parser/expression_parser.h

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -181,23 +181,25 @@ class ExpressionParser {
181181
Token t = tok_.peek();
182182
if constexpr (D == Dialect::PostgreSQL) {
183183
if (PgTypeParser::name_token(t) && !keyword(t, "INTERVAL")) {
184-
auto lookahead = tok_;
185-
lookahead.skip();
186-
const Token next = lookahead.peek();
184+
// Consume the name once. Ordinary references use the real
185+
// tokenizer's cached next token rather than rescanning a copy.
186+
tok_.skip();
187+
const Token next = tok_.peek();
187188
// Avoid scanning a complete type for ordinary column references.
188189
if (next.type == TokenType::TK_STRING || next.type == TokenType::TK_LPAREN ||
189190
next.type == TokenType::TK_DOT || keyword(t, "TIMESTAMP") ||
190191
keyword(t, "TIME") || keyword(t, "DOUBLE") || keyword(t, "CHARACTER") ||
191192
keyword(t, "CHAR") || keyword(t, "NCHAR") || keyword(t, "NATIONAL") || keyword(t, "BIT")) {
192-
lookahead = tok_;
193-
StringRef type = PgTypeParser(lookahead).parse(false, true);
193+
auto lookahead = tok_;
194+
StringRef type = PgTypeParser(lookahead).parse(false, true, &t);
194195
if (!type.empty() && lookahead.peek().type == TokenType::TK_STRING) {
195196
Token literal = lookahead.next_token();
196197
tok_ = lookahead;
197198
AstNode* value = make_node_from_token(arena_, NodeType::NODE_LITERAL_STRING, literal);
198199
return make_cast(value, type);
199200
}
200201
}
202+
return parse_identifier_or_function(t);
201203
}
202204
}
203205

include/sql_parser/pg_type_parser.h

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,11 @@ class PgTypeParser {
2222
token.type == TokenType::TK_DATA || token.type == TokenType::TK_SCHEMA;
2323
}
2424

25-
StringRef parse(bool arrays = true, bool constant = false) {
26-
Token first = tok_.peek();
25+
StringRef parse(bool arrays = true, bool constant = false, const Token* leading = nullptr) {
26+
Token first = leading ? *leading : tok_.peek();
2727
if (!name_token(first)) return {};
28-
take();
28+
if (leading) last_ = first.source;
29+
else take();
2930
bool qualified = false;
3031
while (tok_.peek().type == TokenType::TK_DOT) {
3132
qualified = true;

0 commit comments

Comments
 (0)