diff --git a/CHANGELOG.md b/CHANGELOG.md index 323c746b2d..70fe524313 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ### Fixed - Fixed the `EDATE` function returning the 28th of February instead of the 29th when the shifted date falls in February of a leap year (e.g. `=EDATE(DATE(2028, 1, 31), 1)` now returns `2028-02-29` instead of `2028-02-28`). [#1781](https://github.com/handsontable/hyperformula/pull/1781) +- Fixed the parser rejecting formulas with the function-name prefixes that Excel writes into `.xlsx` files (e.g. `_xlfn.XLOOKUP`), which are now ignored and left out of `getCellFormula()`. [#1655](https://github.com/handsontable/hyperformula/issues/1655) ## [3.5.0] - 2026-10-09 diff --git a/docs/guide/file-import.md b/docs/guide/file-import.md index a937071807..42af33e2ea 100644 --- a/docs/guide/file-import.md +++ b/docs/guide/file-import.md @@ -23,6 +23,27 @@ To import CSV files, use a third-party [CSV parser](https://www.npmjs.com/search To import XLSX files, use a third-party [XLSX parser](https://www.npmjs.com/search?q=xlsx) (e.g., [ExcelJS](https://www.npmjs.com/package/exceljs) or [xlsx](https://www.npmjs.com/package/xlsx)). Then pass the result to HyperFormula as a JavaScript array. +### Excel's internal function prefixes + +Excel stores some function names with an internal prefix. It marks every function added after Excel 2007 this way, so a file can contain `_xlfn.IFS(...)` where the user typed `IFS(...)`. The prefixes are an artifact of how Excel saves the file, not part of the function name. + +Whether the prefixes reach your code depends on the parser you use: ExcelJS, for example, passes them through. + +HyperFormula ignores these prefixes, so you can pass the formula straight to the engine: + +| Prefix | Example | +| --- | --- | +| `_xlfn.` | `=_xlfn.IFS(A1>B1,"Pass",A1<=B1,"Fail")` | +| `_xlfn._xlws.` | `=_xlfn._xlws.FILTER(A1:A9,B1:B9>1)` | +| `_xlws.` | `=_xlws.SORT(A1:A9)` | +| `_xludf.` | `=_xludf.MY_FUNCTION()` | + +The `_xlpm.` prefix marks a `LAMBDA`/`LET` parameter name. HyperFormula does not support `LAMBDA` or `LET`, with or without the prefixes: a `LET` formula returns `#NAME?`, and a `LAMBDA` called in place, such as `=_xlfn.LAMBDA(_xlpm.x,_xlpm.x+1)(1)`, is a parsing error. + +[`getCellFormula()`](../api/classes/hyperformula.md#getcellformula) returns the formula without the prefix, so `=_xlfn.IFS(A1>B1,"Pass",A1<=B1,"Fail")` reads back as `=IFS(A1>B1,"Pass",A1<=B1,"Fail")`. + +A prefix does not add a function. If HyperFormula does not support the function itself, the cell holds a `#NAME?` error. See the [list of supported functions](built-in-functions.md). + ### Example: Import XLSX files in Node This example uses [ExcelJS](https://www.npmjs.com/package/exceljs) to import XLSX files into HyperFormula. diff --git a/hyperformula/dev-docs/PARSER.md b/hyperformula/dev-docs/PARSER.md index a190df5bc6..74b652d85b 100644 --- a/hyperformula/dev-docs/PARSER.md +++ b/hyperformula/dev-docs/PARSER.md @@ -20,7 +20,8 @@ - **The lexer is language-dependent.** Function names, the argument separator, the decimal separator, and error literals all come from the config and the translation package. Never hard-code an English name or an English-locale separator. See [`I18N.md`](I18N.md). - **Parsing and unparsing move together.** A grammar change that `Unparser.ts` does not learn about breaks round-tripping: the formula parses, the engine calculates, and `getCellFormula` returns something the user never typed. Add a round-trip test for every grammar change. - **A function name can look like a cell reference.** `HEX2DEC` contains the cell reference `HEX2`, and `LOG10` is one. A new function with such a name needs a parse test and a round-trip test. -- **The cache is load-bearing.** `ParserWithCaching` does not key on the formula string. `computeHashFromTokens` hashes the lexer tokens with each reference in the AST's address form — relative parts as offsets from the formula's address, `$` parts absolute — and every function name except `OFFSET` made canonical (`OFFSET` has its own token and is hashed as typed); rewritten ASTs are stored and fetched under `computeHashFromAst`. The two must give the same key for the same formula, or `fetchCachedResultForAst` throws `There is no AST with such key in the cache`. A new token kind must hash to the same text in both: `computeHashFromTokens` falls back to the raw token image, so it needs its own branch whenever that image is not canonical, and `computeHashOfAstNode` needs a case for every new AST node type. A token that carries an address must be hashed in that address form, as the existing reference branches do with `cellAddressFromString` and `hash(true)`, never by its raw text — otherwise the same text in another cell reuses an AST built for the first. Anything else the parse result depends on must be in the key too. +- **The cache is load-bearing.** `ParserWithCaching` does not key on the formula string. `computeHashFromTokens` hashes the lexer tokens with each reference in the AST's address form — relative parts as offsets from the formula's address, `$` parts absolute — and every function name except `OFFSET` made canonical (`OFFSET` has its own token and is hashed as typed, less any prefix described below); rewritten ASTs are stored and fetched under `computeHashFromAst`. The two must give the same key for the same formula, or `fetchCachedResultForAst` throws `There is no AST with such key in the cache`. A new token kind must hash to the same text in both: `computeHashFromTokens` falls back to the raw token image, so it needs its own branch whenever that image is not canonical, and `computeHashOfAstNode` needs a case for every new AST node type. A token that carries an address must be hashed in that address form, as the existing reference branches do with `cellAddressFromString` and `hash(true)`, never by its raw text — otherwise the same text in another cell reuses an AST built for the first. Anything else the parse result depends on must be in the key too. +- **A function name can carry an Excel internal prefix.** Excel writes `_xlfn.`, `_xlws.` and similar prefixes into `.xlsx` files (`EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN` in `parser-consts.ts`). The `ProcedureName` and `OffsetProcedureName` tokens accept one, and `canonicalProcedureNameFromToken` and `canonicalOffsetProcedureNameFromToken` drop it. Read a function name from a token through those helpers, never from the raw image. - **Address arithmetic lives in the `*Address.ts` files.** Do not reimplement relative-to-absolute conversion at a call site. - **A parse failure is a value.** It produces a `ParsingErrorVertex` in the dependency graph, not a thrown exception. One bad formula must not take down the engine. diff --git a/hyperformula/src/parser/FormulaParser.ts b/hyperformula/src/parser/FormulaParser.ts index 82c70e1c1d..9783b8c5bd 100644 --- a/hyperformula/src/parser/FormulaParser.ts +++ b/hyperformula/src/parser/FormulaParser.ts @@ -69,6 +69,7 @@ import { ArrayLParen, ArrayRParen, BooleanOp, + canonicalProcedureNameFromToken, CellReference, ColumnRange, ConcatenateOp, @@ -157,8 +158,7 @@ export class FormulaParser extends EmbeddedActionsParser { */ private procedureExpression: AstRule = this.RULE('procedureExpression', () => { const procedureNameToken = this.CONSUME(ProcedureName) as ExtendedToken - const procedureName = procedureNameToken.image.toUpperCase().slice(0, -1) - const canonicalProcedureName = this.lexerConfig.functionMapping[procedureName] ?? procedureName + const canonicalProcedureName = canonicalProcedureNameFromToken(procedureNameToken.image, this.lexerConfig.functionMapping) const args: Ast[] = [] let argument = this.SUBRULE(this.booleanExpressionOrEmpty) diff --git a/hyperformula/src/parser/LexerConfig.ts b/hyperformula/src/parser/LexerConfig.ts index 72fc4bb3b6..5513024c5d 100644 --- a/hyperformula/src/parser/LexerConfig.ts +++ b/hyperformula/src/parser/LexerConfig.ts @@ -9,6 +9,7 @@ import {ParserConfig} from './ParserConfig' import { ALL_WHITESPACE_PATTERN, COLUMN_REFERENCE_PATTERN, + EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN, NON_RESERVED_CHARACTER_PATTERN, ODFF_WHITESPACE_PATTERN, RANGE_OPERATOR, @@ -47,7 +48,58 @@ export const RangeSeparator = createToken({ name: 'RangeSeparator', pattern: new export const ColumnRange = createToken({ name: 'ColumnRange', pattern: new RegExp(`${COLUMN_REFERENCE_PATTERN}${RANGE_OPERATOR}${COLUMN_REFERENCE_PATTERN}`) }) export const RowRange = createToken({ name: 'RowRange', pattern: new RegExp(`${ROW_REFERENCE_PATTERN}${RANGE_OPERATOR}${ROW_REFERENCE_PATTERN}`) }) -export const ProcedureName = createToken({ name: 'ProcedureName', pattern: new RegExp(`([${UNICODE_LETTER_PATTERN}][${NON_RESERVED_CHARACTER_PATTERN}]*)\\(`) }) +export const ProcedureName = createToken({ name: 'ProcedureName', pattern: new RegExp(`(?:${EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN})?([${UNICODE_LETTER_PATTERN}][${NON_RESERVED_CHARACTER_PATTERN}]*)\\(`) }) + +const excelInternalFunctionPrefixRegexp = new RegExp(`^(?:${EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN})`) +const UNDERSCORE_CHAR_CODE = '_'.charCodeAt(0) + +/** + * Strips one of Excel's internal function-name prefixes (see EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN) + * off the front of a token image, if present. + * + * Every prefix starts with `_`, and this function is called on every ProcedureName and + * OffsetProcedureName token in every parse, so the common case — a name with no prefix at all — takes + * a plain character check instead of always paying for the regex match-and-fail. + * + * @param {string} nameWithoutTrailingParen - a token image with any trailing `(` already removed + */ +function stripExcelInternalFunctionPrefix(nameWithoutTrailingParen: string): string { + if (nameWithoutTrailingParen.charCodeAt(0) !== UNDERSCORE_CHAR_CODE) { + return nameWithoutTrailingParen + } + return nameWithoutTrailingParen.replace(excelInternalFunctionPrefixRegexp, '') +} + +/** + * Reads the canonical function name out of a ProcedureName token. + * + * The token image spans the whole match, so it carries the trailing opening parenthesis and, for a + * formula imported from an .xlsx file, one of the prefixes Excel prepends when it serializes a + * workbook. Dropping both here is what lets `_xlfn.IFS(A1)` resolve to the same function as `IFS(A1)`. + * + * The prefix is removed before the name is upper-cased, because the prefixes are matched in lower + * case only. + * + * @param {string} image - image of the ProcedureName token, for example `_xlfn.IFS(` + * @param {Record} functionMapping - maps a translated function name to its canonical English name + */ +export function canonicalProcedureNameFromToken(image: string, functionMapping: Record): string { + const procedureName = stripExcelInternalFunctionPrefix(image.slice(0, -1)).toUpperCase() + return functionMapping[procedureName] ?? procedureName +} + +/** + * Strips an Excel internal function-name prefix off an OffsetProcedureName token's image. + * + * OffsetProcedureName has no trailing parenthesis to remove (unlike ProcedureName — OFFSET's grammar + * rule consumes `(` separately) and no translation lookup (the token pattern already embeds the + * localized OFFSET name), so it needs only the prefix stripped, not the full canonicalization above. + * + * @param {string} image - image of the OffsetProcedureName token, for example `_xlfn.OFFSET` + */ +export function canonicalOffsetProcedureNameFromToken(image: string): string { + return stripExcelInternalFunctionPrefix(image) +} const cellReferenceMatcher = new CellReferenceMatcher() export const CellReference = createToken({ @@ -93,7 +145,22 @@ export const buildLexerConfig = (config: ParserConfig): LexerConfig => { const ArrayRowSeparator = createToken({name: 'ArrayRowSep', pattern: config.arrayRowSeparator}) const ArrayColSeparator = createToken({name: 'ArrayColSep', pattern: config.arrayColumnSeparator}) const NumberLiteral = createToken({ name: 'NumberLiteral', pattern: new RegExp(`(([${config.decimalSeparator}]\\d+)|(\\d+([${config.decimalSeparator}]\\d*)?))(e[+-]?\\d+)?`) }) - const OffsetProcedureName = createToken({ name: 'OffsetProcedureName', pattern: new RegExp(offsetProcedureNameLiteral, 'i') }) + // OFFSET has its own token because it has its own grammar rule, so it needs the prefix handling of + // ProcedureName repeated here. The 'i' flag exists for the translated OFFSET name (pre-dates this + // prefix support) and applies to the whole pattern, so unlike ProcedureName it also makes the prefix + // case-insensitive: `_XLFN.OFFSET(...)` is accepted here where `_XLFN.SUM(...)` is a parsing error. + // Left as-is rather than given a dedicated case-sensitive-prefix matcher: OFFSET predates the OOXML + // cutoff this whole prefix scheme exists for, so Excel can never actually emit a prefixed OFFSET call + // in any case — the asymmetry has no reachable real input, only hand-typed formulas (see the + // "an upper-cased prefix is accepted on OFFSET, unlike everywhere else" test). + // Unlike ProcedureName, this token does not include the '(' itself, so the prefixed form must look + // ahead for one: otherwise it would claim the start of a named expression such as `_xlfn.OFFSET_RATE`, + // which lexes as a single NamedExpression without prefix support. The unprefixed form keeps its + // pre-existing pattern unchanged. + const OffsetProcedureName = createToken({ + name: 'OffsetProcedureName', + pattern: new RegExp(`(?:${EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN})${offsetProcedureNameLiteral}(?=(?:${whitespaceTokenRegexp.source})?\\()|${offsetProcedureNameLiteral}`, 'i'), + }) let ArgSeparator: TokenType let inject: TokenType[] diff --git a/hyperformula/src/parser/ParserWithCaching.ts b/hyperformula/src/parser/ParserWithCaching.ts index a116aa8e56..e8f8eaa157 100644 --- a/hyperformula/src/parser/ParserWithCaching.ts +++ b/hyperformula/src/parser/ParserWithCaching.ts @@ -19,6 +19,8 @@ import {Cache} from './Cache' import {FormulaLexer, FormulaParser, ExtendedToken} from './FormulaParser' import { buildLexerConfig, + canonicalOffsetProcedureNameFromToken, + canonicalProcedureNameFromToken, CellReference, ColumnRange, LexerConfig, @@ -239,9 +241,10 @@ export class ParserWithCaching { hash = hash.concat(cellAddress.hash(true)) } } else if (tokenMatcher(token, ProcedureName)) { - const procedureName = token.image.toUpperCase().slice(0, -1) - const canonicalProcedureName = this.lexerConfig.functionMapping[procedureName] ?? procedureName + const canonicalProcedureName = canonicalProcedureNameFromToken(token.image, this.lexerConfig.functionMapping) hash = hash.concat(canonicalProcedureName, '(') + } else if (tokenMatcher(token, this.lexerConfig.OffsetProcedureName)) { + hash = hash.concat(canonicalOffsetProcedureNameFromToken(token.image)) } else if (tokenMatcher(token, ColumnRange)) { const [start, end] = token.image.split(':') const startAddress = columnAddressFromString(start, baseAddress, this.resolveSheetReference) diff --git a/hyperformula/src/parser/parser-consts.ts b/hyperformula/src/parser/parser-consts.ts index 2572a8f551..6b0d05104c 100644 --- a/hyperformula/src/parser/parser-consts.ts +++ b/hyperformula/src/parser/parser-consts.ts @@ -22,6 +22,31 @@ export const ROW_REFERENCE_PATTERN = `(${SHEET_NAME_PATTERN})?\\${ABSOLUTE_OPERA export const R1C1_CELL_REFERENCE_PATTERN = '[rR][0-9]*[cC][0-9]*' export const CELL_REFERENCE_WITH_NEXT_CHARACTER_PATTERN = `(${CELL_REFERENCE_PATTERN})[^${NON_RESERVED_CHARACTER_PATTERN}]` +/** + * Prefixes that Excel prepends to a function name when it serializes a workbook. + * + * Excel marks every function added after the original OOXML specification (~Excel 2007) with one + * of these prefixes in the stored XML, so a file may contain `_xlfn.IFS(...)` where the user typed + * `IFS(...)`. They are serialization artifacts rather than a part of the function name, and other + * spreadsheet engines drop them on import, so HyperFormula accepts and ignores them too. + * + * | Prefix | Meaning | + * |----------------|----------------------------------------------------------| + * | `_xlfn.` | function newer than the OOXML specification | + * | `_xlfn._xlws.` | as above, and callable only in a worksheet | + * | `_xlws.` | callable only in a worksheet | + * | `_xlpm.` | `LAMBDA`/`LET` parameter name | + * | `_xludf.` | user-defined function (a `LAMBDA` stored in Name Manager) | + * + * The alternatives are ordered longest-first so that `_xlfn._xlws.` is consumed whole instead of + * leaving `_xlws.` behind. Excel always writes them in lower case, so wherever this pattern is used + * to recognize a *function call* (the `ProcedureName` token in LexerConfig.ts) it is matched in lower + * case only: an upper-cased spelling there is not something Excel can produce. The one exception is + * the `OffsetProcedureName` token, which reuses this pattern inside an already case-insensitive regex + * (for the translated OFFSET name) and so matches an upper-cased prefix too — see its comment. + */ +export const EXCEL_INTERNAL_FUNCTION_PREFIX_PATTERN = '_xlfn\\._xlws\\.|_xlfn\\.|_xlws\\.|_xlpm\\.|_xludf\\.' + export const NAMED_EXPRESSION_PATTERN = `[${UNICODE_LETTER_PATTERN}_][${NON_RESERVED_CHARACTER_PATTERN}]*` export const ALL_DIGITS_ARRAY = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']