Repository navigation
Generate the parser from doc/grammars with phplrt - #325
Closed
ondrejmirtes wants to merge 4 commits into
Closed
ondrejmirtes wants to merge 4 commits into
ondrejmirtes wants to merge 4 commits into
Conversation
ConstExprParser, TypeParser and PhpDocParser are no longer written by hand. tools/phplrt/build.php compiles doc/grammars/phpdoc.pp3 with phplrt and writes every rule of the compiled grammar down as a method of src/Parser/Generated/GeneratedParser.php, which reads exactly what the phplrt runtime would read with the same tables and reports the same failures. The AST is built by the reducers of AstBuilder, which extends the generated class. The public API of the three parsers is unchanged. Some details of exceptions thrown for invalid input differ, and a few public helper methods of TypeParser are removed (see UPGRADING.md). Amp-Thread-ID: https://ampcode.com/threads/T-01a17820-94d1-75c8-9206-2e802d9c0556
Lexer::tokenize() now classifies the tokens for the generated parser while lexing, so the parsers no longer rebuild the stream on every read. Amp-Thread-ID: https://ampcode.com/threads/T-01a17820-94d1-75c8-9206-2e802d9c0556
The reducers are given the first and last token index as two ints computed inline, instead of a phplrt Context filled by a call per reduction. Attributes are skipped without a call when none are used, and a reading no longer saves and restores the state of a previous one. Amp-Thread-ID: https://ampcode.com/threads/T-01a17820-94d1-75c8-9206-2e802d9c0556
The generator leaves out token checks the surrounding code already guarantees, and builds children with array_slice() now that no reducer pushes a list onto the value stack. Amp-Thread-ID: https://ampcode.com/threads/T-01a17820-94d1-75c8-9206-2e802d9c0556
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proof of concept: the parser is generated from the grammars in
doc/grammarswith phplrt instead of being written by hand. Meant for the next major version.How it works
tools/phplrt/build.php(make grammars-build, PHP 8.4) compilesdoc/grammars/phpdoc.pp3with phplrt 4.2.4 and passes the compiled grammar totools/phplrt/ParserGenerator.php.src/Parser/Generated/GeneratedParser.php, a PHP 7.4 class with one method per rule. Each method reads its rule exactly the way phplrt'sRecursiveDescentTracerwould with the same prediction tables, including which tokens are reported as expected on failure. Small rules are inlined, and lexemes and token classes become plainisset()checks.AstBuilderextends the generated class. Itsreduce*methods build the AST eagerly while parsing, instead of recording a trace and reducing it afterwards. Every rule exists twice, as a building method and a syntax-only one. Predicates, rules whose reducer doesn't read its children, and error explanations use the syntax-only method.ConstExprParser,TypeParserandPhpDocParserare thin wrappers with the same constructors andparse*()methods as before.Compatibility
ParserExceptiondetails for invalid input changed (the token where the parser stopped, or the expected token); see the test diff.TypeParserpublic helpers are removed:parseGeneric(),parseTemplateTagValue(),isHtml()andenrichWithAttributes(). slevomat/coding-standard and phpstan-src only callparse().ondrejmirtes/phplrt-runtime. With the generated parser, the runtime is only used forPhplrt\Parser\Context, which the reducers receive. Dropping the dependency entirely would be a small change.Performance
Minimum of 4 runs, PHP 8.4 CLI without OPcache, lexing excluded:
vendor//** @var int */@param array<string, int|null> $foo+@return void@paramtags@returnwith array shapes and callablesThe generated parser is 2–4× faster than running the grammar through the phplrt runtime, but on typical PHPDocs it is still about 1.5× slower than the hand-written parser. The exception is pathological input, where it is about 2× faster. What remains is generic value building: a value stack, merging child values and a reducer call per kept rule. No single hotspot is left.