Class definition attributes not parsed correctly

[joeced]

I started working with textmate-grammar-python for the https://github.com/sphinx-contrib/matlabdomain package. Thanks for starting this work - it's much more structured than the pygments tokens.

Given this MATLAB class definition

classdef (AllowedSubclasses = {?SubClass1,?SubClass2}) SuperClass
end

is parsed to these tokens with textmate-grammar-python==0.6.0:

{'token': 'source.matlab',
 'children': [{'token': 'meta.class.matlab',
               'begin': [{'token': 'storage.type.class.matlab', 'content': 'classdef'}],
               'end': [{'token': 'storage.type.class.end.matlab', 'content': 'end'}],
               'children': [{'token': 'meta.class.declaration.matlab',
                             'children': [{'token': 'punctuation.section.parens.begin.matlab', 'content': '('},
                                          {'token': 'storage.modifier.class.matlab', 'content': 'AllowedSubclasses'},    
                                          {'token': 'keyword.operator.assignment.matlab', 'content': '='},
                                          {'token': 'punctuation.separator.modifier.comma.matlab', 'content': ','},      
                                          {'token': 'storage.modifier.class.matlab', 'content': 'SubClass2'},
                                          {'token': 'punctuation.section.parens.end.matlab', 'content': ')'},
                                          {'token': 'entity.name.type.class.matlab', 'content': 'SuperClass'}]}]}]}      

As one can see, we are missing:

• Begin and end curly braces { / }.
• SubClass1
• The ? in front of SubClass1 and SubClass2.

[watermarkhu]

Hi @joeced @apozharski

Good to see that this is being picked up.

Regarding the output of textmate grammars, there are two issues:

• MATLAB-Language-grammar still has quite some issues, especially with more advanced or relatively new MATLAB expressions. I have been slowly working my way there to fix some of the issues, but still quite some remain. A quick way to check is to open VScode and run the Developer: Inspect Editor Tokens and Scopes command. If the nested token logic is incorrect there, it is incorrect in the grammar. There are quite a few occasions where the scope is lost but still displayed correctly.
• textmate-grammar-python is still not fully reproduce the output of vscode-textmate. Recently I started a branch with even more regression testing here main...feat/more-regression-testing. I've got an good idea on why this is the case, but no time to work on it in the coming month.

Anyhow, this will still be an ongoing project of mine. Let's keep in touch.

[apozharski]

I have opened a PR against MATLAB-Language-grammar that (mostly, within the limitations of textmate grammars) addresses this issue: mathworks/MATLAB-Language-grammar#88

Your example A slightly more complex example:

classdef (AllowedSubclasses = {?SubClass1,?SubClass2}, Hidden=true) SuperClass < foo ...
    & bar
end

would now be parsed as:

{'token': 'meta.class.declaration.matlab',
 'children': [{'token': 'storage.modifier.section.class.matlab',
               'children': [{'token': 'storage.modifier.class.matlab', 'content': 'AllowedSubclasses'},
                            {'token': 'keyword.operator.assignment.matlab', 'content': '='},
                            {'token': 'meta.cell.literal.matlab',
                             'begin': [{'token': 'punctuation.section.braces.begin.matlab', 'content': '{'}],
                             'end': [{'token': 'punctuation.section.braces.end.matlab', 'content': '}'}],
                             'children': [{'token': 'keyword.operator.other.question.matlab', 'content': '?'},
                                          {'token': 'variable.other.readwrite.matlab', 'content': 'SubClass1'},
                                          {'token': 'punctuation.separator.comma.matlab', 'content': ','},
                                          {'token': 'keyword.operator.other.question.matlab', 'content': '?'},
                                          {'token': 'variable.other.readwrite.matlab', 'content': 'SubClass2'}]},
                            {'token': 'punctuation.separator.modifier.comma.matlab', 'content': ','},
                            {'token': 'storage.modifier.class.matlab', 'content': 'Hidden'},
                            {'token': 'keyword.operator.assignment.matlab', 'content': '='},
                            {'token': 'constant.language.boolean.matlab', 'content': 'true'}]},
              {'token': 'entity.name.type.class.matlab', 'content': 'SuperClass'},
              {'token': 'punctuation.separator.lt.inheritance.matlab', 'content': '<'},
              {'token': 'meta.inherited-class.matlab', 'children': [{'token': 'entity.other.inherited-class.matlab', 'content': 'foo'}]},
              {'token': 'meta.continuation.line.matlab', 'children': [{'token': 'punctuation.separator.continuation.line.matlab', 'content': '...'}]},
              {'token': 'keyword.operator.type.matlab', 'content': '&'},
              {'token': 'meta.inherited-class.matlab', 'children': [{'token': 'entity.other.inherited-class.matlab', 'content': 'bar'}]}]}

[RedCMD]

here's my VSCode extension to help with authoring TextMate grammars https://marketplace.visualstudio.com/items?itemName=RedCMD.tmlanguage-syntax-highlighter