Introduce token modes

[Lotes]

Draft

What I still need to do

• move main grammar back to use implicit token mode
• add more tests, currently a lot is covered by all "languages", but let's see
• may add more validations for edge cases
• respect the Slack comment link in TODO.md
• remove the TODO.md

What was done

• added explicit token modes (lexer modes)
• if none is given, one is create implicitly
• the (start) default mode is always created
  • if implicit, it collects all keywords, token declarations and token groups from the grammar
• token modes can have following members
  • own token declarations
  • token refs to top-level token declaration
  • own token groups
  • keywords
  • keyword selection using a RegExp
• token commands were also introduced
  • push(name) to push a token mode onto the token mode stack
  • pop to pop the topmost token mode
  • mode(name) to set the topmost token mode
• I also extended token declarations to be keywords as well
• a lot of changes happened in the lexer generator
  • I tidied up the code organization a lot, I hope you like it as well!
  • token types are split from group type and token commands
    • it made sense during the journey in order to write a command/type differently in different modes... we can discuss if it still makes sense
  • "keywords" I renamed conceptually to KeywordSelector

[msujew]

Nice! Can you resolve the merge conflict so we can check the performance in the GitHub CI? Locally, it seems to look good. Only a 2-3% loss on the statemachine example, which is acceptable.

[msujew]

Suggestion: Can you add a benchmark for the lexing performance of the lexer modes?

[Lotes]

I added one with a nested template string. If you want more, tell me what you would like to test.

[Lotes]

@msujew I discovered a problem with my main grammar.

The problem arises when I create a token mode and start to create token declarations within this token mode. As soon as I complete the word "token", the language server seems to end up in a endless loop. It does not react anymore. After completing the declaration and restarting the LS, it works. So the problem is during editing incomplete grammars :-/...

token mode default {
  toke //complete to "token" to see the problem
}

The rule TokenModeMember: TokenDeclUsage | TokenGroupUsage | TokenUsage | KeywordUsage | KeywordSelector is actually having some alternatives with a common (nested) prefix. I hoped that the LL(STAR) is handling this ^^*. Or maybe it is a different problem?

If I would refactor the grammar to have no common prefix in the subrules of TokenModeMember the problem might disappear and the performance would increase.
But it would also make the grammar more difficult I think.

Before I refactor it would be good to discuss whether these members have a good syntax or whether we should alternate it (which could remove the common prefix automatically).

[Lotes]

Ideas...

Some quick fixes or code actions come into my mind:

1. inline a token rule that points to a keyword
2. and the opposite: extract all keywords to a common token rule

Or

3. a hover tooltip over a keyword selector which keywords are included