Extracting a literal out of a Mage rule changes semantics

[samvv]

If we're not careful about documenting certain features, we might trigger unexpected behavior.

In particular, one could argue that converting a literal to a token in a certain rule means that skip will be run before and after the token. The original rule did not make use of skip. Therefore the mage_extract_literals pass changes the semantics of the grammar.

The previous bug can be documented into a feature. The question is whether this is desirable.

[samvv]

We are going to conditionally enable tokenization based on the presence of a pub token rule in the grammar. If the user wishes to enable tokenization on a pure parsing grammar then a special directive might be provided, such as @!tokenize(on).