Perf: Tokenizer rebuilds operator/literal tries on every instantiation

[Kiryous]

Problem

Every new Tokenizer(...) call rebuilds two tries from scratch via createTrie():

// tokenizer.ts constructor
this.opTrie = createTrie(operators);
this.literalTrie = createTrie(literalValues);

The operators and literalValues inputs are identical across all tokenizer instances for a given Liquid engine. In scenarios where many short template strings are parsed in a single pass (e.g. validating hundreds of {{ variable }} expressions in a document), this creates significant overhead from redundant trie construction.

Profiling data

We're using LiquidJS (v10.25.0) inside Kibana's workflow YAML editor to validate Liquid template expressions on every keystroke. For a large document (~1,500 lines, ~360 template variables), CPU profiling shows:

• createTrie: 173ms self-time — the Conversion to Async #10 hottest function by self-time
• It appears in 3 of the top 15 hot call stacks, all through Tokenizer → createTrie
• Example stack: parseTemplateString → parse → readTopLevelToken → readTagToken → TagToken → Tokenizer → createTrie

We already cache parse() results (LRU by template string), which helps on cache hits. But on cache misses, createTrie is the dominant cost inside the tokenizer — more expensive than the actual tokenization work.

Suggested fix

Cache the tries at the Liquid engine level (or Parser level) and pass them into Tokenizer instances, since the operator set and literal values don't change after engine construction:

// In Parser or Liquid engine — build once
this.opTrie = createTrie(this.options.operators);
this.literalTrie = createTrie(literalValues);

// In Tokenizer constructor — reuse
constructor(input, opTrie, literalTrie, file, range) {
    this.opTrie = opTrie;
    this.literalTrie = literalTrie;
    // ...
}

This would be a non-breaking internal change. The Tokenizer constructor isn't part of the public API contract (users go through Liquid.parse()).

Environment

• LiquidJS v10.25.0
• Browser (bundled via webpack)
• Hundreds of parse() calls per validation pass

[harttle]

This is an ancient feature allowing customized operators which turns out to not make much sense. Let's plan remove it in v11 and make the trie a constant.