OmniToken

Pure-Go LLM tokenizer library for local token counting, encoding, and decoding with OpenAI-compatible BPE, custom vocabularies, and provider adapters.

_{If OmniToken is useful in your Go project, a star helps others discover it.}

OmniToken is built for Go services that need fast local token accounting for prompt sizing, context-window planning, tokenizer experiments, and cacheflow analysis without CGO, Rust, or Python runtime dependencies in the root module.

The root module supports Go 1.23+. Some optional comparison tooling in this repository uses dependencies that require newer Go versions.

Features

• Pure Go tokenizer library for LLM applications.
• OpenAI-compatible BPE token counting for cl100k_base, o200k_base, and o200k_harmony.
• Local Encode, EncodeOrdinary, CountTokens, and Decode APIs.
• Zero-allocation CountTokens hot path for supported OpenAI BPE workloads.
• Custom WordPiece and SentencePiece-style vocabularies.
• cacheflow package for prompt-cache boundary and trace analysis.
• Optional adapter modules for Gemini, Llama 3, Mistral, Hugging Face tokenizer.json, OSS SentencePiece models, and Anthropic message token counting.

Benchmarks

Measured on Windows 11 amd64, Intel i7-1250U, Go 1.24.2. See the full benchmark report or regenerate it with benchmarks/.

Comparison	Geomean result
OmniToken CountTokens vs tiktoken-go	15.84x faster
OmniToken EncodeOrdinary vs tiktoken-go	13.09x faster
OmniToken Decode vs tiktoken-go	2.29x faster
OmniToken CountTokens vs OpenAI Rust tiktoken	0.96x, near parity
OmniToken EncodeOrdinary vs OpenAI Rust tiktoken	0.75x, competitive pure Go

Operation	Encoding	Input	ns/op	B/op	allocs/op
CountTokens	cl100k_base	JSON	1,517	0	0
EncodeOrdinary	cl100k_base	JSON	1,661	288	1
CountTokens	o200k_base	JSON	2,152	0	0
EncodeOrdinary	o200k_base	JSON	1,835	288	1

Install

go get github.com/ron2111/omnitoken

Quick Start

package main

import (
	"fmt"

	"github.com/ron2111/omnitoken"
)

func main() {
	engine, err := omnitoken.ForModel("gpt-4o")
	if err != nil {
		panic(err)
	}

	tokens := engine.EncodeOrdinary("hello world")
	count := engine.CountTokens("hello world")
	text := engine.Decode(tokens)

	fmt.Println(count, text)
}

Special Tokens

EncodeOrdinary always treats marker strings such as <|start|> as normal text. For built-in OpenAI BPE engines, Encode follows tiktoken-style special-token handling: known special markers are rejected by default unless explicitly allowed.

tokens, err := omnitoken.Encode(engine, "<|start|>assistant", omnitoken.EncodeOptions{
	AllowAllSpecial: true,
})

Exact Harmony prompt accounting requires special-token-aware encoding or manually inserted special token IDs. Plain string concatenation with EncodeOrdinary is ordinary text encoding only.

Use SpecialTokenID or SpecialTokens on *omnitoken.Engine when constructing token sequences directly.

Support

Family	Status
OpenAI cl100k_base	Supported
OpenAI o200k_base	Supported
OpenAI o200k_harmony	Supported, including Harmony special-token IDs
WordPiece local vocabularies	Supported
SentencePiece-style local vocabularies	Supported
Gemini local text adapter	Optional module
OSS SentencePiece adapter	Optional module
Llama 3 tiktoken-BPE adapter	Optional module
Mistral Tekken adapter	Optional module
Hugging Face WordPiece adapter	Optional module
Anthropic message counter	Optional module

Cacheflow

import "github.com/ron2111/omnitoken/cacheflow"

engine, err := omnitoken.ForModel("gpt-4o")
if err != nil {
	panic(err)
}

report := cacheflow.NewAligner(engine).AlignPromptToProfile(
	systemPrompt,
	cacheflow.ProfileOpenAI,
)

Cacheflow is informational: OmniToken does not edit prompts automatically or claim provider billing parity. See cacheflow.

Custom Models

err := omnitoken.RegisterEncoding("my_wordpiece", func() (omnitoken.ModelEngine, error) {
	return omnitoken.NewWordPiece(vocabBytes, omnitoken.WordPieceOptions{Lowercase: true})
})
if err != nil {
	panic(err)
}

err = omnitoken.RegisterModelPrefix("my-model-", "my_wordpiece")

More

• Architecture
• Benchmarks and correctness
• Cacheflow
• CLI
• Adapters

Future Scope

• Stateful streaming token accounting with boundary buffering.
• More provider adapters with verified local vocab sources.
• Release CI for benchmark regression tracking.

License

MIT License. See LICENSE.