Skip to content

feat: add Android custom text tokenizer#75

Open
werran2 wants to merge 2 commits into
OpenMOSS:mainfrom
werran2:android-custom-text-tokenizer
Open

feat: add Android custom text tokenizer#75
werran2 wants to merge 2 commits into
OpenMOSS:mainfrom
werran2:android-custom-text-tokenizer

Conversation

@werran2

@werran2 werran2 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a small pure Kotlin SentencePiece tokenizer for the exported Nano tokenizer.model
  • support custom text synthesis in the Android ONNX Runtime example UI and engine API
  • keep the existing pre-tokenized demo buttons as smoke-test fallbacks
  • document custom text usage and add tokenizer unit tests, including an optional real-model comparison test

Validation

  • :app:testDebugUnitTest --rerun-tasks
  • MOSS_TOKENIZER_MODEL=/path/to/tokenizer.model :app:testDebugUnitTest --rerun-tasks
  • :app:assembleDebug --rerun-tasks
  • git diff --check

Note: I verified the Android project builds locally, but I did not run a full on-device model synthesis smoke in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant