Skip to content

Fix Gemma stop tokens: read additional EOS from tokenizer config#36

Merged
stikves merged 1 commit into
apple:mainfrom
stikves:sukru/gemma-stop-tokens
Jun 14, 2026
Merged

Fix Gemma stop tokens: read additional EOS from tokenizer config#36
stikves merged 1 commit into
apple:mainfrom
stikves:sukru/gemma-stop-tokens

Conversation

@stikves

@stikves stikves commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

This fixes #25

The root cause is upstream not exposing the required fields, and eventually we should try to have them fix this at the origin. However we can also manually read the tokenizer additional information.

% swift run -c release llm-runner --model exports/gemma_3_4b_it_4bit_dynamic --max-tokens 250
Building for production...
[1 / 1]

Build complete! (1.10 sec)

⏳ Preparing AI asset from source... done in 0.571s

Generating...
I’m doing well, thank you for asking! As a large language model, I don’t really *feel* in the same way humans do, but my systems are running smoothly and I’m ready to chat. 😊

How are *you* doing today? Is there anything you'd like to talk about or any tasks you'd like me to help with?

⏱️  Performance Summary:
==================================================
Model Load: 571.2ms
Prompt:     84.9ms, 15 tokens, 176.6 tokens/sec
Generation: 1873.1ms, 79 tokens, 42.2 tokens/sec
Total:      8.984s
==================================================

@stikves stikves self-assigned this Jun 13, 2026
@stikves stikves force-pushed the sukru/gemma-stop-tokens branch from 341e5d8 to 7751bde Compare June 13, 2026 02:43
@stikves stikves force-pushed the sukru/gemma-stop-tokens branch from 7751bde to 9de0e19 Compare June 13, 2026 02:44
@stikves stikves changed the title Fix Gemma stop tokens: read additional EOS from tokenizer config (#25) Fix Gemma stop tokens: read additional EOS from tokenizer config Jun 13, 2026
@stikves stikves merged commit 02a8edd into apple:main Jun 14, 2026
3 checks passed
@stikves stikves deleted the sukru/gemma-stop-tokens branch June 14, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemma 3 generation does not stop at <end_of_turn>

2 participants