Slow response times

[blevels]

Which model?

Gemma 4 31B

Mac model + RAM

M1 Max 64GB

What happened?

Thanks for creating this repository with the capability I have been desiring for some time. I have one issue that I am not able to resolve (yet). Any assisistance you can provide would be greatly appreciated.

When prompting via the claude CLI, the responses are very slow (78.6s / 3m 23s) and the response doesn't return back to the claude CLI prompt immediately even through I see the response in the server logs. BTW...In order to get things to run properly without prompting for a login to Anthropic, I added the following to my ~.claude.json:

"primaryApiKey": "sk-local"

Any ideas why the responses are very slow? I tried adding the following setting to my ~/.claude/settings.json:

"CLAUDE_CODE_ATTRIBUTION_HEADER" : "0"

This did not change the slowness of the responses. In the posted screenshot you can see the overall time it took for the response was 3m 23s. In the logs, you can see tha tthe response was generated @ 06:55:23, however, in the last log you can see that it was not returned to the CLI until 06:57:03. I also notice the fan on my Mac M1 going crazy. I wonder if this machine has enough power to run the model.

Relevant logs or error output

[nicedreamzapp]

Hey, thanks for trying this out — and nice catch on the primaryApiKey: sk-local workaround, that's a legit fix for the login prompt.

The slowness has two parts:

1. Generation speed (78s for the response itself)
Gemma 4 31B on M1 Max 64GB will work but it's pushing it. The M1 Max memory bandwidth is ~400 GB/s vs ~800 GB/s on newer M4 Max chips, and LLM inference is almost entirely memory-bandwidth-bound. So 31B will always feel sluggish on M1. Two options:

• Drop to a smaller model — Qwen 3.5 4B or Gemma 4 12B will be dramatically faster on your hardware and still very capable for coding tasks
• Reduce context length — if you want to stick with 31B, lowering the max context in the Ollama modelfile helps

2. The ~90s gap between server response and CLI display
This is likely Claude Code's attribution/tool-use processing loop. When running in --bare mode with a local backend, Claude Code still tries to do some post-processing that can stall. I'm looking into optimizing this. For now the generation speed improvement from a smaller model will help the most.

My recommendation: try Gemma 4 12B — it'll saturate your M1 Max bandwidth much better and response times should drop to single-digit seconds.

[holgerfriesz]

Hi there,

you mentioned "Drop to a smaller model — Qwen 3.5 4B or Gemma 4 12B " --
Where can I download / install automatically the smaller models? I checked the files and realized that the smallest model in the scripts is the 31GB one.

many thanks!!