R1 responses are truncated

[akaihola]

I installed newest versions of llm, llm-together and files-to-prompt, added my API key and tried it out. First it seems to work fine, and the thinking and reponse begin streaming ok, but the output is always cut short after approx. 140 to 200 words:

[akaihola@gogo:/tmp]$ files-to-prompt -c . | llm -m deepseek-ai/DeepSeek-R1 \
"Inspect the code and find out why responses from the TogetherAI DeepSeek R1 endpoint are \
truncated after 140 to 200 words."
 The code is supposed to handle the response correctly, but there's an issue where the response
stops prematurely. The problem might be related to how the stop tokens are handled or how the
response is being processed. Analyze the code and determine the root cause of the truncation.
Provide a step-by-step explanation of the issue and suggest a fix.
</think>

Okay, let's try to figure out why the responses from the TogetherAI DeepSeek R1 model are getting
truncated. The user mentioned that after about 140-200 words, the response stops. So I need to
look through the provided code to see where things might be going wrong.

First, looking at the `llm_together.py` file, specifically the `execute` method. The code is
using the Together API to generate responses. Let me check how the stop tokens are handled
because that's a common reason for truncation.

In the `execute` method, there's a line that sets `stop = self.default_stop`, which is "<human>".
Then, if the model has a 'config' with 'stop' defined, it overrides the default. The DeepSeek R1
model might have specific stop tokens that need to be properly configured here. Maybe the
model's config isn't
[akaihola@gogo:/tmp]$ 

[akaihola]

Oh, R1 was able to give me the reason when I asked it through OpenRouter:

[akaihola@gogo:/tmp]$ files-to-prompt -c . | llm -m openrouter/deepseek/deepseek-r1 \
"Inspect the code and find out why responses from the TogetherAI DeepSeek R1 endpoint are \
truncated after 140 to 200 words."

The responses are likely truncated due to the max_tokens parameter in the Together plugin options being set to a default of 256 tokens (approximately 200 words). This limit is defined in the Options class:

# From llm_together.py
class Options(llm.Options):
   max_tokens: Optional[int] = Field(
       description="Maximum number of tokens to generate.", default=256  # <-- This default is too low
   )

Solution: Increase the max_tokens value when calling the model by specifying it explicitly:

llm -m together-model-name "prompt" --max-tokens 1000

Or modify the default in the plugin code with a higher value (e.g., 2000) if you want to handle longer responses by default:

# Change this line in llm_together.py
max_tokens: Optional[int] = Field(description="Maximum number of tokens to generate.", default=2000)

The truncation occurs because the Together API stops generation when either:

1. The response reaches the max_tokens limit
2. A stop sequence is encountered

The current implementation uses the model's suggested stop sequence if available, but the primary limiting factor here is the low token limit in the default configuration.

[akaihola]

So the solution is to specify a larger maximum tokens value on the command line:

[akaihola@gogo:/tmp]$ files-to-prompt -c . \
| llm -m deepseek-ai/DeepSeek-R1 -o max_tokens 100000 \
"Inspect the code and find out why responses from the TogetherAI DeepSeek R1 endpoint are \
truncated after 140 to 200 words."