RKLLM backend uses NPU correctly after rkllama#4 fix, but model enters an unexpected generation loop

[mandresve]

Environment

• Version: beta.35 / latest testing branch
• Platform: Orange Pi 5 Pro (RK3588, 16GB)
• OS: Debian 12 (Bookworm) ARM64
• Backend: rkllama with manual application of rkllama#4 fix
• Model tested: qwen2.5-1.5b-rkllm

Description

After applying the fix from rkllama#4 manually, the RKLLM backend is now correctly using the NPU and inference starts successfully.

This confirms that the previous backend initialization issues are resolved.

However, when running a taOS Agent instance with qwen2.5-1.5b-rkllm, the model appears to enter an unexpected generation loop. The generated output does not correspond to the user prompt and instead starts producing unrelated task/planning text repeatedly.

This behavior appears to be generated by the model itself rather than a backend availability issue.

Steps to Reproduce

1. Apply the rkllama#4 fix manually.
2. Start taOS Agent.
3. Select qwen2.5-1.5b-rkllm.
4. Send a simple prompt: "Test"
5. Observe the generated response.

Expected Behavior

• The model should return a normal response to the user prompt.
• For a simple prompt like "Test" the model should generate a short conversational response.

Actual Behavior

• The backend successfully receives the request.
• RKLLM starts inference.
• NPU execution works.
• The request returns HTTP 200.
• However, the model generates unrelated structured task output and appears to continue an internal workflow instead of answering the prompt.

The output resembles an agent/task state template, for example:

• Objective
• Important Details
• Work State
• Completed
• Active
• Blocked
• Next Move

This continues until the context limit is reached.

Additional Observations

The backend reports that the prompt size exceeds the model context window: Prompt is 11543 tokens but the model context window is 4096 tokens. Later requests also show large prompt sizes.

This may indicate that:

• taOS Agent is sending an unexpectedly large system prompt/history.
• The model context configuration is not aligned with the model capabilities.
• The model is receiving an agent-oriented prompt format that is incompatible with this specific RKLLM conversion.

Current Status

The following issues appear resolved:

• RKLLM backend starts correctly.
• NPU is detected and used.
• Model inference executes successfully.

The remaining issue seems to be related to the interaction between:

• taOS Agent prompt formatting
• model context size
• qwen2.5-1.5b-rkllm behavior

Logs

Jul 08 09:00:47 orangepi5pro python[1272]: 2026-07-08 09:00:47,702 - rkllama.worker - INFO - Worker for model qwen2.5-1.5b-rkllm created and running...
Jul 08 09:00:48 orangepi5pro python[1272]: 2026-07-08 09:00:48,465 - rkllama.server_utils - WARNING - Prompt is 11543 tokens but the model context window is 4096 tokens. Reduce the prompt or conversation history, or increase num_>
Jul 08 09:00:48 orangepi5pro python[1272]: 2026-07-08 09:00:48,466 - werkzeug - INFO - 127.0.0.1 - - [08/Jul/2026 09:00:48] "POST /api/chat HTTP/1.1" 400 -
Jul 08 09:00:50 orangepi5pro python[1272]: 2026-07-08 09:00:50,182 - rkllama.server_utils - WARNING - Prompt is 11543 tokens but the model context window is 4096 tokens. Reduce the prompt or conversation history, or increase num_>
Jul 08 09:00:50 orangepi5pro python[1272]: 2026-07-08 09:00:50,183 - werkzeug - INFO - 127.0.0.1 - - [08/Jul/2026 09:00:50] "POST /api/chat HTTP/1.1" 400 -
Jul 08 09:00:52 orangepi5pro python[1272]: 2026-07-08 09:00:52,649 - rkllama.server_utils - WARNING - Prompt is 11543 tokens but the model context window is 4096 tokens. Reduce the prompt or conversation history, or increase num_>
Jul 08 09:00:52 orangepi5pro python[1272]: 2026-07-08 09:00:52,649 - werkzeug - INFO - 127.0.0.1 - - [08/Jul/2026 09:00:52] "POST /api/chat HTTP/1.1" 400 -
Jul 08 09:00:55 orangepi5pro python[2902]: 2026-07-08 09:00:55,893 - rkllama.worker - INFO - Running inference for model qwen2.5-1.5b-rkllm...
Jul 08 09:00:55 orangepi5pro python[2902]: 2026-07-08 09:00:55,894 - rkllama.rkllm - WARNING - Not found expected prompt cache file for fast inference: /root/rkllama/models/qwen2.5-1.5b-rkllm/cache/AHZ1dJl-E6lK3ZcCP9AMeS9tNF3PDQQ>
Jul 08 09:00:57 orangepi5pro python[2902]: I rkllm: saved prompt cache to /root/rkllama/models/qwen2.5-1.5b-rkllm/cache/AHZ1dJl-E6lK3ZcCP9AMeS9tNF3PDQQgQw5CSI9TNKQ_005
Jul 08 09:00:57 orangepi5pro python[2902]: ##
Jul 08 09:00:57 orangepi5pro python[1272]: 2026-07-08 09:00:57,435 - werkzeug - INFO - 127.0.0.1 - - [08/Jul/2026 09:00:57] "POST /api/chat HTTP/1.1" 200 -
Jul 08 09:00:57 orangepi5pro python[2902]:  Objective
Jul 08 09:00:58 orangepi5pro python[2902]: - [Create a new function to generate JSON data based on input parameters.]
Jul 08 09:01:42 orangepi5pro python[2902]: ## Important Details
Jul 08 09:01:42 orangepi5pro python[2902]: - (none)
Jul 08 09:01:43 orangepi5pro python[2902]: ## Work State
Jul 08 09:01:43 orangepi5pro python[2902]: ### Completed
Jul 08 09:01:44 orangepi5pro python[2902]: - The initial version of the function successfully generates a single sample JSON object.
Jul 08 09:01:45 orangepi5pro python[2902]: - However, it lacks comprehensive error handling and validation checks.
Jul 08 09:01:45 orangepi5pro python[2902]: ### Active
Jul 08 09:01:46 orangepi5pro python[2902]: - Current code snippet for generating JSON data is ready.
Jul 08 09:01:46 orangepi5pro python[2902]: ### Blocked
Jul 08 09:01:47 orangepi5pro python[2902]: -
Jul 08 09:01:47 orangepi5pro python[1272]: 2026-07-08 09:01:47,091 - werkzeug - INFO - 127.0.0.1 - - [08/Jul/2026 09:01:47] "GET /api/tags HTTP/1.1" 200 -
Jul 08 09:01:47 orangepi5pro python[2902]:  There are no known blockages
Jul 08 09:01:47 orangepi5pro python[1272]: 2026-07-08 09:01:47,571 - werkzeug - INFO - 127.0.0.1 - - [08/Jul/2026 09:01:47] "GET /api/tags HTTP/1.1" 200 -
Jul 08 09:01:48 orangepi5pro python[2902]:  preventing the process from continuing.
Jul 08 09:01:48 orangepi5pro python[2902]: ### Next Move
Jul 08 09:01:50 orangepi5pro python[2902]: 1. Implement comprehensive error handling to manage potential issues such as invalid input or network errors during data retrieval.
Jul 08 09:01:52 orangepi5pro python[2902]: 2. Add additional validation logic to ensure the generated JSON adheres to specific schema requirements for all possible scenarios.
Jul 08 09:01:52 orangepi5pro python[2902]: 2026-07-08 09:01:52,056 - rkllama.worker - INFO - Clearing KV cache for model qwen2.5-1.5b-rkllm...
Jul 08 09:01:53 orangepi5pro python[1272]: 2026-07-08 09:01:53,009 - rkllama.server_utils - WARNING - Prompt is 7849 tokens but the model context window is 4096 tokens. Reduce the prompt or conversation history, or increase num_ctx.
...
...
(endless loop)

[kilo-code-bot]

This issue appears to be a duplicate of #1732.

RKLLM fails when prompt exceeds max context, causing worker crash and EOFError (#1732)

Similarity score: 91%

This comment was generated by Kilo Auto-Triage.

[mandresve]

Update

While testing the agent path further, I found that the issue is not exclusive to qwen2.5-1.5b-rkllm. Testing with another available model (qwen3-embedding-0.6b) produces similar unexpected behavior: the agent starts generating endless analysis text instead of a normal response.

This suggests that the problem may not be related only to a specific model conversion, but potentially to the agent initialization flow, prompt formatting, or system prompt configuration.

Current observations

• RKLLM backend:

  • Starts correctly.
  • Uses the NPU correctly.
  • Performs inference successfully.
  • Returns HTTP 200 responses.

• The issue happens after inference starts:

  • The model generates repeated internal reasoning-like output.
  • The response is not aligned with the user prompt.
  • The agent eventually fails with transport errors due to the excessive/unbounded output.

Example behavior:

analysis analysis analysis analysis analysis
analysis analysis analysis analysis analysis
analysis分析analysis analysis analysis

New hypotheses

At this point, the following hypotheses seem possible:

1. Incorrect system prompt / agent initialization

   The model may be receiving an unexpected system prompt that forces it into an internal planning or reasoning format instead of a user-facing chat response.

2. Wrong model role configuration

   Some models (especially embedding models) are not intended for chat completion. If taOS exposes them as selectable agent models, the resulting behavior may be undefined.

3. Chat template mismatch

   The RKLLM conversion may require a specific chat template or prompt format. If taOS sends a generic OpenAI-style chat payload without adapting the template, the model may not understand the expected interaction format.

4. Context window / prompt size issue

   Previous logs showed very large prompts compared with the model context window: Prompt is 11543 tokens but the model context window is 4096 tokens. This could indicate that the agent initialization prompt plus conversation history is too large, causing unexpected model behavior.

5. Model initialization parameters

   Parameters such as:

• num_ctx

• system prompt

• stop tokens

• temperature

• max tokens

• chat template

  may need validation for RKLLM-based models.

Requested investigation

Could you confirm the expected initialization flow for taOS Agent?

Specifically:

• What exact system prompt is sent to the model during agent initialization?
• Is there a way to inspect the final prompt payload sent from LiteLLM/OpenCode to rkllama?
• Are embedding models expected to appear as selectable agent models?
• Are there recommended model parameters or chat templates for RKLLM chat models?

I can provide additional logs if needed. At this point the backend/NPU path appears healthy, so the remaining issue seems to be in the agent-model interaction layer.

[mandresve]

Update: Direct API test successful with qwen2.5-1.5b-rkllm

After further investigation, I tested the rkllama backend directly, bypassing the taOS Agent layer.

The direct /api/chat endpoint works correctly with qwen2.5-1.5b-rkllm.

Test performed:

curl -s http://127.0.0.1:7833/api/chat \
  -d '{"model":"qwen2.5-1.5b-rkllm","messages":[{"role":"user","content":"Hello"}]}'

Response:

{
  "created_at": "2026-07-08T11:05:41.996960Z",
  "done": true,
  "done_reason": "stop",
  "eval_count": 25,
  "eval_duration": 1807,
  "load_duration": 100000000,
  "message": {
    "content": "I am an AI assistant created by Anthropic to be helpful, harmless, and honest. How can I help you today?",
    "role": "assistant"
  },
  "model": "qwen2.5-1.5b-rkllm",
  "prompt_eval_count": 34,
  "prompt_eval_duration": 176,
  "total_duration": 1996005535
}

This confirms that:

• The rkllama backend is able to load the model.
• The NPU inference path is working.
• The /api/chat endpoint is returning valid responses.
• The issue is likely not related to RKLLM initialization or the NPU driver.

The remaining suspicious behavior seems to be at the taOS Agent / OpenCode integration layer, possibly related to:

• Agent prompt construction.
• System prompt size or formatting.
• Context window management.
• Handling of streamed responses.
• How taOS processes long-running or malformed model outputs.

Additional observation:

When testing through taOS Agent, some models entered an apparent endless analysis loop, while the rkllama backend itself continued generating output. The UI appeared frozen until the backend services were restarted.

This suggests that the backend inference is functional, but the agent layer may not correctly handle:

• Very long outputs.
• Models exceeding context limits.
• Non-terminating generations.
• Streaming updates from the backend.

The direct API test with qwen2.5-1.5b-rkllm works as expected, so further debugging should focus on the taOS Agent request pipeline rather than the RKLLM backend.

[jaylfc]

This is excellent isolation work, thank you. Confirming your conclusion: the direct rkllama /api/chat path is healthy (NPU, model, and endpoint all correct), and the problem is in the taOS Agent request pipeline, not the backend. Two root causes on our side:

1. Embedding models should never have been selectable as agent models. qwen3-embedding-0.6b is an embedding model and cannot do chat completion, so its output is undefined. We already classify chat versus embedding internally, but that filter was not being applied to the agent-model picker. That is a bug and I am fixing it so only chat-capable models appear.

2. The agent harness prompt exceeding the model context is the loop. Your 11543-tokens-versus-4096 observation is the key point: the agent runs through a harness whose system prompt plus tool definitions are large, so a small-context local model (a 1.5B at 4096) receives a truncated prompt and degenerates into the repeated "analysis" output instead of a normal turn. The direct call works because a plain "Hello" is tiny and fits comfortably.

To your specific questions:

• The agent path goes through the harness rather than the simple chat endpoint, which is why the payload is large and hard to see from outside. Making the effective prompt inspectable is part of the fix.
• Embedding models are NOT expected to be selectable as agent models. That is the bug in (1).
• For RKLLM chat use today, choose a chat model with enough context headroom for the agent system prompt. A 4096-context model is too small for the full agent harness, which is exactly the guard we are adding.

The UI freezing during the runaway generation is the same thing you raised in #1741, and it needs the agent layer to bound and stream long output. That is on the list. Fixes incoming, and I will update here as they land. Thanks again for the precise, well-isolated report.

[jaylfc]

Update: both backend fixes for this are merged and I verified them on a live RK3588 (Orange Pi 5).

1. Embedding models can no longer be assigned as an agent chat model. Setting an agent to qwen3-embedding-0.6b now returns a clear 400 ("embedding model cannot be used as an agent chat model") instead of silently accepting it and looping. Verified on hardware: the assignment is rejected and the agent keeps its previous model.

2. A model whose context window is too small for the agent harness (for example a 4096-token RK model) now returns a non-blocking warning that the system prompt may be truncated, so the too-small-context loop you traced is surfaced rather than silent.

Both ship in the next beta. The remaining piece, the agent window showing liveness during a long or stalled generation, is tracked separately in #1741. Thanks again for the precise isolation that made these straightforward to fix.