Otel and misc fixes

.agents/docs/adr/0001-context-window-token-tracking.md

@@ -0,0 +1,103 @@
+# ADR-0001: Context Window Token Tracking and Auto-Summarization
+
+## Status
+
+Proposed
+
+## Context
+
+Crush's TUI displayed a context window usage percentage in the header and sidebar panels. The old implementation computed this percentage from `session.PromptTokens + session.CompletionTokens` — cumulative counters that the API returns at the end of each request. This caused two problems:
+
+1. **Stale display**: The UI always showed the token count from the *previous* request. After a new user message was sent but before the API responded, the displayed percentage was out of date.
+
+2. **Auto-summarization was unreliable**: The auto-summarization trigger used the same cumulative counters (`PromptTokens + CompletionTokens`) to decide whether to summarize. This meant a large incoming prompt could push the context window past the model's limit before the next API response corrected the counters, resulting in 400 Bad Request errors from the provider.
+
+3. **Not user-configurable**: The summarization threshold used a fixed buffer strategy (different thresholds for large vs. small context windows) rather than a percentage-based approach the user could tune.
+
+## Decision
+
+We introduced a new `CurrentTokens` field on the session model that tracks the *estimated* token count of the active conversation context in real time, independent of the API's cumulative counters.
+
+### Changes
+
+1. **New `CurrentTokens` field on `Session`** (`internal/db/models.go`, `internal/session/session.go`, `internal/db/sql/sessions.sql`, migration `20260608000000_add_current_tokens.sql`)
+
+   - Stored in SQLite as `INTEGER NOT NULL DEFAULT 0`.
+   - Propagated through all sqlc-generated queries (insert, select, update).
+
+2. **Real-time estimation at session start** (`internal/agent/agent.go:263`)
+
+   - Before each API request, `estimateMessageTokensForMessage()` walks all session messages and sums token estimates (text via `approxTokenCount`, tool calls, media, etc.).
+   - The result is saved to `session.CurrentTokens`.
+
+3. **Post-response update** (`internal/agent/agent.go:573`)
+
+   - After the API response, `CurrentTokens` is set to `PromptTokens + CompletionTokens` from the API's usage report, keeping the estimate in sync with the provider's actual count.
+
+4. **Percentage-based auto-summarization threshold** (`internal/config/config.go`)
+
+   - New `SummarizeThreshold` config option (default 0.8, range 0–1).
+   - Values of 0 or negative fall back to the default (80%).
+   - Replaces the old fixed-buffer strategy with `float64(CurrentTokens) / float64(ContextWindow) >= threshold`.
+   - Wired through from config to the agent in `coordinator.go` and `agentic_fetch_tool.go`.
+
+5. **Hard overflow guard** (`internal/agent/agent.go:304-325`)
+
+   - After adding the user message (before sending to the API), total tokens are re-estimated.
+   - If `totalTokens > ContextWindow`, summarization is forced regardless of the threshold setting.
+   - This prevents 400 errors from context overflow when a large prompt would otherwise slip past the soft threshold.
+
+6. **UI fallback for zero `CurrentTokens`** (`internal/ui/model/header.go`, `internal/ui/model/sidebar.go`)
+
+   - Both the header percentage and the sidebar context bar fall back to `PromptTokens + CompletionTokens` when `CurrentTokens == 0`.
+   - This handles two cases: new sessions before the first request (agent hasn't estimated yet), and existing sessions before the migration (DB column default is 0).
+   - Once the agent runs, `CurrentTokens` is populated and the UI uses it directly.
+
+### Consequences
+
+**Positive:**
+- The UI token count stays accurate through the full request cycle — it reflects the actual context window usage at the moment of display.
+- Auto-summarization now triggers based on the real context usage, not stale cumulative counters.
+- Users can tune the summarization trigger via `summarize_threshold` in `crush.json`.
+- The hard overflow guard prevents 400 errors from context overflow that the soft threshold alone couldn't catch.
+- The UI fallback (`CurrentTokens == 0 → PromptTokens + CompletionTokens`) ensures the display never shows 0% for sessions before the first request or before the migration.
+
+**Negative:**
+- `CurrentTokens` is an *estimate* (based on `approxTokenCount`), not the provider's exact count. The estimate may differ from the API's actual token count, especially for complex content (media, reasoning blocks).
+- The migration adds a column to the `sessions` table, which is a one-time schema change.
+- The UI fallback means the displayed value can still be stale (one request behind) in the edge case where `CurrentTokens` hasn't been populated yet. This is acceptable for now but could be improved with a backfill migration.
+
+**Open Questions:**
+- The estimate could be more accurate by using the provider's token counter directly when available (fantasy provides token counts per step).
+- A startup backfill migration to set `CurrentTokens = PromptTokens + CompletionTokens` for all existing sessions would eliminate the fallback entirely and make the UI accurate from the first render.
+
+## Alternatives
+
+### LLM/User-Driven Compaction via `new_session` Tool (PR #2333)
+
+An alternative approach was proposed in [PR #2333](https://github.com/charmbracelet/crush/pull/2333) by @taoeffect. It introduced:
+
+1. A **`new_session` tool** that allows the LLM to create a fresh session when context gets full, carrying forward a summary of progress and remaining work.
+2. A **`<context_status>` block** injected into the system prompt each turn, so the model can track its own context usage and proactively call `new_session` (default at ~75% context remaining).
+3. A **compaction method toggle** (auto vs LLM/user-driven) selectable via the command palette, persisting across sessions in the data config.
+4. A **UI renderer** for the tool's pending/completed states and a compaction method switching menu.
+
+### Why This Was Rejected
+
+We chose the simpler auto-summarization approach for now because:
+
+- **Less surface area**: The `new_session` approach requires a new tool, new config fields (`CompactionMethod`), new UI components (toggle, context status renderer), and changes to the system prompt injection layer. Our approach only adds a config option and a few lines in the agent loop.
+- **No LLM dependency**: The auto-summarization is fully automated and doesn't rely on the model to recognize context pressure and decide to call a tool. The model can fail to trigger the tool (as seen in PR #2333's own fix: `compactionFlags()` was disabling the auto-summarize safety net in LLM mode).
+- **No session fragmentation**: `new_session` creates entirely new sessions, which fragments conversation history and file version tracking (PR #2333 had to fix a UNIQUE constraint violation in file history as a result). Auto-summarization keeps everything in the same session.
+- **Faster iteration**: This approach is simpler to test, debug, and iterate on. The `SummarizeThreshold` config option already gives users control over when summarization triggers.
+
+The LLM/user-driven approach may be worth revisiting later as a complementary feature — for example, allowing the model to explicitly request compaction when it detects a complex multi-step task that would benefit from a clean slate, even before hitting the context limit.
+
+## References
+
+- `internal/agent/agent.go` — token estimation, auto-summarization, hard guard
+- `internal/config/config.go` — `SummarizeThreshold` option
+- `internal/session/session.go` — `CurrentTokens` field
+- `internal/db/migrations/20260608000000_add_current_tokens.sql` — schema migration
+- `internal/ui/model/header.go` — header percentage display
+- `internal/ui/model/sidebar.go` — sidebar context bar display

.agents/docs/adr/0002-structured-tool-observation-for-parsing-failures.md

@@ -0,0 +1,173 @@
+# ADR-0002: Structured Tool Observation for Parsing Failures
+
+## Status
+
+Accepted
+
+## Context
+
+### Original Behavior: Raw Errors in Session Messages
+
+Crush's agent harness originally included the raw error message and 400 Bad Request response from the inference API directly in the session's message history. This caused a critical failure: when a framework-level tool call validation error occurred (e.g., missing parameters, invalid JSON, malformed patterns), the error response was stored as a tool result in the session. The presence of this error message in the conversation history corrupted the session state — the session would become unusable and refuse further input, forcing the user to force-quit the application.
+
+### The Protective Strip
+
+To prevent sessions from becoming unusable, the error was handled by silently stripping the failed tool call and its error response from the conversation history before the next API request. This approach prevented the session corruption issue, but introduced a critical problem: the agent was left "blind" to why its tool call failed.
+
+### The Blind Agent Problem
+
+When the TUI displayed a tool error to the user, the model had no corresponding context in the conversation history to explain the failure. This led to the model confidently repeating the same broken tool call, producing repetitive and incorrect error messages, and in some cases entering retry loops that degraded the user experience.
+
+The core challenge was balancing two competing concerns:
+
+1. **Session integrity**: The conversation history must not contain raw error responses that corrupt the session and make it unusable.
+2. **Agent awareness**: The model must understand *why* a tool call failed so it can self-correct.
+
+## Decision
+
+We implemented a **Structured Tool Observation** pattern that intercepts framework-level validation errors, translates them into human-readable, actionable messages, and injects them as synthetic tool result messages into the conversation history — rather than silently stripping the failed call.
+
+### Architecture
+
+The implementation spans three layers:
+
+#### 1. Error Translation Layer (`internal/agent/tool_observation.go`)
+
+The `translateToObservation` function converts framework-level validation errors into semantic, model-friendly messages. It matches error message patterns and produces structured "Tool Observation" messages that explain the failure:
+
+```
+Tool Observation: Your previous attempt failed because the required parameter "pattern" was missing for tool "glob". Please review the tool definition and provide all necessary inputs.
+```
+
+The translation layer recognizes specific error categories:
+
+| Error Pattern | Translation |
+|---|---|
+| `missing required parameter` | Identifies the specific missing parameter (extracted via `extractParamName`) and the tool name. |
+| `invalid pattern` | Instructs the model to use standard regex syntax. |
+| `invalid json` / `malformed` / `parse error` | Instructs the model to ensure valid JSON with quoted keys. |
+| `extra data` | Instructs the model to provide only expected inputs. |
+| `context overflow` / `too long` / `overflow` | Instructs the model to provide shorter input. |
+| Generic tool errors | Provides the raw error message with guidance to review the tool definition. |
+
+The `injectToolObservation` method on `sessionAgent` creates a synthetic `message.Tool` message with `IsError: true` and appends it to the session's message store, preserving the original tool call ID so it passes the orphan filter.
+
+#### 2. Recovery Flow in the Coordinator (`internal/agent/coordinator.go:270-314`)
+
+The coordinator's `Run` method implements a three-phase recovery strategy for 400 Bad Request errors:
+
+1. **First retry**: On a 400 error, retry once. vLLM's tool call parser can produce malformed output that triggers context overflow; the model often produces valid output on the second attempt.
+
+2. **Strip + Observation**: If the second attempt also fails, the coordinator:
+   - Calls `injectToolObservation` to insert the translated error message into the session.
+   - Tags the context with `WithStripLastToolCall` so the agent skips the last assistant tool call when building the conversation history.
+   - Runs the agent again with the stripped history and the injected observation.
+
+3. **Hard stop**: If the third attempt also fails with 400, the coordinator does **not** retry further. The error is treated as permanent (e.g., model misconfiguration, invalid parameters).
+
+```go
+// Pseudocode of the recovery flow:
+if c.isBadRequest(originalErr) {
+    result, originalErr = run()                    // Attempt 1 (original)
+    if c.isBadRequest(originalErr) {
+        result, originalErr = run()                // Attempt 2 (retry)
+        if c.isBadRequest(originalErr) {
+            c.injectToolObservation(stripCtx, ...)  // Inject observation
+            result, originalErr = c.currentAgent.Run(stripCtx, ...)  // Attempt 3 (strip + observe)
+            // If still 400: do NOT retry further
+        }
+    }
+}
+```
+
+The `isBadRequest` function (`coordinator.go:1149`) filters 400 errors to only those that are potentially recoverable — matching on keywords like `tool`, `malformed`, `invalid json`, `extra data`, `context overflow`, and `parse error`. This prevents retrying on permanent errors (e.g., model not found).
+
+#### 3. Conversation History Stripping (`internal/agent/agent.go:1016`)
+
+When the `IsStripLastToolCall` context flag is set, the agent's `preparePrompt` method identifies the last assistant message with tool calls and skips its tool call parts when building the fantasy conversation history. This prevents the malformed tool call from being re-sent to the API while still preserving the rest of the conversation context.
+
+#### 4. System Prompt Constraint (`internal/agent/agent.go:454`)
+
+A `<tool-observation-instructions>` block is appended to the system prompt:
+
+```
+If you receive a 'Tool Observation' in a tool result message, you are required to analyze the error and adjust your strategy before retrying. Do not repeat the same failed tool call with the same input. Review the tool definition, correct the input parameters, and ensure valid JSON syntax.
+```
+
+This gives the model explicit instructions on how to respond when it receives a Tool Observation.
+
+#### 5. Consecutive Failure Guardrail (`internal/agent/loop_detection.go`)
+
+A separate loop detection mechanism tracks consecutive tool failures per tool name across the last 10 steps (`loopDetectionWindowSize`). If a specific tool fails more than 2 times in a row (`toolFailureMaxCount`), the system treats it as a loop and prevents further retries for that tool. A successful call for the same tool resets the counter.
+
+```go
+// Key constants:
+loopDetectionWindowSize  = 10
+toolFailureMaxCount      = 2
+```
+
+The `hasConsecutiveToolFailures` function scans steps backwards, tracking failure counts per tool name. A missing result (no tool result for a call) is also treated as a failure.
+
+### Context Key Design (`internal/agent/runid.go`)
+
+Two unexported context keys coordinate the flow between coordinator and agent:
+
+- `stripLastToolCallContextKey`: Signals the agent to skip the last assistant tool call.
+- `toolObservationErrorKey`: Carries the original validation error and tool name from the coordinator to the agent.
+
+These keys are passed via `context.WithValue` and checked via `IsStripLastToolCall` and `ToolObservationErrorFromContext`, keeping the coordinator-agent boundary clean without changing function signatures.
+
+## Consequences
+
+**Positive:**
+
+- **Informed self-correction**: The model understands *why* a tool call failed and can adjust its input on the next attempt. This eliminates the "blind retry" problem that previously caused repetitive, confident errors.
+- **Preserved context**: The conversation history reflects reality — the TUI error message matches what the model sees in the context window.
+- **Reduced noise**: The model is not forced to re-reason over its own broken input; it receives a clear, translated explanation instead.
+- **Loop prevention**: The combination of the hard stop (3 attempts max) and the consecutive failure guardrail (`toolFailureMaxCount = 2`) prevents infinite retry loops.
+- **Improved stability**: The structured observation has measurably improved the stability of the Crush agent in production, particularly for tools with strict input requirements (e.g., `glob`, `view`, `edit`).
+
+**Negative:**
+
+- **Increased message history**: Injecting synthetic tool result messages adds entries to the session, slightly increasing the context window usage.
+- **Translation fidelity**: The `translateToObservation` function relies on string matching against error messages. If the underlying framework changes error message formats, translations may become stale.
+- **Complexity**: The three-phase recovery flow (retry → strip + observe → hard stop) is more complex than the previous simple strip approach. This increases the cognitive load for future maintainers.
+- **Hard-coded thresholds**: The retry limit (3 attempts) and failure count (2) are hard-coded constants. Different models may recover from errors with varying success rates, making a configurable policy desirable in the future.
+
+**Open Questions:**
+
+- **Configurable policy**: The RFC proposes a `ToolPolicy` struct with `MaxRetries`, `Strict`, and `RequiresHuman` fields to make the failure handling configurable per-tool. This would allow users to define safety profiles based on the tool's risk level.
+- **Translation robustness**: The error translation relies on substring matching. A more robust approach might parse structured error types from the framework rather than matching on error message strings.
+- **Per-tool failure limits**: The current `toolFailureMaxCount = 2` applies uniformly to all tools. Some tools (e.g., `bash`) may tolerate more retries than others (e.g., `edit`).
+
+## Alternatives
+
+### Silent Strip (Previous Approach)
+
+The original approach silently stripped failed tool calls from the conversation history. This prevented infinite loops but left the agent blind to failures, causing it to repeat the same broken calls with confidence.
+
+**Rejected because**: The blind retry problem caused significant user-facing issues — the agent repeatedly produced the same errors without understanding why, degrading the user experience and making the agent appear unreliable.
+
+### Full Error Propagation to User
+
+An alternative would be to immediately surface tool failures to the user without attempting any retry. This would be the simplest approach but would sacrifice the agent's ability to self-correct on transient errors (e.g., vLLM parser producing malformed output that succeeds on retry).
+
+**Rejected because**: Many 400 errors are transient and recoverable. The three-phase recovery strategy captures these cases while still preventing infinite loops.
+
+### Strip Without Observation
+
+Strip the failed tool call but do not inject a Tool Observation message. This is simpler than the current approach but still leaves the agent without context about why the call failed.
+
+**Rejected because**: Without the observation, the agent still lacks the information needed to self-correct. The system prompt constraint alone is insufficient — the model needs the specific error context.
+
+## References
+
+- `internal/agent/tool_observation.go` — error translation, observation injection
+- `internal/agent/coordinator.go:270-314` — three-phase recovery flow
+- `internal/agent/coordinator.go:1149-1168` — `isBadRequest` error filtering
+- `internal/agent/coordinator.go:1170-1216` — `injectToolObservation` implementation
+- `internal/agent/agent.go:1016` — conversation history stripping
+- `internal/agent/agent.go:454` — system prompt constraint
+- `internal/agent/loop_detection.go` — consecutive failure guardrail
+- `internal/agent/runid.go` — context key design
+- RFC: `.agents/docs/rfc/structured-observation-layer-for-tool-failures.md`