Release Candidate v5.19.0

docs/open-api-docs.yaml

@@ -2,7 +2,7 @@ openapi: 3.0.3
 info:
   title: The Agent's user-facing API
   description: The user-facing parts of The Agent's API service (excluding system-level endpoints, chat completion, maintenance endpoints, etc.)
-  version: 5.18.0
+  version: 5.19.0
   license:
     name: MIT
     url: https://opensource.org/licenses/MIT

openspec/changes/archive/2026-06-09-add-grok-search-tools/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-06-09

openspec/changes/archive/2026-06-09-add-grok-search-tools/design.md

@@ -0,0 +1,61 @@
+## Context
+
+The search tool currently routes configured search requests through `AIWebSearch`. That executor supports Perplexity through LangChain and Google through the Gemini SDK with `GoogleSearch` grounding. Grok/xAI models are present in the external tool library, and `xai-sdk` is already installed, but Grok models are not selectable as `ToolType.search` and `AIWebSearch` has no xAI provider branch.
+
+xAI supports non-streaming Grok calls with server-side `web_search` and `x_search` tools enabled in the same request. The SDK exposes these tools as descriptors via `xai_sdk.tools.web_search()` and `xai_sdk.tools.x_search()`. The actual search work happens remotely during the Grok request.
+
+xAI also returns exact provider cost through `response.usage.cost_in_usd_ticks`, where `10_000_000_000` ticks equals one US dollar. Those ticks are monetary precision units, not runtime units. The returned cost includes token charges, prompt caching effects, and server-side tool invocation costs for that request.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Make selected Grok chat models usable as configured search tools.
+- Run Grok search as a single non-streaming xAI request with both web and X search tools enabled.
+- Preserve the existing `AIWebSearch` call contract for callers.
+- Track xAI search cost from provider-reported cost ticks instead of estimating token and tool-call costs.
+- Keep pre-flight credit checks conservative using existing `CostEstimate` fields.
+
+**Non-Goals:**
+- Do not add streaming search behavior.
+- Do not expose separate user-facing "web search" and "X search" modes.
+- Do not add a new database column unless implementation proves existing usage record fields are insufficient.
+- Do not change Google or Perplexity search behavior.
+- Do not replace the separate X/Twitter API post reader.
+
+## Decisions
+
+1. Use one xAI branch in `AIWebSearch`.
+
+   Add `XAI` to the provider dispatch in `AIWebSearch.execute()` and implement an xAI-specific path. The path should create an xAI chat with the configured Grok model, append the existing search prompt/query, and call `sample()` rather than `stream()`.
+
+   Alternative considered: use the OpenAI-compatible Responses API. The installed xAI SDK already exists in the project and exposes the required tools, so using it avoids adding another client path.
+
+2. Enable both `web_search` and `x_search` for Grok search.
+
+   From the product perspective, the app exposes one search tool. The xAI request should therefore provide both remote search tools and let Grok decide which to call.
+
+   Alternative considered: only enable `x_search`. That would make Grok search narrower than Google/Perplexity search and would not match the user's expectation that X search is part of the same search experience.
+
+3. Track exact xAI cost through a separate usage tracking method.
+
+   Add a method to `UsageTrackingService` for provider-reported request costs. It should create one usage record using the exact converted xAI cost, plus the normal maintenance fee. With the current DB shape, store the provider-reported request cost in `model_cost_credits` and keep `api_call_cost_credits` and `remote_runtime_cost_credits` at zero.
+
+   Alternative considered: reconstruct cost from input tokens, output tokens, and server-side tool usage. xAI's own response already returns the actual billed cost after discounts and all server-side tool calls, so reconstruction would be less accurate and could double-count.
+
+4. Keep `server_side_tool_usage` out of billing.
+
+   The implementation may log server-side tool usage for diagnostics, but billing should use `cost_in_usd_ticks` when present.
+
+   Alternative considered: emit separate usage records per xAI server-side tool call. This matches Google's current query-count approach but is not appropriate when xAI returns an all-inclusive provider cost.
+
+5. Preserve existing pre-flight validation.
+
+   Since exact xAI cost is only known after the response, Grok search models should still have approximate `CostEstimate` values sufficient for pre-flight credit validation. The estimate does not need to exactly match the final provider-reported charge.
+
+## Risks / Trade-offs
+
+- Provider cost metadata missing -> Treat as an external empty/unexpected response for platform-billed calls, or fall back to estimate only if the implementation deliberately accepts estimate drift.
+- Exact xAI cost exceeds pre-flight estimate -> Existing spending deduction can make the payer balance negative and logs a warning; keep estimates conservative.
+- Source formatting may differ from Google/Perplexity -> Prefer response citations or inline citation metadata if available; otherwise return the answer without a sources section rather than failing an otherwise valid answer.
+- Existing `XAIUsageTrackingDecorator` only wraps image calls -> Extending it for chat search must avoid changing image generation accounting unintentionally.
+- xAI SDK response shape may vary by model/tool output -> Add focused unit tests around cost extraction and response validation using lightweight fakes.

openspec/changes/archive/2026-06-09-add-grok-search-tools/proposal.md

@@ -0,0 +1,31 @@
+## Why
+
+Users can already choose Google Gemini Flash and Perplexity models for the app's search tool, but Grok models are only available for chat/vision/image workflows. xAI now supports non-streaming server-side `web_search` and `x_search` tools in the same Grok request, which lets Grok provide search behavior through the existing user-facing search abstraction.
+
+## What Changes
+
+- Allow selected Grok/xAI chat models to be configured as `ToolType.search` tools.
+- Add an xAI search execution path that performs one non-streaming Grok call with both `web_search` and `x_search` enabled.
+- Return the Grok answer through the same `AIWebSearch` interface used by Google and Perplexity search.
+- Track xAI search usage using xAI's provider-reported `cost_in_usd_ticks` value instead of estimating separate token/tool invocation costs.
+- Keep rough xAI cost estimates only for pre-flight credit validation.
+
+## Capabilities
+
+### New Capabilities
+- `grok-search-tools`: Covers using Grok/xAI models as configured search tools with server-side web and X search, including exact provider-reported cost tracking.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- Affected code:
+  - `src/features/external_tools/external_tool_library.py`
+  - `src/features/web_browsing/ai_web_search.py`
+  - `src/features/accounting/usage/usage_tracking_service.py`
+  - `src/features/accounting/usage/decorators/x_ai_usage_tracking_decorator.py`
+  - `src/di/di.py`
+- No database migration is expected if provider-reported xAI cost is stored in existing usage record cost fields.
+- No new external dependency is expected; `xai-sdk` is already installed and exposes `web_search()` and `x_search()`.

openspec/changes/archive/2026-06-09-add-grok-search-tools/specs/grok-search-tools/spec.md

@@ -0,0 +1,53 @@
+## ADDED Requirements
+
+### Requirement: Grok models are selectable for search
+The system SHALL allow supported Grok/xAI chat models to be configured as search tools through the existing search tool selection mechanism.
+
+#### Scenario: Supported Grok model appears as search-capable
+- **WHEN** the system builds the list of available search tools
+- **THEN** supported Grok/xAI chat models are included with `ToolType.search`
+
+#### Scenario: Unsupported xAI tools remain unavailable for search
+- **WHEN** the system builds the list of available search tools
+- **THEN** xAI image-generation tools are not included as search tools
+
+### Requirement: Grok search uses web and X search in one request
+The system SHALL execute Grok-backed search with one non-streaming xAI request that enables both server-side web search and X search tools.
+
+#### Scenario: Grok search execution
+- **WHEN** a user invokes the search tool with a Grok/xAI model selected
+- **THEN** the system sends one non-streaming Grok request with both `web_search` and `x_search` enabled
+
+#### Scenario: Grok search returns an answer
+- **WHEN** xAI returns a non-empty search response
+- **THEN** the system returns the answer through the existing AI web search result interface
+
+#### Scenario: Empty Grok search response
+- **WHEN** xAI returns no answer content for a Grok search request
+- **THEN** the system raises a structured external service error
+
+### Requirement: Grok search uses provider-reported cost
+The system SHALL track Grok search usage using xAI's provider-reported per-request cost when `cost_in_usd_ticks` is available.
+
+#### Scenario: Provider-reported cost is recorded
+- **WHEN** a Grok search request succeeds and includes `cost_in_usd_ticks`
+- **THEN** the system records one usage entry using the converted provider-reported cost
+
+#### Scenario: Server-side tool usage is not double-counted
+- **WHEN** a Grok search response includes server-side web or X tool usage counts
+- **THEN** the system does not create additional billable usage entries for those internal tool calls
+
+#### Scenario: Cost metadata is missing
+- **WHEN** a Grok search response is missing provider-reported cost metadata
+- **THEN** the system handles the response according to structured external-service error handling
+
+### Requirement: Existing search providers are unchanged
+The system SHALL preserve current Google and Perplexity search behavior while adding Grok-backed search.
+
+#### Scenario: Google search still uses Google grounding
+- **WHEN** a user invokes search with a Google search model selected
+- **THEN** the system uses the existing Google search execution path
+
+#### Scenario: Perplexity search still uses Perplexity
+- **WHEN** a user invokes search with a Perplexity search model selected
+- **THEN** the system uses the existing Perplexity search execution path

openspec/changes/archive/2026-06-09-add-grok-search-tools/tasks.md

@@ -0,0 +1,28 @@
+## 1. Tool Catalog
+
+- [x] 1.1 Add `ToolType.search` to supported Grok/xAI chat model definitions in `external_tool_library.py`.
+- [x] 1.2 Add conservative xAI search cost estimates for pre-flight credit checks without using them for final billing.
+- [x] 1.3 Verify xAI image tools remain excluded from search-capable tool lists.
+
+## 2. xAI Search Execution
+
+- [x] 2.1 Add `XAI` provider dispatch to `AIWebSearch.execute()`.
+- [x] 2.2 Implement non-streaming Grok search using `xai_sdk.tools.web_search()` and `xai_sdk.tools.x_search()` in one request.
+- [x] 2.3 Validate Grok search responses and raise structured `ExternalServiceError` errors for empty or unexpected responses.
+- [x] 2.4 Add source formatting for xAI citations or inline citation metadata when available, while allowing answers without sources if xAI provides no source data.
+
+## 3. Provider-Reported Cost Tracking
+
+- [x] 3.1 Add a separate `UsageTrackingService` method for provider-reported request costs.
+- [x] 3.2 Convert xAI `cost_in_usd_ticks` to credits using the existing project credit scale.
+- [x] 3.3 Store the provider-reported request cost in existing usage record cost fields without introducing a migration.
+- [x] 3.4 Extend `XAIUsageTrackingDecorator` to wrap the chat search call while preserving existing image tracking behavior.
+- [x] 3.5 Log `server_side_tool_usage` for diagnostics without creating additional billable usage records.
+
+## 4. Tests and Verification
+
+- [x] 4.1 Update `test/features/web_browsing/test_ai_web_search.py` for xAI provider routing, both enabled tools, non-streaming execution, and empty-response handling.
+- [x] 4.2 Update `test/features/accounting/usage/test_usage_tracking_service.py` for provider-reported cost records and maintenance fee behavior.
+- [x] 4.3 Update `test/features/accounting/usage/decorators/test_x_ai_usage_tracking_decorator.py` for chat search tracking and image tracking regression coverage.
+- [x] 4.4 Verify Grok search availability through xAI search provider routing coverage without adding catalog-constant assertions.
+- [x] 4.5 Run the focused test files with `pipenv run`, then run `pipenv run pre-commit run --all-files --show-diff-on-failure`.

openspec/specs/grok-search-tools/spec.md

@@ -0,0 +1,57 @@
+# grok-search-tools Specification
+
+## Purpose
+Define how Grok/xAI models participate in the existing search tool abstraction, including server-side web/X search execution and provider-reported cost tracking.
+
+## Requirements
+### Requirement: Grok models are selectable for search
+The system SHALL allow supported Grok/xAI chat models to be configured as search tools through the existing search tool selection mechanism.
+
+#### Scenario: Supported Grok model appears as search-capable
+- **WHEN** the system builds the list of available search tools
+- **THEN** supported Grok/xAI chat models are included with `ToolType.search`
+
+#### Scenario: Unsupported xAI tools remain unavailable for search
+- **WHEN** the system builds the list of available search tools
+- **THEN** xAI image-generation tools are not included as search tools
+
+### Requirement: Grok search uses web and X search in one request
+The system SHALL execute Grok-backed search with one non-streaming xAI request that enables both server-side web search and X search tools.
+
+#### Scenario: Grok search execution
+- **WHEN** a user invokes the search tool with a Grok/xAI model selected
+- **THEN** the system sends one non-streaming Grok request with both `web_search` and `x_search` enabled
+
+#### Scenario: Grok search returns an answer
+- **WHEN** xAI returns a non-empty search response
+- **THEN** the system returns the answer through the existing AI web search result interface
+
+#### Scenario: Empty Grok search response
+- **WHEN** xAI returns no answer content for a Grok search request
+- **THEN** the system raises a structured external service error
+
+### Requirement: Grok search uses provider-reported cost
+The system SHALL track Grok search usage using xAI's provider-reported per-request cost when `cost_in_usd_ticks` is available.
+
+#### Scenario: Provider-reported cost is recorded
+- **WHEN** a Grok search request succeeds and includes `cost_in_usd_ticks`
+- **THEN** the system records one usage entry using the converted provider-reported cost
+
+#### Scenario: Server-side tool usage is not double-counted
+- **WHEN** a Grok search response includes server-side web or X tool usage counts
+- **THEN** the system does not create additional billable usage entries for those internal tool calls
+
+#### Scenario: Cost metadata is missing
+- **WHEN** a Grok search response is missing provider-reported cost metadata
+- **THEN** the system handles the response according to structured external-service error handling
+
+### Requirement: Existing search providers are unchanged
+The system SHALL preserve current Google and Perplexity search behavior while adding Grok-backed search.
+
+#### Scenario: Google search still uses Google grounding
+- **WHEN** a user invokes search with a Google search model selected
+- **THEN** the system uses the existing Google search execution path
+
+#### Scenario: Perplexity search still uses Perplexity
+- **WHEN** a user invokes search with a Perplexity search model selected
+- **THEN** the system uses the existing Perplexity search execution path

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "the-agent"
-version = "5.18.0"
+version = "5.19.0"
 
 [tool.setuptools]
 package-dir = {"" = "src"}