diff --git a/docs/architecture.md b/docs/architecture.md index ec542789..2b7c34bd 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -366,6 +366,9 @@ However: - **Not all Maestro features are exposed** - only the subset needed for Trailblaze tools - **The driver is replaceable** - the architecture supports alternative implementations +**Revyl integration:** A standalone agent, `RevylTrailblazeAgent`, implements `TrailblazeAgent` directly (no Maestro) +and talks to Revyl cloud devices via HTTP. See [Revyl integration](revyl-integration.md) for details. + ### Driver Interface Drivers must implement command execution: diff --git a/docs/revyl-integration.md b/docs/revyl-integration.md new file mode 100644 index 00000000..0223ea08 --- /dev/null +++ b/docs/revyl-integration.md @@ -0,0 +1,158 @@ +--- +title: Revyl Cloud Device Integration +--- + +# Revyl Cloud Device Integration + +Trailblaze can use [Revyl](https://revyl.ai) cloud devices instead of local ADB or Maestro. This lets you run the same AI-powered tests against managed Android and iOS devices without a local device or emulator. + +## Overview + +The Revyl integration provides: + +- **RevylCliClient** (`trailblaze-revyl`) – Shells out to the `revyl` CLI binary for all device interactions. +- **RevylTrailblazeAgent** (`trailblaze-host`) – Maps every Trailblaze tool to a `revyl device` CLI command. +- **RevylNativeToolSet** (`trailblaze-revyl`) – Revyl-specific LLM tools (tap, type, swipe, assert) using natural language targeting and AI-powered visual grounding. + +Core data classes and the CLI client live in the `trailblaze-revyl` module. Host-level wiring (agent, blaze support) lives in `trailblaze-host/.../host/revyl/`. + +## Prerequisites + +1. Install the `revyl` CLI binary on your PATH: + + ```bash + curl -fsSL https://raw.githubusercontent.com/RevylAI/revyl-cli/main/scripts/install.sh | sh + # Or with Homebrew: + brew install RevylAI/tap/revyl + ``` + +2. Set the `REVYL_API_KEY` environment variable (or configure it in Settings > Environment Variables in the desktop app). + +**Optional overrides:** + +- `REVYL_BINARY` – Path to a specific `revyl` binary (skips PATH lookup). + +## Architecture + +```mermaid +flowchart LR + subgraph trailblaze["Trailblaze"] + LLM["LLM Agent"] + AGENT["RevylTrailblazeAgent"] + CLI["RevylCliClient"] + end + subgraph cli["revyl CLI"] + BIN["revyl device *"] + end + subgraph revyl["Revyl Cloud"] + PROXY["Backend Proxy"] + WORKER["Worker"] + end + DEVICE["Cloud Device"] + LLM --> AGENT + AGENT --> CLI + CLI -->|"ProcessBuilder"| BIN + BIN --> PROXY + PROXY --> WORKER + WORKER --> DEVICE +``` + +1. The LLM calls Trailblaze tools (tap, inputText, swipe, etc.). +2. **RevylTrailblazeAgent** dispatches each tool to **RevylCliClient**. +3. **RevylCliClient** runs the corresponding `revyl device` command via `ProcessBuilder` and parses the JSON output. +4. The `revyl` CLI handles auth, backend proxy routing, and AI-powered target grounding transparently. +5. The cloud device executes the action and returns results. + +## Quick start + +```kotlin +// Prerequisites: revyl CLI on PATH + REVYL_API_KEY set +val client = RevylCliClient() + +// Start a cloud device with an app installed +val session = client.startSession( + platform = "android", + appUrl = "https://example.com/my-app.apk", +) +println("Viewer: ${session.viewerUrl}") + +// Interact using natural language targets +client.tapTarget("Sign In button") +client.typeText("user@example.com", target = "email field") +client.tapTarget("Log In") + +// Screenshot +client.screenshot("after-login.png") + +// Clean up +client.stopSession() +``` + +## Supported operations + +All 12 Trailblaze tools are fully implemented: + +| Trailblaze tool | CLI command | +|-----------------|-------------| +| tap (coordinates) | `revyl device tap --x N --y N` | +| tap (grounded) | `revyl device tap --target "..."` | +| inputText | `revyl device type --text "..." [--target "..."]` | +| swipe | `revyl device swipe --direction ` | +| longPress | `revyl device long-press --target "..."` | +| launchApp | `revyl device launch --bundle-id ` | +| installApp | `revyl device install --app-url ` | +| eraseText | `revyl device clear-text` | +| pressBack | `revyl device back` | +| pressKey | `revyl device key --key ENTER` | +| openUrl | `revyl device navigate --url "..."` | +| screenshot | `revyl device screenshot --out ` | + +## Native instruction and validation steps + +Revyl devices can optionally use Revyl's own agent pipeline for natural-language +prompt steps instead of routing through Trailblaze's LLM agent. This is +controlled by the `useRevylNativeSteps` flag on `TrailblazeConfig` (default +`false` -- LLM pipeline is used until native step reporting with screenshots +is implemented). + +| YAML key | PromptStep type | CLI command | What happens | +|----------|-----------------|-------------|--------------| +| `- step:` | `DirectionStep` | `revyl device instruction "..."` | Revyl's worker agent plans and executes the action | +| `- verify:` | `VerificationStep` | `revyl device validation "..."` | Revyl's worker agent asserts against the current screen | + +When `useRevylNativeSteps` is `true`: + +- Each `- step:` prompt is sent directly to `revyl device instruction` in a + single round-trip. Revyl's agent handles grounding and execution natively. +- Each `- verify:` prompt is sent to `revyl device validation`, which performs + a visual assertion without needing a view hierarchy. +- Explicit tool steps (e.g. `- tapOn:`, `- inputText:`) are **not** affected + and still route through `RevylTrailblazeAgent` -> individual CLI commands. + +When `useRevylNativeSteps` is `false`: + +- All prompt steps go through Trailblaze's LLM agent pipeline, which decides + which tools to call (tap, type, swipe, etc.) and dispatches them via + `RevylTrailblazeAgent`. This is the original pre-flag behavior. + +```yaml +# Example trail using both step types +- prompts: + - step: Open the Search tab + - step: Type "beetle" in the search field + - verify: Search results are visible + - step: Tap the first result + - verify: Product detail page is shown +``` + +## Limitations + +- No local ADB or Maestro; all device interaction goes through Revyl cloud devices. +- View hierarchy from Revyl is minimal (screenshot-based AI grounding is used instead). +- Requires network access to the Revyl backend. + +## See also + +- [Architecture](architecture.md) – Revyl as an alternative to HostMaestroTrailblazeAgent. +- [Revyl CLI](https://github.com/RevylAI/revyl-cli) – Command-line tool for devices and tests. +- [Revyl docs](https://docs.revyl.ai) – Full CLI and SDK documentation. diff --git a/settings.gradle.kts b/settings.gradle.kts index 8d9970ff..8010941f 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -35,6 +35,7 @@ include( ":trailblaze-models", ":trailblaze-compose", ":trailblaze-playwright", + ":trailblaze-revyl", ":trailblaze-ui", ":trailblaze-report", ":trailblaze-server", diff --git a/trailblaze-desktop/build.gradle.kts b/trailblaze-desktop/build.gradle.kts index 59b152f7..d8e7a58d 100644 --- a/trailblaze-desktop/build.gradle.kts +++ b/trailblaze-desktop/build.gradle.kts @@ -42,6 +42,7 @@ dependencies { implementation(project(":trailblaze-common")) implementation(project(":trailblaze-compose")) implementation(project(":trailblaze-host")) + implementation(project(":trailblaze-revyl")) implementation(project(":trailblaze-models")) implementation(project(":trailblaze-report")) implementation(project(":trailblaze-server")) diff --git a/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopApp.kt b/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopApp.kt index a5536d69..5e263551 100644 --- a/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopApp.kt +++ b/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopApp.kt @@ -1,6 +1,7 @@ package xyz.block.trailblaze.desktop import xyz.block.trailblaze.compose.driver.tools.ComposeToolSet +import xyz.block.trailblaze.revyl.tools.RevylNativeToolSet import xyz.block.trailblaze.host.rules.TrailblazeHostDynamicLlmClientProvider import xyz.block.trailblaze.host.rules.TrailblazeHostDynamicLlmTokenProvider import xyz.block.trailblaze.host.yaml.DesktopYamlRunner @@ -31,6 +32,7 @@ class OpenSourceTrailblazeDesktopApp : TrailblazeDesktopApp( TrailblazeJsonInstance = TrailblazeJson.createTrailblazeJsonInstance( allToolClasses = TrailblazeToolSet.AllBuiltInTrailblazeToolsForSerializationByToolName + ComposeToolSet.toolClassesByToolName + + RevylNativeToolSet.RevylLlmToolSet.toolClasses.associateBy { it.toolName() } + desktopAppConfig.availableAppTargets.flatMap { it.getAllCustomToolClassesForSerialization() } .associateBy { it.toolName() }, ) diff --git a/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopAppConfig.kt b/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopAppConfig.kt index 7660eece..6855e8ed 100644 --- a/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopAppConfig.kt +++ b/trailblaze-desktop/src/main/java/xyz/block/trailblaze/desktop/OpenSourceTrailblazeDesktopAppConfig.kt @@ -13,6 +13,7 @@ import xyz.block.trailblaze.llm.providers.OpenRouterTrailblazeLlmModelList import xyz.block.trailblaze.mcp.utils.JvmLLMProvidersUtil import xyz.block.trailblaze.model.TrailblazeHostAppTarget import xyz.block.trailblaze.report.utils.LogsRepo +import xyz.block.trailblaze.revyl.RevylCliClient import xyz.block.trailblaze.ui.TrailblazeDesktopUtil import xyz.block.trailblaze.ui.TrailblazeSettingsRepo import xyz.block.trailblaze.ui.models.AppIconProvider @@ -35,6 +36,8 @@ class OpenSourceTrailblazeDesktopAppConfig : TrailblazeDesktopAppConfig( TrailblazeDriverType.IOS_HOST, TrailblazeDriverType.PLAYWRIGHT_NATIVE, TrailblazeDriverType.PLAYWRIGHT_ELECTRON, + TrailblazeDriverType.REVYL_ANDROID, + TrailblazeDriverType.REVYL_IOS, ) // Start with no platforms enabled by default - user must explicitly enable them @@ -76,7 +79,7 @@ class OpenSourceTrailblazeDesktopAppConfig : TrailblazeDesktopAppConfig( ALL_MODEL_LISTS.mapNotNull { trailblazeLlmModelList -> val trailblazeLlmProvider = trailblazeLlmModelList.provider JvmLLMProvidersUtil.getEnvironmentVariableKeyForLlmProvider(trailblazeLlmProvider) - } + } + RevylCliClient.REVYL_API_KEY_ENV override fun getCurrentlyAvailableLlmModelLists(): Set { val modelLists = JvmLLMProvidersUtil.getAvailableTrailblazeLlmProviderModelLists(ALL_MODEL_LISTS) diff --git a/trailblaze-host/build.gradle.kts b/trailblaze-host/build.gradle.kts index 73e6aed2..eb0e243b 100644 --- a/trailblaze-host/build.gradle.kts +++ b/trailblaze-host/build.gradle.kts @@ -90,6 +90,7 @@ dependencies { api(libs.slf4j.api) implementation(project(":trailblaze-common")) + implementation(project(":trailblaze-revyl")) implementation(project(":trailblaze-compose")) implementation(libs.compose.ui.test.junit4) implementation(project(":trailblaze-playwright")) diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/cli/TrailblazeCli.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/cli/TrailblazeCli.kt index 3703aca9..8f43d920 100644 --- a/trailblaze-host/src/main/java/xyz/block/trailblaze/cli/TrailblazeCli.kt +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/cli/TrailblazeCli.kt @@ -1329,6 +1329,9 @@ open class RunCommand : Callable { BasePlaywrightElectronTest.ELECTRON_BUILT_IN_TOOL_CLASSES TrailblazeDriverType.COMPOSE -> ComposeToolSet.LlmToolSet.toolClasses + TrailblazeDriverType.REVYL_ANDROID, + TrailblazeDriverType.REVYL_IOS -> + xyz.block.trailblaze.revyl.tools.RevylNativeToolSet.RevylLlmToolSet.toolClasses else -> emptySet() } diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/host/TrailblazeHostYamlRunner.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/host/TrailblazeHostYamlRunner.kt index 6dc40eaa..5204a3ea 100644 --- a/trailblaze-host/src/main/java/xyz/block/trailblaze/host/TrailblazeHostYamlRunner.kt +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/host/TrailblazeHostYamlRunner.kt @@ -36,8 +36,12 @@ import xyz.block.trailblaze.devices.TrailblazeDevicePlatform import xyz.block.trailblaze.devices.TrailblazeDriverType import xyz.block.trailblaze.exception.TrailblazeException import xyz.block.trailblaze.exception.TrailblazeSessionCancelledException -import xyz.block.trailblaze.host.ios.MobileDeviceUtils import xyz.block.trailblaze.host.golden.SnapshotGoldenComparison +import xyz.block.trailblaze.host.ios.MobileDeviceUtils +import xyz.block.trailblaze.host.revyl.RevylTrailblazeAgent +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.revyl.RevylScreenState +import xyz.block.trailblaze.revyl.RevylSession import xyz.block.trailblaze.host.rules.BaseComposeTest import xyz.block.trailblaze.host.rules.BaseHostTrailblazeTest import xyz.block.trailblaze.host.rules.BasePlaywrightElectronTest @@ -73,8 +77,10 @@ import xyz.block.trailblaze.ui.TrailblazeDeviceManager import xyz.block.trailblaze.util.Console import xyz.block.trailblaze.util.GitUtils import xyz.block.trailblaze.util.HostAndroidDeviceConnectUtils +import xyz.block.trailblaze.yaml.DirectionStep import xyz.block.trailblaze.yaml.ElectronAppConfig import xyz.block.trailblaze.yaml.TrailYamlItem +import xyz.block.trailblaze.yaml.VerificationStep import xyz.block.trailblaze.yaml.createTrailblazeYaml object TrailblazeHostYamlRunner { @@ -143,6 +149,9 @@ object TrailblazeHostYamlRunner { runPlaywrightElectronYaml(dynamicLlmClient, runOnHostParams, deviceManager) TrailblazeDriverType.COMPOSE -> runComposeYaml(dynamicLlmClient, runOnHostParams, deviceManager) + TrailblazeDriverType.REVYL_ANDROID, + TrailblazeDriverType.REVYL_IOS -> + runRevylYaml(dynamicLlmClient, runOnHostParams, deviceManager) else -> runMaestroHostYaml(dynamicLlmClient, runOnHostParams, deviceManager) } @@ -695,6 +704,273 @@ object TrailblazeHostYamlRunner { } } + /** + * Revyl cloud device path: provisions a device via [RevylCliClient] and runs + * the trail using [RevylTrailblazeAgent] with standard mobile tools. + * + * The CLI handles device provisioning, app install, and AI-powered target + * grounding. Screenshots come from [RevylScreenState]. + */ + private suspend fun runRevylYaml( + dynamicLlmClient: DynamicLlmClient, + runOnHostParams: RunOnHostParams, + deviceManager: TrailblazeDeviceManager, + ): SessionId? { + val onProgressMessage = runOnHostParams.onProgressMessage + val runYamlRequest = runOnHostParams.runYamlRequest + val trailblazeDeviceId = runYamlRequest.trailblazeDeviceId + val platform = if (runOnHostParams.trailblazeDriverType == TrailblazeDriverType.REVYL_ANDROID) "android" else "ios" + + val instanceId = trailblazeDeviceId.instanceId + val deviceLabel = if (instanceId.startsWith("revyl-model:")) + instanceId.removePrefix("revyl-model:") else "$platform (default)" + + if (System.getenv(RevylCliClient.REVYL_API_KEY_ENV).isNullOrBlank()) { + onProgressMessage("Error: ${RevylCliClient.REVYL_API_KEY_ENV} is not set. Configure it in Settings → Environment Variables.") + return null + } + + onProgressMessage("Provisioning Revyl cloud $deviceLabel...") + + val cliClient: RevylCliClient + val session: RevylSession + try { + cliClient = RevylCliClient() + session = if (instanceId.startsWith("revyl-model:")) { + val payload = instanceId.removePrefix("revyl-model:") + val parts = payload.split("::", limit = 2) + val modelName = parts[0] + val osVer = parts.getOrNull(1)?.takeIf { it.isNotBlank() } + if (osVer != null) { + cliClient.startSession(platform = platform, deviceModel = modelName, osVersion = osVer) + } else { + Console.log("RevylYaml: device '$modelName' missing OS version — using platform default") + cliClient.startSession(platform = platform) + } + } else { + cliClient.startSession(platform = platform) + } + } catch (e: Exception) { + Console.log("Revyl session provisioning failed: ${e::class.simpleName}: ${e.message}") + onProgressMessage("Error: ${e.message}") + return null + } + onProgressMessage("Revyl $deviceLabel ready — viewer: ${session.viewerUrl}") + + // Outer try-finally guarantees the cloud device is stopped even if setup + // (e.g. LLM client creation) fails before the inner execution try block. + try { + val trailblazeDeviceInfo = TrailblazeDeviceInfo( + trailblazeDeviceId = trailblazeDeviceId, + trailblazeDriverType = runOnHostParams.trailblazeDriverType, + widthPixels = session.screenWidth.takeIf { it > 0 } + ?: xyz.block.trailblaze.revyl.RevylDefaults.dimensionsForPlatform(platform).first, + heightPixels = session.screenHeight.takeIf { it > 0 } + ?: xyz.block.trailblaze.revyl.RevylDefaults.dimensionsForPlatform(platform).second, + metadata = mapOf("revyl_viewer_url" to session.viewerUrl), + classifiers = listOf( + TrailblazeDeviceClassifier(platform), + TrailblazeDeviceClassifier("revyl-cloud"), + ), + ) + + val screenStateProvider: () -> ScreenState = { + RevylScreenState(cliClient, platform, session.screenWidth, session.screenHeight) + } + + TrailblazeJsonInstance = TrailblazeJson.createTrailblazeJsonInstance( + allToolClasses = TrailblazeToolSet.AllBuiltInTrailblazeToolsForSerializationByToolName + + xyz.block.trailblaze.revyl.tools.RevylNativeToolSet.RevylLlmToolSet.toolClasses.associateBy { it.toolName() }, + ) + + val loggingRule = HostTrailblazeLoggingRule( + trailblazeDeviceInfoProvider = { trailblazeDeviceInfo }, + ) + + val agent = RevylTrailblazeAgent( + cliClient = cliClient, + platform = platform, + trailblazeLogger = loggingRule.logger, + trailblazeDeviceInfoProvider = { trailblazeDeviceInfo }, + sessionProvider = { + loggingRule.session ?: error("Session not available - ensure test is running") + }, + ) + + val toolRepo = TrailblazeToolRepo( + xyz.block.trailblaze.revyl.tools.RevylNativeToolSet.RevylLlmToolSet, + ) + + val trailblazeRunner = TrailblazeRunner( + screenStateProvider = screenStateProvider, + agent = agent, + llmClient = dynamicLlmClient.createLlmClient(), + trailblazeLlmModel = runYamlRequest.trailblazeLlmModel, + trailblazeToolRepo = toolRepo, + trailblazeLogger = loggingRule.logger, + sessionProvider = { + loggingRule.session ?: error("Session not available - ensure test is running") + }, + ) + + val elementComparator = TrailblazeElementComparator( + screenStateProvider = screenStateProvider, + llmClient = dynamicLlmClient.createLlmClient(), + trailblazeLlmModel = runYamlRequest.trailblazeLlmModel, + toolRepo = toolRepo, + ) + + val trailblazeYaml = createTrailblazeYaml( + customTrailblazeToolClasses = xyz.block.trailblaze.revyl.tools.RevylNativeToolSet.RevylLlmToolSet.toolClasses, + ) + + val trailblazeRunnerUtil = TrailblazeRunnerUtil( + trailblazeRunner = trailblazeRunner, + runTrailblazeTool = { trailblazeTools: List -> + val result = agent.runTrailblazeTools( + trailblazeTools, + null, + screenState = screenStateProvider(), + elementComparator = elementComparator, + screenStateProvider = screenStateProvider, + ) + when (val toolResult = result.result) { + is TrailblazeToolResult.Success -> toolResult + is TrailblazeToolResult.Error -> throw TrailblazeException(toolResult.errorMessage) + } + }, + trailblazeLogger = loggingRule.logger, + sessionProvider = { + loggingRule.session ?: error("Session not available - ensure test is running") + }, + ) + + val sessionManager = loggingRule.sessionManager + + val overrideSessionId = runYamlRequest.config.overrideSessionId + val trailblazeSession = if (overrideSessionId != null) { + sessionManager.createSessionWithId(overrideSessionId) + } else { + sessionManager.startSession(runYamlRequest.testName) + } + loggingRule.setSession(trailblazeSession) + + return try { + onProgressMessage("Executing YAML test via Revyl cloud device...") + Console.log("Starting Revyl execution for device: ${trailblazeDeviceId.instanceId}") + + val trailItems: List = trailblazeYaml.decodeTrail(runYamlRequest.yaml) + val trailConfig = trailblazeYaml.extractTrailConfig(trailItems) + + if (runYamlRequest.config.sendSessionStartLog) { + loggingRule.logger.log( + trailblazeSession, + TrailblazeLog.TrailblazeSessionStatusChangeLog( + sessionStatus = SessionStatus.Started( + trailConfig = trailConfig, + trailFilePath = runYamlRequest.trailFilePath, + testClassName = "Revyl", + testMethodName = "run", + trailblazeDeviceInfo = trailblazeDeviceInfo, + rawYaml = runYamlRequest.yaml, + hasRecordedSteps = trailblazeYaml.hasRecordedSteps(trailItems), + trailblazeDeviceId = trailblazeDeviceId, + ), + session = trailblazeSession.sessionId, + timestamp = Clock.System.now(), + ), + ) + } + + val useRevylNativeSteps = runYamlRequest.config.useRevylNativeSteps + for (item in trailItems) { + val itemResult = when (item) { + is TrailYamlItem.PromptsTrailItem -> { + if (useRevylNativeSteps) { + for (prompt in item.promptSteps) { + when (prompt) { + is VerificationStep -> { + Console.log("RevylYaml: validation — '${prompt.verify}'") + val result = cliClient.validation(prompt.verify) + if (!result.success) { + throw TrailblazeException( + "Validation failed: ${prompt.verify}" + + (result.statusReason?.let { " — $it" } ?: ""), + ) + } + } + is DirectionStep -> { + Console.log("RevylYaml: instruction — '${prompt.step}'") + val result = cliClient.instruction(prompt.step) + if (!result.success) { + throw TrailblazeException( + "Instruction failed: ${prompt.step}" + + (result.statusReason?.let { " — $it" } ?: ""), + ) + } + } + } + } + TrailblazeToolResult.Success() + } else { + trailblazeRunnerUtil.runPromptSuspend(item.promptSteps, runYamlRequest.useRecordedSteps) + } + } + is TrailYamlItem.ToolTrailItem -> + trailblazeRunnerUtil.runTrailblazeTool(item.tools.map { it.trailblazeTool }) + is TrailYamlItem.ConfigTrailItem -> + item.config.context?.let { trailblazeRunner.appendToSystemPrompt(it) } + } + if (itemResult is TrailblazeToolResult.Error) { + throw TrailblazeException(itemResult.errorMessage) + } + } + + Console.log("Revyl execution completed for device: ${trailblazeDeviceId.instanceId}") + onProgressMessage("Test execution completed successfully") + + if (runYamlRequest.config.sendSessionEndLog) { + sessionManager.endSession(trailblazeSession, isSuccess = true) + } + + generateAndSaveRecording( + sessionId = trailblazeSession.sessionId, + customToolClasses = xyz.block.trailblaze.revyl.tools.RevylNativeToolSet.RevylLlmToolSet.toolClasses, + ) + + trailblazeSession.sessionId + } catch (e: TrailblazeSessionCancelledException) { + Console.log("TrailblazeSessionCancelledException caught for device: ${trailblazeDeviceId.instanceId}") + onProgressMessage("Test session cancelled") + null + } catch (e: CancellationException) { + Console.log("CancellationException caught for device: ${trailblazeDeviceId.instanceId} - ${e.message}") + onProgressMessage("Test execution cancelled") + throw e + } catch (e: Exception) { + Console.log("Exception caught in runRevylYaml for device: ${trailblazeDeviceId.instanceId} - ${e::class.simpleName}: ${e.message}") + onProgressMessage("Test execution failed: ${e.message}") + captureFailureScreenshot(trailblazeSession, loggingRule, screenStateProvider) + sessionManager.endSession(trailblazeSession, isSuccess = false, exception = e) + null + } finally { + Console.log("Finally block executing for Revyl device: ${trailblazeDeviceId.instanceId}") + exportAndSaveTrace(trailblazeSession.sessionId, loggingRule) + loggingRule.setSession(null) + deviceManager.cancelSessionForDevice(trailblazeDeviceId) + Console.log("Finally block completed for Revyl device: ${trailblazeDeviceId.instanceId}") + } + } catch (e: CancellationException) { + throw e + } catch (e: Exception) { + Console.log("Revyl setup failed for device: ${trailblazeDeviceId.instanceId} - ${e::class.simpleName}: ${e.message}") + onProgressMessage("Error: ${e.message}") + return null + } finally { + try { cliClient.stopSession() } catch (_: Exception) { } + } + } + /** * Original Maestro-based path for Android/iOS/web-playwright-host devices. */ diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/host/revyl/RevylBlazeSupport.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/host/revyl/RevylBlazeSupport.kt new file mode 100644 index 00000000..25458a4b --- /dev/null +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/host/revyl/RevylBlazeSupport.kt @@ -0,0 +1,109 @@ +package xyz.block.trailblaze.host.revyl + +import xyz.block.trailblaze.agent.AgentUiActionExecutor +import xyz.block.trailblaze.agent.BlazeConfig +import xyz.block.trailblaze.agent.ScreenAnalyzer +import xyz.block.trailblaze.agent.TrailblazeElementComparator +import xyz.block.trailblaze.agent.blaze.BlazeGoalPlanner +import xyz.block.trailblaze.devices.TrailblazeDeviceId +import xyz.block.trailblaze.devices.TrailblazeDeviceInfo +import xyz.block.trailblaze.devices.TrailblazeDevicePlatform +import xyz.block.trailblaze.devices.TrailblazeDriverType +import xyz.block.trailblaze.logs.client.TrailblazeLogger +import xyz.block.trailblaze.logs.client.TrailblazeSession +import xyz.block.trailblaze.logs.model.SessionId +import xyz.block.trailblaze.toolcalls.TrailblazeToolRepo +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.revyl.RevylDefaults +import xyz.block.trailblaze.revyl.RevylScreenState + +/** + * Factory for wiring [BlazeGoalPlanner] to run on Revyl cloud devices. + * + * Blaze mode requires LLM infrastructure ([ScreenAnalyzer], [TrailblazeToolRepo], + * [TrailblazeElementComparator]) that the host application must provide. This + * utility bridges those host-level dependencies with the Revyl device layer + * ([RevylTrailblazeAgent] + [RevylScreenState]). + * + * Usage: + * ``` + * val cliClient = RevylCliClient() + * cliClient.startSession(platform = "android", appUrl = "...") + * + * val planner = RevylBlazeSupport.createBlazeRunner( + * cliClient = cliClient, + * platform = "android", + * screenAnalyzer = myScreenAnalyzer, // from your LLM config + * toolRepo = myToolRepo, // from TrailblazeToolSet + * elementComparator = myElementComparator, // from your LLM config + * ) + * + * val result = planner.execute(BlazeState(objective = "Tap the Search tab")) + * ``` + */ +object RevylBlazeSupport { + + /** + * Creates a [BlazeGoalPlanner] backed by Revyl cloud devices. + * + * Constructs [AgentUiActionExecutor] with a [RevylTrailblazeAgent] for + * device actions and [RevylScreenState] for screenshot capture, then + * wires them into a [BlazeGoalPlanner] ready for AI-driven exploration. + * + * @param cliClient Authenticated CLI client with an active session. + * @param platform Device platform enum. + * @param screenAnalyzer LLM-powered screen analyzer (provided by host). + * @param toolRepo Tool repository for deserializing tool calls (provided by host). + * @param elementComparator Element comparator for assertions (provided by host). + * @param config Blaze exploration settings. Defaults to [BlazeConfig.DEFAULT]. + * @return A configured [BlazeGoalPlanner] targeting the Revyl cloud device. + */ + fun createBlazeRunner( + cliClient: RevylCliClient, + platform: TrailblazeDevicePlatform, + screenAnalyzer: ScreenAnalyzer, + toolRepo: TrailblazeToolRepo, + elementComparator: TrailblazeElementComparator, + config: BlazeConfig = BlazeConfig.DEFAULT, + ): BlazeGoalPlanner { + val driverType = when (platform) { + TrailblazeDevicePlatform.IOS -> TrailblazeDriverType.REVYL_IOS + else -> TrailblazeDriverType.REVYL_ANDROID + } + val defaultDimensions = when (platform) { + TrailblazeDevicePlatform.IOS -> Pair(RevylDefaults.IOS_DEFAULT_WIDTH, RevylDefaults.IOS_DEFAULT_HEIGHT) + else -> Pair(RevylDefaults.ANDROID_DEFAULT_WIDTH, RevylDefaults.ANDROID_DEFAULT_HEIGHT) + } + val deviceInfo = TrailblazeDeviceInfo( + trailblazeDeviceId = TrailblazeDeviceId(instanceId = "revyl-blaze", trailblazeDevicePlatform = platform), + trailblazeDriverType = driverType, + widthPixels = defaultDimensions.first, + heightPixels = defaultDimensions.second, + ) + val platformStr = when (platform) { + TrailblazeDevicePlatform.IOS -> RevylCliClient.PLATFORM_IOS + else -> RevylCliClient.PLATFORM_ANDROID + } + val agent = RevylTrailblazeAgent( + cliClient = cliClient, + platform = platformStr, + trailblazeLogger = TrailblazeLogger.createNoOp(), + trailblazeDeviceInfoProvider = { deviceInfo }, + sessionProvider = { + TrailblazeSession(sessionId = SessionId("revyl-blaze"), startTime = kotlinx.datetime.Clock.System.now()) + }, + ) + val screenStateProvider = { RevylScreenState(cliClient, platformStr) } + val executor = AgentUiActionExecutor( + agent = agent, + screenStateProvider = screenStateProvider, + toolRepo = toolRepo, + elementComparator = elementComparator, + ) + return BlazeGoalPlanner( + config = config, + screenAnalyzer = screenAnalyzer, + executor = executor, + ) + } +} diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/host/revyl/RevylTrailblazeAgent.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/host/revyl/RevylTrailblazeAgent.kt new file mode 100644 index 00000000..0ad81c95 --- /dev/null +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/host/revyl/RevylTrailblazeAgent.kt @@ -0,0 +1,242 @@ +@file:Suppress("DEPRECATION") + +package xyz.block.trailblaze.host.revyl + +import maestro.SwipeDirection +import xyz.block.trailblaze.AgentMemory +import xyz.block.trailblaze.TrailblazeAgentContext +import xyz.block.trailblaze.api.ScreenState +import xyz.block.trailblaze.api.TrailblazeAgent +import xyz.block.trailblaze.api.TrailblazeAgent.RunTrailblazeToolsResult +import xyz.block.trailblaze.devices.TrailblazeDeviceInfo +import xyz.block.trailblaze.logs.client.TrailblazeLogger +import xyz.block.trailblaze.logs.client.TrailblazeSessionProvider +import xyz.block.trailblaze.logs.model.TraceId +import xyz.block.trailblaze.toolcalls.TrailblazeTool +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.toolcalls.commands.EraseTextTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.HideKeyboardTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.InputTextTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.LaunchAppTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.ObjectiveStatusTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.OpenUrlTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.PressBackTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.PressKeyTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.ScrollUntilTextIsVisibleTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.SwipeTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.TakeSnapshotTool +import xyz.block.trailblaze.toolcalls.commands.TapOnElementByNodeIdTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.TapOnPointTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.WaitForIdleSyncTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.NetworkConnectionTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.LongPressOnElementWithTextTrailblazeTool +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.commands.memory.MemoryTrailblazeTool +import xyz.block.trailblaze.toolcalls.getToolNameFromAnnotation +import xyz.block.trailblaze.utils.ElementComparator +import xyz.block.trailblaze.util.Console +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.revyl.RevylDefaults +import xyz.block.trailblaze.revyl.RevylScreenState +import xyz.block.trailblaze.revyl.tools.RevylExecutableTool + +/** + * [TrailblazeAgent] implementation that routes all device actions through + * the Revyl CLI binary via [RevylCliClient]. + * + * Each [TrailblazeTool] is mapped to the corresponding `revyl device` + * subcommand. The CLI handles auth, backend proxying, and AI-powered + * target grounding transparently. + * + * @property cliClient CLI-based client for Revyl device interactions. + * @property platform "ios" or "android" — used for ScreenState construction. + * @property trailblazeLogger Logger for tool execution and snapshot events. + * @property trailblazeDeviceInfoProvider Provides device info for logging context. + * @property sessionProvider Provides current session for logging operations. + */ +class RevylTrailblazeAgent( + private val cliClient: RevylCliClient, + private val platform: String, + override val trailblazeLogger: TrailblazeLogger, + override val trailblazeDeviceInfoProvider: () -> TrailblazeDeviceInfo, + override val sessionProvider: TrailblazeSessionProvider, +) : TrailblazeAgent, TrailblazeAgentContext { + + override val memory = AgentMemory() + + /** + * Dispatches a list of [TrailblazeTool]s by mapping each tool to + * a `revyl device` CLI command via [RevylCliClient]. + * + * @param tools Ordered list of tools to execute sequentially. + * @param traceId Optional trace ID for log correlation. + * @param screenState Cached screen state from the most recent LLM turn. + * @param elementComparator Comparator for memory-based assertions. + * @param screenStateProvider Lazy provider for a fresh device screenshot. + * @return Execution result containing the input tools, executed tools, and outcome. + */ + override fun runTrailblazeTools( + tools: List, + traceId: TraceId?, + screenState: ScreenState?, + elementComparator: ElementComparator, + screenStateProvider: (() -> ScreenState)?, + ): RunTrailblazeToolsResult { + val executed = mutableListOf() + + for (tool in tools) { + executed.add(tool) + val result = executeTool(tool, screenStateProvider) + if (result !is TrailblazeToolResult.Success) { + return RunTrailblazeToolsResult( + inputTools = tools, + executedTools = executed, + result = result, + ) + } + } + + return RunTrailblazeToolsResult( + inputTools = tools, + executedTools = executed, + result = TrailblazeToolResult.Success(), + ) + } + + private fun executeTool( + tool: TrailblazeTool, + screenStateProvider: (() -> ScreenState)?, + ): TrailblazeToolResult { + val toolName = tool.getToolNameFromAnnotation() + Console.log("RevylAgent: executing tool '$toolName'") + + return try { + when (tool) { + is TapOnPointTrailblazeTool -> { + val target = tool.reasoning?.takeIf { it.isNotBlank() } + if (tool.longPress) { + val desc = target ?: "element at (${tool.x}, ${tool.y})" + val r = cliClient.longPress(desc) + TrailblazeToolResult.Success(message = "Long-pressed '$desc' at (${r.x}, ${r.y})") + } else { + val r = if (target != null) cliClient.tapTarget(target) else cliClient.tap(tool.x, tool.y) + TrailblazeToolResult.Success(message = "Tapped '${target ?: "${tool.x},${tool.y}"}' at (${r.x}, ${r.y})") + } + } + is InputTextTrailblazeTool -> { + val r = cliClient.typeText(tool.text, target = "text input field") + TrailblazeToolResult.Success(message = "Typed '${tool.text}' at (${r.x}, ${r.y})") + } + is SwipeTrailblazeTool -> { + val direction = when (tool.direction) { + SwipeDirection.UP -> "up" + SwipeDirection.DOWN -> "down" + SwipeDirection.LEFT -> "left" + SwipeDirection.RIGHT -> "right" + } + val target = tool.swipeOnElementText ?: "center of screen" + val r = cliClient.swipe(direction, target = target) + TrailblazeToolResult.Success(message = "Swiped $direction on '$target' from (${r.x}, ${r.y})") + } + is LaunchAppTrailblazeTool -> { + cliClient.launchApp(tool.appId) + TrailblazeToolResult.Success(message = "Launched ${tool.appId}") + } + is EraseTextTrailblazeTool -> { + val r = cliClient.clearText(target = "focused input field") + TrailblazeToolResult.Success(message = "Cleared text at (${r.x}, ${r.y})") + } + is HideKeyboardTrailblazeTool -> { + TrailblazeToolResult.Success() + } + is PressBackTrailblazeTool -> { + val r = cliClient.back() + TrailblazeToolResult.Success(message = "Pressed back at (${r.x}, ${r.y})") + } + is PressKeyTrailblazeTool -> { + val r = cliClient.pressKey(tool.keyCode.name) + TrailblazeToolResult.Success(message = "Pressed ${tool.keyCode.name} at (${r.x}, ${r.y})") + } + is OpenUrlTrailblazeTool -> { + cliClient.navigate(tool.url) + TrailblazeToolResult.Success(message = "Navigated to ${tool.url}") + } + is TakeSnapshotTool -> { + cliClient.screenshot() + TrailblazeToolResult.Success(message = "Screenshot captured") + } + is WaitForIdleSyncTrailblazeTool -> { + Thread.sleep(1000) + TrailblazeToolResult.Success() + } + is ScrollUntilTextIsVisibleTrailblazeTool -> { + val direction = when (tool.direction) { + maestro.ScrollDirection.UP -> "up" + maestro.ScrollDirection.DOWN -> "down" + maestro.ScrollDirection.LEFT -> "left" + maestro.ScrollDirection.RIGHT -> "right" + } + val target = tool.text.ifBlank { "center of screen" } + val r = cliClient.swipe(direction, target = target) + TrailblazeToolResult.Success(message = "Scrolled $direction on '$target' from (${r.x}, ${r.y})") + } + is NetworkConnectionTrailblazeTool -> { + Console.log("RevylAgent: network toggle not yet implemented for cloud devices") + TrailblazeToolResult.Success() + } + is TapOnElementByNodeIdTrailblazeTool -> { + val target = tool.reasoning?.takeIf { it.isNotBlank() } ?: "element with node id ${tool.nodeId}" + val r = if (tool.longPress) cliClient.longPress(target) else cliClient.tapTarget(target) + val action = if (tool.longPress) "Long-pressed" else "Tapped" + TrailblazeToolResult.Success(message = "$action '$target' at (${r.x}, ${r.y})") + } + is LongPressOnElementWithTextTrailblazeTool -> { + val r = cliClient.longPress(tool.text) + TrailblazeToolResult.Success(message = "Long-pressed '${tool.text}' at (${r.x}, ${r.y})") + } + is RevylExecutableTool -> { + kotlinx.coroutines.runBlocking { + tool.executeWithRevyl(cliClient, buildRevylToolContext(screenStateProvider)) + } + } + is ObjectiveStatusTrailblazeTool -> TrailblazeToolResult.Success() + is MemoryTrailblazeTool -> TrailblazeToolResult.Success() + else -> { + Console.log("RevylAgent: unsupported tool ${tool::class.simpleName}") + TrailblazeToolResult.Success() + } + } + } catch (e: Exception) { + Console.error("RevylAgent: tool '$toolName' failed: ${e.message}") + TrailblazeToolResult.Error.ExceptionThrown( + errorMessage = "CLI execution failed for '$toolName': ${e.message}", + command = tool, + stackTrace = e.stackTraceToString(), + ) + } + } + + /** + * Builds a [TrailblazeToolExecutionContext] for executing [RevylExecutableTool] + * instances, using this agent's logger, session provider, and device info. + * + * @param screenStateProvider Optional lazy provider for fresh screenshots. + * @return A context suitable for Revyl native tool execution. + */ + private fun buildRevylToolContext( + screenStateProvider: (() -> ScreenState)?, + ): TrailblazeToolExecutionContext { + val deviceInfo = trailblazeDeviceInfoProvider() + val effectiveScreenStateProvider = screenStateProvider + ?: { RevylScreenState(cliClient, platform) } + return TrailblazeToolExecutionContext( + screenState = null, + traceId = null, + trailblazeDeviceInfo = deviceInfo, + sessionProvider = sessionProvider, + screenStateProvider = effectiveScreenStateProvider, + trailblazeLogger = trailblazeLogger, + memory = memory, + ) + } +} diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/mcp/TrailblazeMcpBridgeImpl.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/mcp/TrailblazeMcpBridgeImpl.kt index 227d1f51..e087111a 100644 --- a/trailblaze-host/src/main/java/xyz/block/trailblaze/mcp/TrailblazeMcpBridgeImpl.kt +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/mcp/TrailblazeMcpBridgeImpl.kt @@ -851,10 +851,11 @@ class TrailblazeMcpBridgeImpl( } override fun getInnerAgentBuiltInToolClasses(): Set> { - return if (getDriverType() == TrailblazeDriverType.PLAYWRIGHT_NATIVE) { - PlaywrightNativeToolSet.LlmToolSet.toolClasses - } else { - emptySet() + return when (getDriverType()) { + TrailblazeDriverType.PLAYWRIGHT_NATIVE -> PlaywrightNativeToolSet.LlmToolSet.toolClasses + TrailblazeDriverType.REVYL_ANDROID, + TrailblazeDriverType.REVYL_IOS -> xyz.block.trailblaze.revyl.tools.RevylNativeToolSet.RevylLlmToolSet.toolClasses + else -> emptySet() } } diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeDeviceManager.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeDeviceManager.kt index a14766a4..e3b11e4a 100644 --- a/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeDeviceManager.kt +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeDeviceManager.kt @@ -68,6 +68,8 @@ import xyz.block.trailblaze.host.rules.BasePlaywrightElectronTest import xyz.block.trailblaze.host.rules.BasePlaywrightNativeTest import xyz.block.trailblaze.util.Console import xyz.block.trailblaze.util.isMacOs +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.revyl.RevylScreenState /** * Manages device discovery, selection, and state across the application. @@ -133,6 +135,8 @@ class TrailblazeDeviceManager( TrailblazeDriverType.PLAYWRIGHT_NATIVE, TrailblazeDriverType.PLAYWRIGHT_ELECTRON, TrailblazeDriverType.COMPOSE -> isWebMode + TrailblazeDriverType.REVYL_ANDROID, + TrailblazeDriverType.REVYL_IOS -> true else -> settingsRepo.getEnabledDriverTypes().contains(connectedDeviceSummary.trailblazeDriverType) } } @@ -386,6 +390,14 @@ class TrailblazeDeviceManager( Console.log("⚠️ Screen state capture not supported for ${driverType.name} driver") null } + TrailblazeDriverType.REVYL_ANDROID, + TrailblazeDriverType.REVYL_IOS -> { + val platform = if (driverType == TrailblazeDriverType.REVYL_ANDROID) "android" else "ios" + RevylScreenState( + revylCliClient, + platform, + ) + } } } @@ -587,6 +599,42 @@ class TrailblazeDeviceManager( ) ) } + + // Revyl cloud devices — default presets always included as fallbacks. + add( + TrailblazeConnectedDeviceSummary( + trailblazeDriverType = TrailblazeDriverType.REVYL_ANDROID, + instanceId = "revyl-android-phone", + description = "Revyl Android (Default)", + ) + ) + add( + TrailblazeConnectedDeviceSummary( + trailblazeDriverType = TrailblazeDriverType.REVYL_IOS, + instanceId = "revyl-ios-iphone", + description = "Revyl iOS (Default)", + ) + ) + + // Dynamically register each model from the Revyl device catalog. + // Uses `revyl device targets --json` so the list stays in sync + // with the backend without code changes. + try { + val targets = revylCliClient.getDeviceTargets() + for (target in targets) { + val driverType = if (target.platform == "android") + TrailblazeDriverType.REVYL_ANDROID else TrailblazeDriverType.REVYL_IOS + add( + TrailblazeConnectedDeviceSummary( + trailblazeDriverType = driverType, + instanceId = "revyl-model:${target.model}::${target.osVersion}", + description = "Revyl ${target.model} (${target.osVersion})", + ) + ) + } + } catch (e: Exception) { + Console.log("Revyl device catalog unavailable: ${e.message}") + } } // Always filter for device state — all three Android driver variants share the same @@ -705,6 +753,8 @@ class TrailblazeDeviceManager( fun getCurrentSelectedTargetApp(): TrailblazeHostAppTarget? = settingsRepo.getCurrentSelectedTargetApp() + private val revylCliClient: RevylCliClient by lazy { RevylCliClient() } + // Store running test instances per device - allows forceful driver shutdown private val maestroDriverByDeviceMap: MutableMap = java.util.concurrent.ConcurrentHashMap() diff --git a/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeSettingsRepo.kt b/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeSettingsRepo.kt index 76ecc3b4..b59fa313 100644 --- a/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeSettingsRepo.kt +++ b/trailblaze-host/src/main/java/xyz/block/trailblaze/ui/TrailblazeSettingsRepo.kt @@ -65,28 +65,19 @@ class TrailblazeSettingsRepo( fun applyTestingEnvironment(environment: TrailblazeServerState.TestingEnvironment) { updateAppConfig { config -> val existingDriverTypes = config.selectedTrailblazeDriverTypes - val driverTypes = when (environment) { - TrailblazeServerState.TestingEnvironment.MOBILE -> { - val defaultDriverTypes = mapOf( - TrailblazeDevicePlatform.ANDROID to TrailblazeDriverType.DEFAULT_ANDROID_ON_DEVICE, - TrailblazeDevicePlatform.IOS to TrailblazeDriverType.IOS_HOST, - ) - defaultDriverTypes.mapValues { (platform, defaultDriverType) -> - existingDriverTypes[platform] ?: defaultDriverType - } - } - TrailblazeServerState.TestingEnvironment.WEB -> { - val defaultDriverTypes = mapOf( - TrailblazeDevicePlatform.WEB to TrailblazeDriverType.PLAYWRIGHT_NATIVE, - ) - defaultDriverTypes.mapValues { (platform, defaultDriverType) -> - existingDriverTypes[platform] ?: defaultDriverType - } - } + val defaults = when (environment) { + TrailblazeServerState.TestingEnvironment.MOBILE -> mapOf( + TrailblazeDevicePlatform.ANDROID to TrailblazeDriverType.DEFAULT_ANDROID_ON_DEVICE, + TrailblazeDevicePlatform.IOS to TrailblazeDriverType.IOS_HOST, + ) + TrailblazeServerState.TestingEnvironment.WEB -> mapOf( + TrailblazeDevicePlatform.WEB to TrailblazeDriverType.PLAYWRIGHT_NATIVE, + ) } + val merged = existingDriverTypes + defaults.filterKeys { it !in existingDriverTypes } config.copy( testingEnvironment = environment, - selectedTrailblazeDriverTypes = driverTypes, + selectedTrailblazeDriverTypes = merged, showDevicesTab = environment == TrailblazeServerState.TestingEnvironment.WEB, ) } diff --git a/trailblaze-host/src/test/java/xyz/block/trailblaze/host/revyl/RevylDemo.kt b/trailblaze-host/src/test/java/xyz/block/trailblaze/host/revyl/RevylDemo.kt new file mode 100644 index 00000000..c01db957 --- /dev/null +++ b/trailblaze-host/src/test/java/xyz/block/trailblaze/host/revyl/RevylDemo.kt @@ -0,0 +1,99 @@ +package xyz.block.trailblaze.host.revyl + +import xyz.block.trailblaze.revyl.RevylCliClient + +/** + * Standalone demo that drives a Revyl cloud device through a Bug Bazaar + * e-commerce flow using [RevylCliClient] — the same integration layer + * that [RevylTrailblazeAgent] uses for all tool execution. + * + * Each step prints the equivalent `revyl device` CLI command and the + * resolved coordinates returned by the CLI's AI grounding. + * + * Usage: + * ./gradlew :trailblaze-host:run -PmainClass=xyz.block.trailblaze.host.revyl.RevylDemoKt + */ + +private const val BUG_BAZAAR_APK = + "https://pub-b03f222a53c447c18ef5f8d365a2f00e.r2.dev/bug-bazaar/bug-bazaar-preview.apk" + +private const val BUG_BAZAAR_IOS = + "https://pub-b03f222a53c447c18ef5f8d365a2f00e.r2.dev/bug-bazaar/bug-bazaar-preview-simulator.tar.gz" + +private const val BUNDLE_ID = "com.bugbazaar.app" + +fun main() { + val client = RevylCliClient() + + println("\n=== Trailblaze x Revyl Demo ===") + println("Each step shows the Kotlin call, CLI command, AND resolved coordinates.\n") + + // ── Step 0: Provision device + install app ───────────────────────── + // CLI: revyl device start --platform android --app-url --json + println("Step 0: Start device + install Bug Bazaar") + val session = client.startSession( + platform = "android", + appUrl = BUG_BAZAAR_APK, + ) + println(" Viewer: ${session.viewerUrl}") + println(" Screen: ${session.screenWidth}x${session.screenHeight}") + + // CLI: revyl device launch --bundle-id com.bugbazaar.app --json + println("\nStep 0b: Launch app") + client.launchApp(BUNDLE_ID) + Thread.sleep(2000) + + // ── Step 1: Screenshot home ──────────────────────────────────────── + // CLI: revyl device screenshot --out flow-01-home.png --json + println("\nStep 1: Screenshot home screen") + client.screenshot("flow-01-home.png") + + // ── Step 2: Navigate to search ───────────────────────────────────── + // CLI: revyl device tap --target "Search tab" --json + println("\nStep 2: Tap Search tab") + val r2 = client.tapTarget("Search tab") + println(" -> Tapped at (${r2.x}, ${r2.y})") + Thread.sleep(1000) + + // ── Step 3: Search for "beetle" ──────────────────────────────────── + // CLI: revyl device type --target "search input field" --text "beetle" --json + println("\nStep 3: Type 'beetle' in search field") + val r3 = client.typeText("beetle", target = "search input field") + println(" -> Typed at (${r3.x}, ${r3.y})") + Thread.sleep(1000) + client.screenshot("flow-02-search.png") + + // ── Step 4: Open product detail ──────────────────────────────────── + // CLI: revyl device tap --target "Hercules Beetle" --json + println("\nStep 4: Tap Hercules Beetle result") + val r4 = client.tapTarget("Hercules Beetle") + println(" -> Tapped at (${r4.x}, ${r4.y})") + Thread.sleep(1000) + client.screenshot("flow-03-product.png") + + // ── Step 5: Add to cart ──────────────────────────────────────────── + // CLI: revyl device tap --target "Add to Cart button" --json + println("\nStep 5: Tap Add to Cart") + val r5 = client.tapTarget("Add to Cart button") + println(" -> Tapped at (${r5.x}, ${r5.y})") + Thread.sleep(1000) + + // ── Step 6: Back to home ─────────────────────────────────────────── + // CLI: revyl device back --json + println("\nStep 6: Navigate back to home") + val r6a = client.back() + println(" -> Back at (${r6a.x}, ${r6a.y})") + val r6b = client.back() + println(" -> Back at (${r6b.x}, ${r6b.y})") + Thread.sleep(1000) + client.screenshot("flow-04-done.png") + + // ── Done ─────────────────────────────────────────────────────────── + println("\n=== Demo complete ===") + println("Session viewer: ${session.viewerUrl}") + println("Screen dimensions: ${session.screenWidth}x${session.screenHeight}") + println("Screenshots: flow-01-home.png … flow-04-done.png") + println("\nStop device:") + println(" CLI: revyl device stop") + println(" Kotlin: client.stopSession()") +} diff --git a/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/devices/TrailblazeDriverType.kt b/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/devices/TrailblazeDriverType.kt index 34eeb14c..becae776 100644 --- a/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/devices/TrailblazeDriverType.kt +++ b/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/devices/TrailblazeDriverType.kt @@ -28,6 +28,14 @@ enum class TrailblazeDriverType( platform = TrailblazeDevicePlatform.WEB, isHost = true, ), + REVYL_ANDROID( + platform = TrailblazeDevicePlatform.ANDROID, + isHost = true, + ), + REVYL_IOS( + platform = TrailblazeDevicePlatform.IOS, + isHost = true, + ), // COMPOSE intentionally uses WEB platform: Compose Desktop testing reuses the web // platform's view hierarchy filtering and device infrastructure. Adding a separate // DESKTOP platform would require updating all exhaustive `when` expressions on diff --git a/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/model/TrailblazeConfig.kt b/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/model/TrailblazeConfig.kt index 2b4400a4..42246713 100644 --- a/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/model/TrailblazeConfig.kt +++ b/trailblaze-models/src/commonMain/kotlin/xyz/block/trailblaze/model/TrailblazeConfig.kt @@ -45,6 +45,11 @@ enum class NodeSelectorMode { * if false, disables AI fallback (useful for debugging recorded steps). * @property browserHeadless If true, the Playwright browser runs headless (no visible window); * if false, the browser window is shown on screen. + * @property useRevylNativeSteps If true, Revyl device prompt steps use + * `revyl device instruction` / `validation` instead of the + * LLM agent pipeline. Defaults to false (LLM pipeline) until + * native step reporting with screenshots is implemented. + * Only affects REVYL_ANDROID/REVYL_IOS drivers. */ @Serializable data class TrailblazeConfig( @@ -55,6 +60,7 @@ data class TrailblazeConfig( val overrideSessionId: SessionId? = null, val aiFallback: Boolean = AI_FALLBACK_DEFAULT, val browserHeadless: Boolean = true, + val useRevylNativeSteps: Boolean = false, val nodeSelectorMode: NodeSelectorMode = FORCE_LEGACY, ) { companion object { diff --git a/trailblaze-revyl/build.gradle.kts b/trailblaze-revyl/build.gradle.kts new file mode 100644 index 00000000..1b793c3e --- /dev/null +++ b/trailblaze-revyl/build.gradle.kts @@ -0,0 +1,20 @@ +plugins { + alias(libs.plugins.kotlin.jvm) + alias(libs.plugins.kotlin.serialization) +} + +dependencies { + api(project(":trailblaze-common")) + api(project(":trailblaze-agent")) + implementation(project(":trailblaze-tracing")) + + implementation(libs.kotlinx.serialization.json) + implementation(libs.koog.agents.tools) + + testImplementation(libs.kotlin.test.junit4) + testImplementation(libs.assertk) +} + +tasks.test { useJUnit() } + +project.tasks.named("check") { dependsOn.removeIf { it.toString().contains("test") } } diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylActionResult.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylActionResult.kt new file mode 100644 index 00000000..93688a2d --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylActionResult.kt @@ -0,0 +1,59 @@ +package xyz.block.trailblaze.revyl + +import kotlinx.serialization.SerialName +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.JsonElement +import xyz.block.trailblaze.logs.client.TrailblazeJsonInstance +import xyz.block.trailblaze.util.Console + +/** + * Structured result from a Revyl CLI device action (tap, type, swipe, etc.). + * + * Maps 1:1 to the `ActionResult` struct returned by the Go CLI's `--json` + * output. Contains the resolved coordinates after AI grounding, which + * downstream consumers like Trailblaze use for click overlay rendering. + * + * @property action The action type (e.g. "tap", "swipe", "type"). + * @property x Resolved horizontal pixel coordinate. + * @property y Resolved vertical pixel coordinate. + * @property target The natural language target that was grounded, if any. + * @property success Whether the worker reported success. + * @property latencyMs Round-trip latency in milliseconds. + * @property durationMs Hold duration for long-press actions. + * @property text Typed text for type actions. + * @property direction Swipe direction for swipe actions. + */ +@Serializable +data class RevylActionResult( + val action: String = "", + val x: Int = 0, + val y: Int = 0, + val target: String? = null, + val success: Boolean = true, + @SerialName("latency_ms") + val latencyMs: JsonElement? = null, + @SerialName("duration_ms") + val durationMs: Int = 0, + val text: String? = null, + val direction: String? = null, +) { + companion object { + /** + * Parses a CLI JSON stdout line into a [RevylActionResult]. + * + * Falls back to a default (success=true, coords=0,0) if parsing fails, + * since the CLI already succeeded if we got stdout. + * + * @param jsonString Raw JSON from `revyl device --json`. + * @return Parsed result with coordinates and metadata. + */ + fun fromJson(jsonString: String): RevylActionResult { + return try { + TrailblazeJsonInstance.decodeFromString(jsonString.trim()) + } catch (e: Exception) { + Console.log("RevylActionResult: JSON parse failed, using default: ${e.message}") + RevylActionResult(success = true) + } + } + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylCliClient.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylCliClient.kt new file mode 100644 index 00000000..02a21374 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylCliClient.kt @@ -0,0 +1,606 @@ +package xyz.block.trailblaze.revyl + +import kotlinx.serialization.json.int +import kotlinx.serialization.json.intOrNull +import kotlinx.serialization.json.jsonArray +import kotlinx.serialization.json.jsonObject +import kotlinx.serialization.json.jsonPrimitive +import xyz.block.trailblaze.logs.client.TrailblazeJsonInstance +import xyz.block.trailblaze.util.Console +import java.io.File + +/** + * Device interaction client that delegates to the `revyl` CLI binary. + * + * Every method builds a `revyl device --json` process, + * executes it, and parses the structured JSON output. The CLI handles + * auth, backend proxy routing, and AI-powered target grounding. + * + * The `revyl` binary is resolved from PATH by default, or can be pointed to via + * `REVYL_BINARY` for local/dev overrides. If not found, [startSession] throws a + * [RevylCliException] with install instructions. + * + * @property revylBinaryOverride Optional explicit path to the revyl binary. + * Defaults to `REVYL_BINARY`; if null, PATH lookup uses `revyl`. + * @property workingDirectory Optional working directory for CLI + * invocations. Defaults to the JVM's current directory. + */ +class RevylCliClient( + private val revylBinaryOverride: String? = System.getenv("REVYL_BINARY"), + private val workingDirectory: File? = null, +) { + + private val json = TrailblazeJsonInstance + private val sessions = mutableMapOf() + private var activeSessionIndex: Int = ACTIVE_SESSION + + private val resolvedBinary: String = revylBinaryOverride ?: "revyl" + private var cliVerified = false + + /** + * Returns the currently active Revyl session, or null if none has been started. + */ + fun getActiveRevylSession(): RevylSession? = sessions[activeSessionIndex] + + /** + * Returns the session at the given index, or null if not found. + * + * @param index Session index to retrieve. + */ + fun getSession(index: Int): RevylSession? = sessions[index] + + /** + * Finds a session by its workflow run ID. + * + * @param workflowRunId The Hatchet workflow run identifier. + * @return The matching session, or null if not found. + */ + fun getSession(workflowRunId: String): RevylSession? = + sessions.values.firstOrNull { it.workflowRunId == workflowRunId } + + /** + * Returns all active sessions. + */ + fun getAllSessions(): Map = sessions.toMap() + + /** + * Switches the active session to the given index. + * + * @param index Session index to make active. + * @throws IllegalArgumentException If no session exists at the given index. + */ + fun useSession(index: Int) { + require(sessions.containsKey(index)) { "No session at index $index. Active sessions: ${sessions.keys}" } + activeSessionIndex = index + Console.log("RevylCli: switched to session $index (${sessions[index]!!.platform})") + } + + companion object { + /** Sentinel value for "use the currently active session". */ + const val ACTIVE_SESSION = -1 + + /** Environment variable name for the Revyl API key. */ + const val REVYL_API_KEY_ENV = "REVYL_API_KEY" + + /** Platform identifier for iOS devices. */ + const val PLATFORM_IOS = "ios" + + /** Platform identifier for Android devices. */ + const val PLATFORM_ANDROID = "android" + } + + // --------------------------------------------------------------------------- + // CLI verification + // --------------------------------------------------------------------------- + + /** + * Verifies the revyl CLI binary is available on PATH. Called lazily on first + * device provisioning so construction never throws. + * + * @throws RevylCliException If the binary is not found, with install instructions. + */ + private fun verifyCliAvailable() { + if (cliVerified) return + try { + val process = ProcessBuilder(resolvedBinary, "--version") + .redirectErrorStream(true) + .start() + val output = process.inputStream.bufferedReader().readText().trim() + if (process.waitFor() == 0) { + Console.log("RevylCli: $output") + cliVerified = true + val installedVersion = output.substringAfterLast(" ", "").takeIf { it.startsWith("v") } + if (installedVersion != null) checkForUpgrade(installedVersion) + return + } + } catch (_: Exception) { /* binary not found */ } + + throw RevylCliException( + "revyl CLI not found on PATH.\n\n" + + "Install:\n" + + " curl -fsSL https://raw.githubusercontent.com/RevylAI/revyl-cli/main/scripts/install.sh | sh\n\n" + + "Or with Homebrew:\n" + + " brew install RevylAI/tap/revyl\n\n" + + "Or download from:\n" + + " https://github.com/RevylAI/revyl-cli/releases\n\n" + + "Then set REVYL_API_KEY and try again." + ) + } + + /** + * Checks whether a newer CLI version is available on GitHub and logs + * a warning if so. Never throws -- network failures are silently ignored. + * + * @param installedVersion The currently installed version tag (e.g. "v0.1.14"). + */ + private fun checkForUpgrade(installedVersion: String) { + try { + val url = java.net.URL("https://github.com/RevylAI/revyl-cli/releases/latest") + val conn = url.openConnection() as java.net.HttpURLConnection + conn.instanceFollowRedirects = false + conn.connectTimeout = 3000 + conn.readTimeout = 3000 + val location = conn.getHeaderField("Location") + conn.disconnect() + val latest = location?.substringAfterLast("/")?.takeIf { it.startsWith("v") } ?: return + if (latest != installedVersion) { + Console.log( + "RevylCli: update available $installedVersion -> $latest.\n" + + " Upgrade: curl -fsSL https://raw.githubusercontent.com/RevylAI/revyl-cli/main/scripts/install.sh | sh\n" + + " Or: brew upgrade revyl" + ) + } + } catch (_: Exception) { /* network unavailable -- skip silently */ } + } + + /** + * Returns true if the revyl CLI binary is available on PATH. + * Does not throw -- suitable for feature-gating Revyl device types. + */ + fun isCliAvailable(): Boolean { + if (cliVerified) return true + return try { + val process = ProcessBuilder(resolvedBinary, "--version") + .redirectErrorStream(true) + .start() + val available = process.waitFor() == 0 + if (available) cliVerified = true + available + } catch (_: Exception) { + false + } + } + + // --------------------------------------------------------------------------- + // Session lifecycle + // --------------------------------------------------------------------------- + + /** + * Provisions a cloud device by running `revyl device start`. + * + * @param platform "ios" or "android". + * @param appUrl Optional public URL to an .apk/.ipa to install on start. + * @param appLink Optional deep-link to open after launch. + * @param deviceModel Optional explicit device model (e.g. "Pixel 7"). + * @param osVersion Optional explicit OS version (e.g. "Android 14"). + * @param appId Optional Revyl `apps` table UUID. The backend resolves the + * latest build artifact and package name from this id. + * @param buildVersionId Optional `builds` table UUID. When provided the + * backend uses this exact build instead of resolving the latest. + * @return The newly created [RevylSession] parsed from CLI JSON output. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun startSession( + platform: String, + appUrl: String? = null, + appLink: String? = null, + deviceModel: String? = null, + osVersion: String? = null, + appId: String? = null, + buildVersionId: String? = null, + ): RevylSession { + verifyCliAvailable() + val args = mutableListOf("device", "start", "--platform", platform.lowercase()) + if (!appUrl.isNullOrBlank()) { + args += listOf("--app-url", appUrl) + } + if (!appLink.isNullOrBlank()) { + args += listOf("--app-link", appLink) + } + if (!deviceModel.isNullOrBlank()) { + args += listOf("--device-model", deviceModel) + } + if (!osVersion.isNullOrBlank()) { + args += listOf("--os-version", osVersion) + } + if (!appId.isNullOrBlank()) { + args += listOf("--app-id", appId) + } + if (!buildVersionId.isNullOrBlank()) { + args += listOf("--build-version-id", buildVersionId) + } + + val result = runCli(args) + val obj = json.parseToJsonElement(result).jsonObject + + val session = RevylSession( + index = obj["index"]?.jsonPrimitive?.int ?: 0, + sessionId = obj["session_id"]?.jsonPrimitive?.content ?: "", + workflowRunId = obj["workflow_run_id"]?.jsonPrimitive?.content ?: "", + workerBaseUrl = obj["worker_base_url"]?.jsonPrimitive?.content ?: "", + viewerUrl = obj["viewer_url"]?.jsonPrimitive?.content ?: "", + platform = platform.lowercase(), + screenWidth = obj["screen_width"]?.jsonPrimitive?.intOrNull ?: 0, + screenHeight = obj["screen_height"]?.jsonPrimitive?.intOrNull ?: 0, + ) + sessions[session.index] = session + activeSessionIndex = session.index + Console.log("RevylCli: device ready (session ${session.index}, ${session.platform})") + Console.log(" Viewer: ${session.viewerUrl}") + if (session.screenWidth > 0) { + Console.log(" Screen: ${session.screenWidth}x${session.screenHeight}") + } + return session + } + + /** + * Stops a device session and removes it from the local session map. + * + * @param index Session index to stop. Defaults to [ACTIVE_SESSION]. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun stopSession(index: Int = ACTIVE_SESSION) { + val targetIndex = if (index >= 0) index else activeSessionIndex + val args = mutableListOf("device", "stop") + if (targetIndex >= 0) args += listOf("-s", targetIndex.toString()) + runCli(args) + sessions.remove(targetIndex) + if (activeSessionIndex == targetIndex) { + activeSessionIndex = if (sessions.isNotEmpty()) sessions.keys.first() else ACTIVE_SESSION + } + } + + /** + * Stops all active device sessions. + * + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun stopAllSessions() { + runCli(listOf("device", "stop", "--all")) + sessions.clear() + } + + // --------------------------------------------------------------------------- + // Device actions — all return RevylActionResult with coordinates + // --------------------------------------------------------------------------- + + private fun deviceArgs(vararg args: String): List { + val base = mutableListOf("device") + if (activeSessionIndex >= 0) { + base += listOf("-s", activeSessionIndex.toString()) + } + base += args.toList() + return base + } + + /** + * Captures a PNG screenshot and returns the raw bytes. + * + * @param outPath File path to write the screenshot to. + * @return Raw PNG bytes read from the output file. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun screenshot(outPath: String = createTempScreenshotPath()): ByteArray { + runCli(deviceArgs("screenshot", "--out", outPath)) + val file = File(outPath) + if (!file.exists()) { + throw RevylCliException("Screenshot file not found at $outPath") + } + return try { + file.readBytes() + } finally { + if (file.absolutePath.contains("revyl-screenshot-")) { + file.delete() + } + } + } + + /** + * Taps at exact pixel coordinates on the device screen. + * + * @param x Horizontal pixel coordinate. + * @param y Vertical pixel coordinate. + * @return Action result with the tapped coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun tap(x: Int, y: Int): RevylActionResult { + val stdout = runCli(deviceArgs("tap", "--x", x.toString(), "--y", y.toString())) + return RevylActionResult.fromJson(stdout) + } + + /** + * Taps a UI element identified by natural language description. + * + * @param target Natural language description (e.g. "Sign In button"). + * @return Action result with the resolved coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun tapTarget(target: String): RevylActionResult { + val stdout = runCli(deviceArgs("tap", "--target", target)) + return RevylActionResult.fromJson(stdout) + } + + /** + * Double-taps a UI element identified by natural language description. + * + * @param target Natural language description of the element to double-tap. + * @return Action result with the resolved coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun doubleTap(target: String): RevylActionResult { + val stdout = runCli(deviceArgs("double-tap", "--target", target)) + return RevylActionResult.fromJson(stdout) + } + + /** + * Types text into an input field, optionally targeting a specific element. + * + * @param text The text to type. + * @param target Optional natural language element description to tap first. + * @param clearFirst If true, clears the field before typing. + * @return Action result with the field coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun typeText(text: String, target: String? = null, clearFirst: Boolean = false): RevylActionResult { + val args = deviceArgs("type", "--text", text).toMutableList() + if (!target.isNullOrBlank()) args += listOf("--target", target) + args += "--clear-first=$clearFirst" + val stdout = runCli(args) + return RevylActionResult.fromJson(stdout) + } + + /** + * Swipes in the given direction, optionally from a targeted element. + * + * @param direction One of "up", "down", "left", "right". + * @param target Optional natural language element description for swipe origin. + * @return Action result with the swipe origin coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun swipe(direction: String, target: String? = null): RevylActionResult { + val args = deviceArgs("swipe", "--direction", direction).toMutableList() + if (!target.isNullOrBlank()) args += listOf("--target", target) + val stdout = runCli(args) + return RevylActionResult.fromJson(stdout) + } + + /** + * Long-presses a UI element identified by natural language description. + * + * @param target Natural language description of the element. + * @return Action result with the pressed coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun longPress(target: String): RevylActionResult { + val stdout = runCli(deviceArgs("long-press", "--target", target)) + return RevylActionResult.fromJson(stdout) + } + + /** + * Presses the Android back button. + * + * @return Action result (coordinates are 0,0 for back). + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun back(): RevylActionResult { + val stdout = runCli(deviceArgs("back")) + return RevylActionResult.fromJson(stdout) + } + + /** + * Sends a key press event (ENTER or BACKSPACE). + * + * @param key Key name: "ENTER" or "BACKSPACE". + * @return Action result (coordinates are 0,0 for key presses). + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun pressKey(key: String): RevylActionResult { + val stdout = runCli(deviceArgs("key", "--key", key.uppercase())) + return RevylActionResult.fromJson(stdout) + } + + /** + * Opens a URL or deep link on the device. + * + * @param url The URL or deep link to open. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun navigate(url: String) { + runCli(deviceArgs("navigate", "--url", url)) + } + + /** + * Clears text from the currently focused input field. + * + * @param target Optional natural language element description. + * @return Action result with the field coordinates. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun clearText(target: String? = null): RevylActionResult { + val args = deviceArgs("clear-text").toMutableList() + if (!target.isNullOrBlank()) args += listOf("--target", target) + val stdout = runCli(args) + return RevylActionResult.fromJson(stdout) + } + + /** + * Installs an app on the device from a public URL. + * + * @param appUrl Direct download URL for the .apk or .ipa. + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun installApp(appUrl: String) { + runCli(deviceArgs("install", "--app-url", appUrl)) + } + + /** + * Launches an installed app by its bundle/package ID. + * + * @param bundleId The app's bundle identifier (iOS) or package name (Android). + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun launchApp(bundleId: String) { + runCli(deviceArgs("launch", "--bundle-id", bundleId)) + } + + /** + * Navigates to the device home screen. + * + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun home() { + runCli(deviceArgs("home")) + } + + /** + * Toggles device network connectivity (airplane mode). + * + * @param connected true to enable network (disable airplane mode), + * false to disable network (enable airplane mode). + * @throws RevylCliException If the CLI exits with a non-zero code. + */ + fun setNetworkConnected(connected: Boolean) { + runCli(deviceArgs("network", if (connected) "--connected" else "--disconnected")) + } + + // --------------------------------------------------------------------------- + // Device catalog + // --------------------------------------------------------------------------- + + /** + * Queries the available device models from the Revyl backend catalog. + * + * Calls `revyl device targets --json` and parses the response into a + * deduplicated list of [RevylDeviceTarget] entries (one per unique model). + * + * @return Available device models grouped by platform. + * @throws RevylCliException If the CLI command fails. + */ + fun getDeviceTargets(): List { + val stdout = runCli(listOf("device", "targets")) + val root = json.parseToJsonElement(stdout).jsonObject + val results = mutableListOf() + val seenModels = mutableSetOf() + + for (platform in listOf(PLATFORM_ANDROID, PLATFORM_IOS)) { + val entries = root[platform]?.jsonArray ?: continue + for (entry in entries) { + val model = entry.jsonObject["Model"]?.jsonPrimitive?.content ?: continue + val runtime = entry.jsonObject["Runtime"]?.jsonPrimitive?.content ?: continue + if (seenModels.add("$platform:$model")) { + results.add(RevylDeviceTarget(platform = platform, model = model, osVersion = runtime)) + } + } + } + return results + } + + // --------------------------------------------------------------------------- + // High-level steps (instruction / validation) + // --------------------------------------------------------------------------- + + /** + * Executes a natural-language instruction step on the active device via + * `revyl device instruction "" --json`. + * + * Revyl's worker agent handles planning, grounding, and execution + * in a single round-trip. + * + * @param description Natural-language instruction (e.g. "Tap the Search tab"). + * @return Parsed [RevylLiveStepResult] with success flag and step output. + * @throws RevylCliException If the CLI process exits with a non-zero code. + */ + fun instruction(description: String): RevylLiveStepResult { + val stdout = runCli(deviceArgs("instruction", description)) + return RevylLiveStepResult.fromJson(stdout) + } + + /** + * Executes a natural-language validation step on the active device via + * `revyl device validation "" --json`. + * + * Revyl's worker agent performs a visual assertion against the current + * screen state and returns a pass/fail result. + * + * @param description Natural-language assertion (e.g. "The search results are visible"). + * @return Parsed [RevylLiveStepResult] with success flag and step output. + * @throws RevylCliException If the CLI process exits with a non-zero code. + */ + fun validation(description: String): RevylLiveStepResult { + val stdout = runCli(deviceArgs("validation", description)) + return RevylLiveStepResult.fromJson(stdout) + } + + // --------------------------------------------------------------------------- + // CLI execution + // --------------------------------------------------------------------------- + + /** + * Executes a revyl CLI command with `--json` and returns stdout. + * + * Inherits `REVYL_API_KEY` and other env vars from the parent process. + * On non-zero exit, throws [RevylCliException] with stderr content. + * + * @param args Command arguments after the binary name. + * @return Stdout content from the CLI process. + * @throws RevylCliException If the process exits with a non-zero code. + */ + private fun runCli(args: List): String { + val command = listOf(resolvedBinary) + args + "--json" + Console.log("RevylCli: ${command.joinToString(" ")}") + + val processBuilder = ProcessBuilder(command) + .redirectErrorStream(false) + + if (workingDirectory != null) { + processBuilder.directory(workingDirectory) + } + + val process = processBuilder.start() + val stdout = process.inputStream.bufferedReader().readText() + val stderr = process.errorStream.bufferedReader().readText() + val exitCode = process.waitFor() + + if (exitCode != 0) { + val errorDetail = stderr.ifBlank { stdout } + throw RevylCliException( + "revyl ${args.firstOrNull() ?: ""} ${args.getOrNull(1) ?: ""} " + + "failed (exit $exitCode): ${errorDetail.take(500)}" + ) + } + + return stdout.trim() + } + + private fun createTempScreenshotPath(): String { + val tempFile = File.createTempFile("revyl-screenshot-", ".png") + tempFile.deleteOnExit() + return tempFile.absolutePath + } +} + +/** + * Exception thrown when a revyl CLI command fails. + * + * @property message Human-readable description including the exit code and stderr. + */ +class RevylCliException(message: String) : RuntimeException(message) + +/** + * A device model available in the Revyl cloud catalog. + * + * @property platform One of [RevylCliClient.PLATFORM_IOS] or [RevylCliClient.PLATFORM_ANDROID]. + * @property model Human-readable model name (e.g. "iPhone 16", "Pixel 7"). + * @property osVersion Runtime / OS version string (e.g. "Android 14", "iOS 18.2"). + */ +data class RevylDeviceTarget(val platform: String, val model: String, val osVersion: String) diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylDefaults.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylDefaults.kt new file mode 100644 index 00000000..14dd99f7 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylDefaults.kt @@ -0,0 +1,30 @@ +package xyz.block.trailblaze.revyl + +/** + * Default screen dimensions for Revyl cloud devices. + * + * These are fallback values used when the session does not report + * screen dimensions. They represent the native resolution of the + * default device provisioned by the Revyl backend for each platform. + */ +object RevylDefaults { + + /** iPhone 16 Pro native resolution (logical points x 3). */ + const val IOS_DEFAULT_WIDTH = 1170 + const val IOS_DEFAULT_HEIGHT = 2532 + + /** Pixel 7 native resolution. */ + const val ANDROID_DEFAULT_WIDTH = 1080 + const val ANDROID_DEFAULT_HEIGHT = 2400 + + /** + * Returns the default (width, height) pair for the given platform string. + * + * @param platform "ios" or "android". + * @return Default dimensions as a Pair. + */ + fun dimensionsForPlatform(platform: String): Pair = when (platform.lowercase()) { + RevylCliClient.PLATFORM_IOS -> Pair(IOS_DEFAULT_WIDTH, IOS_DEFAULT_HEIGHT) + else -> Pair(ANDROID_DEFAULT_WIDTH, ANDROID_DEFAULT_HEIGHT) + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylLiveStepResult.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylLiveStepResult.kt new file mode 100644 index 00000000..39de555a --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylLiveStepResult.kt @@ -0,0 +1,58 @@ +package xyz.block.trailblaze.revyl + +import kotlinx.serialization.SerialName +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.JsonObject +import kotlinx.serialization.json.contentOrNull +import kotlinx.serialization.json.jsonPrimitive +import xyz.block.trailblaze.logs.client.TrailblazeJsonInstance +import xyz.block.trailblaze.util.Console + +/** + * Structured result from a Revyl CLI live-step command (`revyl device instruction` + * or `revyl device validation`). + * + * Maps to the `LiveStepResponse` struct returned by the Go CLI's `--json` output. + * The [stepOutput] object carries agent-specific detail such as `status`, + * `status_reason`, and `validation_result`. + * + * @property success Whether the step completed successfully. + * @property stepType The step type that was executed ("instruction" or "validation"). + * @property stepId Unique identifier assigned to the step by the worker. + * @property stepOutput Agent output detail including status reason and validation result. + */ +@Serializable +data class RevylLiveStepResult( + val success: Boolean = false, + @SerialName("step_type") val stepType: String = "", + @SerialName("step_id") val stepId: String = "", + @SerialName("step_output") val stepOutput: JsonObject? = null, +) { + + /** + * Human-readable reason for the step outcome, extracted from [stepOutput]. + * Returns null when the field is absent or the output is unparseable. + */ + val statusReason: String? + get() = stepOutput?.get("status_reason")?.jsonPrimitive?.contentOrNull + + companion object { + /** + * Parses CLI JSON stdout into a [RevylLiveStepResult]. + * + * Falls back to a failure result if parsing fails so callers always + * get a non-null object to inspect. + * + * @param jsonString Raw JSON from `revyl device instruction/validation --json`. + * @return Parsed result with success flag and step output. + */ + fun fromJson(jsonString: String): RevylLiveStepResult { + return try { + TrailblazeJsonInstance.decodeFromString(jsonString.trim()) + } catch (e: Exception) { + Console.log("RevylLiveStepResult: JSON parse failed: ${e.message}") + RevylLiveStepResult(success = false) + } + } + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylScreenState.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylScreenState.kt new file mode 100644 index 00000000..608130be --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylScreenState.kt @@ -0,0 +1,100 @@ +package xyz.block.trailblaze.revyl + +import xyz.block.trailblaze.api.ScreenState +import xyz.block.trailblaze.api.ViewHierarchyTreeNode +import xyz.block.trailblaze.devices.TrailblazeDeviceClassifier +import xyz.block.trailblaze.devices.TrailblazeDevicePlatform +import java.nio.ByteBuffer +import java.nio.ByteOrder + +/** + * [ScreenState] backed by Revyl CLI screenshots. + * + * Since Revyl uses AI-powered visual grounding (not accessibility trees), + * the view hierarchy is a minimal root node. The LLM agent relies on + * screenshot-based reasoning instead of element trees. + * + * Screen dimensions are resolved in priority order: + * 1. Session-reported dimensions from the worker health endpoint. + * 2. PNG IHDR header extraction from the captured screenshot. + * 3. Default fallback constants from [RevylDefaults]. + * + * @param cliClient CLI client used to capture screenshots. + * @param platform The device platform ("ios" or "android"). + * @param sessionScreenWidth Session-reported width in pixels (0 = unknown). + * @param sessionScreenHeight Session-reported height in pixels (0 = unknown). + */ +class RevylScreenState( + private val cliClient: RevylCliClient, + private val platform: String, + sessionScreenWidth: Int = 0, + sessionScreenHeight: Int = 0, +) : ScreenState { + + private val capturedScreenshot: ByteArray? = try { + cliClient.screenshot() + } catch (_: Exception) { + null + } + + private val defaultDimensions = RevylDefaults.dimensionsForPlatform(platform) + + private val dimensions: Pair = when { + sessionScreenWidth > 0 && sessionScreenHeight > 0 -> + Pair(sessionScreenWidth, sessionScreenHeight) + else -> + capturedScreenshot?.let { extractPngDimensions(it) } + ?: defaultDimensions + } + + override val screenshotBytes: ByteArray? = capturedScreenshot + + override val deviceWidth: Int = dimensions.first + + override val deviceHeight: Int = dimensions.second + + private val rootViewHierarchy: ViewHierarchyTreeNode = ViewHierarchyTreeNode( + nodeId = 1, + text = "RevylRootNode", + className = "RevylCloudDevice", + dimensions = "${deviceWidth}x$deviceHeight", + centerPoint = "${deviceWidth / 2},${deviceHeight / 2}", + clickable = true, + enabled = true, + ) + + override val viewHierarchy: ViewHierarchyTreeNode = rootViewHierarchy + + override val trailblazeDevicePlatform: TrailblazeDevicePlatform = when (platform.lowercase()) { + RevylCliClient.PLATFORM_IOS -> TrailblazeDevicePlatform.IOS + else -> TrailblazeDevicePlatform.ANDROID + } + + override val deviceClassifiers: List = listOf( + trailblazeDevicePlatform.asTrailblazeDeviceClassifier(), + TrailblazeDeviceClassifier("revyl-cloud"), + ) + + companion object { + /** + * Extracts width and height from a PNG file's IHDR chunk header. + * + * @param data Raw PNG bytes. + * @return (width, height) pair, or null if the data is not valid PNG. + */ + private fun extractPngDimensions(data: ByteArray): Pair? { + if (data.size < 24) return null + val pngSignature = byteArrayOf( + 0x89.toByte(), 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A + ) + for (i in pngSignature.indices) { + if (data[i] != pngSignature[i]) return null + } + val buffer = ByteBuffer.wrap(data, 16, 8).order(ByteOrder.BIG_ENDIAN) + val width = buffer.int + val height = buffer.int + if (width <= 0 || height <= 0) return null + return Pair(width, height) + } + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylSession.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylSession.kt new file mode 100644 index 00000000..4d6b20a4 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylSession.kt @@ -0,0 +1,44 @@ +package xyz.block.trailblaze.revyl + +import xyz.block.trailblaze.devices.TrailblazeDevicePlatform +import xyz.block.trailblaze.devices.TrailblazeDriverType + +/** + * Represents an active Revyl cloud device session. + * + * @property index Local session index (0-based). + * @property sessionId Unique identifier returned by the Revyl backend. + * @property workflowRunId Hatchet workflow run powering this session. + * @property workerBaseUrl HTTP base URL of the device worker (e.g. "https://worker-xxx.revyl.ai"). + * @property viewerUrl Browser URL for live device screen. + * @property platform "ios" or "android". + * @property screenWidth Device screen width in pixels (0 when unknown). + * @property screenHeight Device screen height in pixels (0 when unknown). + */ +data class RevylSession( + val index: Int, + val sessionId: String, + val workflowRunId: String, + val workerBaseUrl: String, + val viewerUrl: String, + val platform: String, + val screenWidth: Int = 0, + val screenHeight: Int = 0, +) { + + /** + * Maps this session's platform string to the corresponding [TrailblazeDriverType]. + */ + fun toDriverType(): TrailblazeDriverType = when (platform) { + RevylCliClient.PLATFORM_IOS -> TrailblazeDriverType.REVYL_IOS + else -> TrailblazeDriverType.REVYL_ANDROID + } + + /** + * Maps this session's platform string to the corresponding [TrailblazeDevicePlatform]. + */ + fun toDevicePlatform(): TrailblazeDevicePlatform = when (platform) { + RevylCliClient.PLATFORM_IOS -> TrailblazeDevicePlatform.IOS + else -> TrailblazeDevicePlatform.ANDROID + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylToolAgent.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylToolAgent.kt new file mode 100644 index 00000000..0b62a39e --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/RevylToolAgent.kt @@ -0,0 +1,125 @@ +package xyz.block.trailblaze.revyl + +import kotlinx.datetime.Clock +import xyz.block.trailblaze.AgentMemory +import xyz.block.trailblaze.api.ScreenState +import xyz.block.trailblaze.api.TrailblazeAgent +import xyz.block.trailblaze.api.TrailblazeAgent.RunTrailblazeToolsResult +import xyz.block.trailblaze.devices.TrailblazeDeviceId +import xyz.block.trailblaze.devices.TrailblazeDeviceInfo +import xyz.block.trailblaze.devices.TrailblazeDevicePlatform +import xyz.block.trailblaze.devices.TrailblazeDriverType +import xyz.block.trailblaze.logs.client.TrailblazeLogger +import xyz.block.trailblaze.logs.client.TrailblazeSession +import xyz.block.trailblaze.logs.client.TrailblazeSessionProvider +import xyz.block.trailblaze.logs.model.SessionId +import xyz.block.trailblaze.logs.model.TraceId +import xyz.block.trailblaze.revyl.tools.RevylExecutableTool +import xyz.block.trailblaze.toolcalls.TrailblazeTool +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.toolcalls.commands.ObjectiveStatusTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.memory.MemoryTrailblazeTool +import xyz.block.trailblaze.toolcalls.getToolNameFromAnnotation +import xyz.block.trailblaze.util.Console +import xyz.block.trailblaze.utils.ElementComparator + +/** + * [TrailblazeAgent] that dispatches [RevylExecutableTool] instances against a + * Revyl cloud device via [RevylCliClient]. + * + * Unlike [RevylTrailblazeAgent] in trailblaze-host (which maps generic Trailblaze + * mobile tools to CLI commands), this agent handles Revyl-specific tool types that + * use natural language targeting and return resolved coordinates. + * + * @param cliClient CLI-based client for Revyl device interactions. + * @param platform "ios" or "android" for ScreenState construction. + */ +class RevylToolAgent( + private val cliClient: RevylCliClient, + private val platform: String, +) : TrailblazeAgent { + + override fun runTrailblazeTools( + tools: List, + traceId: TraceId?, + screenState: ScreenState?, + elementComparator: ElementComparator, + screenStateProvider: (() -> ScreenState)?, + ): RunTrailblazeToolsResult { + val executed = mutableListOf() + val effectiveScreenStateProvider = screenStateProvider ?: { RevylScreenState(cliClient, platform) } + + for (tool in tools) { + executed.add(tool) + val result = dispatchTool(tool, effectiveScreenStateProvider) + if (result !is TrailblazeToolResult.Success) { + return RunTrailblazeToolsResult( + inputTools = tools, + executedTools = executed, + result = result, + ) + } + } + + return RunTrailblazeToolsResult( + inputTools = tools, + executedTools = executed, + result = TrailblazeToolResult.Success(), + ) + } + + private fun dispatchTool( + tool: TrailblazeTool, + screenStateProvider: () -> ScreenState, + ): TrailblazeToolResult { + val toolName = tool.getToolNameFromAnnotation() + Console.log("RevylToolAgent: executing '$toolName'") + + return try { + when (tool) { + is RevylExecutableTool -> { + kotlinx.coroutines.runBlocking { + tool.executeWithRevyl(cliClient, buildMinimalContext(screenStateProvider)) + } + } + is ObjectiveStatusTrailblazeTool -> TrailblazeToolResult.Success() + is MemoryTrailblazeTool -> TrailblazeToolResult.Success() + else -> { + Console.log("RevylToolAgent: unsupported tool ${tool::class.simpleName}") + TrailblazeToolResult.Error.UnknownTrailblazeTool(tool) + } + } + } catch (e: Exception) { + Console.error("RevylToolAgent: '$toolName' failed: ${e.message}") + TrailblazeToolResult.Error.ExceptionThrown( + errorMessage = "Revyl tool '$toolName' failed: ${e.message}", + command = tool, + stackTrace = e.stackTraceToString(), + ) + } + } + + private fun buildMinimalContext( + screenStateProvider: () -> ScreenState, + ): TrailblazeToolExecutionContext { + val devicePlatform = if (platform == RevylCliClient.PLATFORM_IOS) TrailblazeDevicePlatform.IOS else TrailblazeDevicePlatform.ANDROID + val defaultDimensions = RevylDefaults.dimensionsForPlatform(platform) + return TrailblazeToolExecutionContext( + screenState = null, + traceId = null, + trailblazeDeviceInfo = TrailblazeDeviceInfo( + trailblazeDeviceId = TrailblazeDeviceId(instanceId = "revyl", trailblazeDevicePlatform = devicePlatform), + trailblazeDriverType = if (platform == RevylCliClient.PLATFORM_IOS) TrailblazeDriverType.REVYL_IOS else TrailblazeDriverType.REVYL_ANDROID, + widthPixels = defaultDimensions.first, + heightPixels = defaultDimensions.second, + ), + sessionProvider = TrailblazeSessionProvider { + TrailblazeSession(sessionId = SessionId("revyl-tool-agent"), startTime = Clock.System.now()) + }, + screenStateProvider = screenStateProvider, + trailblazeLogger = TrailblazeLogger.createNoOp(), + memory = AgentMemory(), + ) + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylExecutableTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylExecutableTool.kt new file mode 100644 index 00000000..83ce563f --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylExecutableTool.kt @@ -0,0 +1,39 @@ +package xyz.block.trailblaze.revyl.tools + +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.ExecutableTrailblazeTool +import xyz.block.trailblaze.toolcalls.ReasoningTrailblazeTool +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult + +/** + * Base class for tools that execute against a Revyl cloud device via [RevylCliClient]. + * + * Analogous to PlaywrightExecutableTool, but uses natural language targets + * resolved by Revyl's AI grounding instead of element IDs or ARIA descriptors. + * Each tool returns resolved x,y coordinates so consumers like Trailblaze UI + * can render click overlays. + * + * Subclasses implement [executeWithRevyl]; the default [execute] throws to + * direct callers through [RevylToolAgent] which calls [executeWithRevyl] directly. + */ +abstract class RevylExecutableTool : ExecutableTrailblazeTool, ReasoningTrailblazeTool { + + /** + * Executes this tool against the given Revyl CLI client. + * + * @param revylClient The CLI client with an active device session. + * @param context The tool execution context with session, logging, and memory. + * @return The result of tool execution, including coordinates when applicable. + */ + abstract suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult + + override suspend fun execute( + toolExecutionContext: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + error("RevylExecutableTool must be executed via RevylToolAgent") + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeAssertTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeAssertTool.kt new file mode 100644 index 00000000..e071e547 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeAssertTool.kt @@ -0,0 +1,49 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Runs a visual assertion on the Revyl cloud device screen. + * + * Uses Revyl's AI-powered validation step to verify a condition + * described in natural language. + */ +@Serializable +@TrailblazeToolClass("revyl_assert") +@LLMDescription( + "Assert a visual condition on the device screen. Describe what should be true. " + + "Examples: 'the cart total shows \$42.99', 'a success message is visible', " + + "'the Sign In button is disabled', 'there are at least 3 search results', " + + "'the settings screen is open', 'the order confirmation page is shown'.", +) +class RevylNativeAssertTool( + @param:LLMDescription( + "The condition to verify, described in natural language. " + + "Examples: 'the cart badge shows 2 items', 'the profile tab is selected'.", + ) + val assertion: String, + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + Console.log("### Asserting: $assertion") + val result = revylClient.validation(assertion) + return if (result.success) { + TrailblazeToolResult.Success(message = "Assertion passed: '$assertion'") + } else { + TrailblazeToolResult.Error.ExceptionThrown( + errorMessage = "Assertion failed: '$assertion'" + + (result.statusReason?.let { " -- $it" } ?: ""), + ) + } + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeBackTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeBackTool.kt new file mode 100644 index 00000000..2515dec1 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeBackTool.kt @@ -0,0 +1,36 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Presses the device back button on the Revyl cloud device. + * + * On Android this triggers the system back navigation. On iOS the Revyl + * agent uses the app's UI-level back navigation (e.g. navigation bar + * back button) since there is no hardware back key. + */ +@Serializable +@TrailblazeToolClass("revyl_back") +@LLMDescription( + "Press the device back button to go to the previous screen. " + + "On Android, triggers the system back. On iOS, navigates back using the app's UI navigation.", +) +class RevylNativeBackTool( + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + Console.log("### Pressing back") + val result = revylClient.back() + return TrailblazeToolResult.Success(message = "Pressed back at (${result.x}, ${result.y})") + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeDoubleTapTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeDoubleTapTool.kt new file mode 100644 index 00000000..858fbbc5 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeDoubleTapTool.kt @@ -0,0 +1,40 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Double-taps a UI element on the Revyl cloud device using natural language targeting. + * + * Useful for zoom-in gestures on maps, image viewers, and double-tap UI patterns. + * The Revyl CLI resolves the target description to screen coordinates via + * AI-powered visual grounding, then performs the double-tap. + */ +@Serializable +@TrailblazeToolClass("revyl_doubleTap") +@LLMDescription( + "Double-tap a UI element on the device screen. Describe the element in natural language. " + + "Use for zoom-in gestures on maps, image viewers, or any double-tap UI pattern.", +) +class RevylNativeDoubleTapTool( + @param:LLMDescription("Element to double-tap, described in natural language.") + val target: String, + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + Console.log("### Double-tapping: $target") + val result = revylClient.doubleTap(target) + val feedback = "Double-tapped '$target' at (${result.x}, ${result.y})" + Console.log("### $feedback") + return TrailblazeToolResult.Success(message = feedback) + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeNavigateTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeNavigateTool.kt new file mode 100644 index 00000000..5affd3c0 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeNavigateTool.kt @@ -0,0 +1,31 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Opens a URL or deep link on the Revyl cloud device. + */ +@Serializable +@TrailblazeToolClass("revyl_navigate") +@LLMDescription("Open a URL or deep link on the device.") +class RevylNativeNavigateTool( + @param:LLMDescription("The URL or deep link to open.") + val url: String, + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + Console.log("### Navigating to: $url") + revylClient.navigate(url) + return TrailblazeToolResult.Success(message = "Navigated to $url") + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativePressKeyTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativePressKeyTool.kt new file mode 100644 index 00000000..762d3de0 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativePressKeyTool.kt @@ -0,0 +1,39 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Sends a key press event on the Revyl cloud device. + * + * Supports ENTER, BACKSPACE, and ANDROID_BACK. The ANDROID_BACK key + * is routed through [RevylCliClient.back] for correct platform handling. + */ +@Serializable +@TrailblazeToolClass("revyl_pressKey") +@LLMDescription("Press a key on the device keyboard (ENTER, BACKSPACE, or ANDROID_BACK).") +class RevylNativePressKeyTool( + @param:LLMDescription("Key to press: 'ENTER', 'BACKSPACE', or 'ANDROID_BACK'.") + val key: String, + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + Console.log("### Pressing key: $key") + val normalized = key.uppercase() + val result = if (normalized == "ANDROID_BACK") { + revylClient.back() + } else { + revylClient.pressKey(normalized) + } + return TrailblazeToolResult.Success(message = "Pressed $normalized at (${result.x}, ${result.y})") + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeSwipeTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeSwipeTool.kt new file mode 100644 index 00000000..4682bf09 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeSwipeTool.kt @@ -0,0 +1,47 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Swipes on the Revyl cloud device in a given direction. + * + * Optionally starts from a targeted element. Returns the origin coordinates. + */ +@Serializable +@TrailblazeToolClass("revyl_swipe") +@LLMDescription( + "Swipe on the device screen in a direction (up, down, left, right). " + + "Optionally start from a specific element.", +) +class RevylNativeSwipeTool( + @param:LLMDescription("Swipe direction: 'up', 'down', 'left', or 'right'.") + val direction: String, + @param:LLMDescription("Optional element to start the swipe from.") + val target: String = "", + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + require(direction in VALID_DIRECTIONS) { + "Invalid swipe direction '$direction'. Must be one of: $VALID_DIRECTIONS" + } + Console.log("### Swiping $direction") + val result = revylClient.swipe(direction, target.ifBlank { null }) + val feedback = "Swiped $direction from (${result.x}, ${result.y})" + Console.log("### $feedback") + return TrailblazeToolResult.Success(message = feedback) + } + + companion object { + private val VALID_DIRECTIONS = setOf("up", "down", "left", "right") + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeTapTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeTapTool.kt new file mode 100644 index 00000000..3fd09a94 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeTapTool.kt @@ -0,0 +1,46 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Taps (or long-presses) a UI element on the Revyl cloud device using natural + * language targeting. + * + * The Revyl CLI resolves the target description to screen coordinates via + * AI-powered visual grounding, then performs the action. The resolved (x, y) + * coordinates are returned in the success message for overlay rendering. + */ +@Serializable +@TrailblazeToolClass("revyl_tap") +@LLMDescription( + "Tap a UI element on the device screen. Describe the element in natural language " + + "(e.g. 'Sign In button', 'search icon', 'first product card'). " + + "Set longPress to true for a long-press gesture.", +) +class RevylNativeTapTool( + @param:LLMDescription("Element to tap, described in natural language.") + val target: String, + @param:LLMDescription("If true, perform a long-press instead of a tap.") + val longPress: Boolean = false, + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + val action = if (longPress) "Long-pressing" else "Tapping" + Console.log("### $action: $target") + val result = if (longPress) revylClient.longPress(target) else revylClient.tapTarget(target) + val verb = if (longPress) "Long-pressed" else "Tapped" + val feedback = "$verb '$target' at (${result.x}, ${result.y})" + Console.log("### $feedback") + return TrailblazeToolResult.Success(message = feedback) + } +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeToolSet.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeToolSet.kt new file mode 100644 index 00000000..b376d8b4 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeToolSet.kt @@ -0,0 +1,50 @@ +package xyz.block.trailblaze.revyl.tools + +import xyz.block.trailblaze.toolcalls.TrailblazeToolSet.DynamicTrailblazeToolSet +import xyz.block.trailblaze.toolcalls.commands.ObjectiveStatusTrailblazeTool + +/** + * Tool sets for the Revyl cloud device agent. + * + * Provides mobile-native tools that operate against Revyl cloud devices + * using natural language targeting and AI-powered visual grounding. + */ +object RevylNativeToolSet { + + /** Core tools for mobile interaction -- tap, type, swipe, navigate, etc. */ + val RevylCoreToolSet = + DynamicTrailblazeToolSet( + name = "Revyl Native Core", + toolClasses = + setOf( + RevylNativeTapTool::class, + RevylNativeDoubleTapTool::class, + RevylNativeTypeTool::class, + RevylNativeSwipeTool::class, + RevylNativeNavigateTool::class, + RevylNativeBackTool::class, + RevylNativePressKeyTool::class, + ObjectiveStatusTrailblazeTool::class, + ), + ) + + /** Revyl assertion tools for visual verification. */ + val RevylAssertionToolSet = + DynamicTrailblazeToolSet( + name = "Revyl Native Assertions", + toolClasses = + setOf( + RevylNativeAssertTool::class, + ), + ) + + /** Full LLM tool set -- core tools plus assertions and memory tools. */ + val RevylLlmToolSet = + DynamicTrailblazeToolSet( + name = "Revyl Native LLM", + toolClasses = + RevylCoreToolSet.toolClasses + + RevylAssertionToolSet.toolClasses + + xyz.block.trailblaze.toolcalls.TrailblazeToolSet.RememberTrailblazeToolSet.toolClasses, + ) +} diff --git a/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeTypeTool.kt b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeTypeTool.kt new file mode 100644 index 00000000..fd22f552 --- /dev/null +++ b/trailblaze-revyl/src/main/java/xyz/block/trailblaze/revyl/tools/RevylNativeTypeTool.kt @@ -0,0 +1,44 @@ +package xyz.block.trailblaze.revyl.tools + +import ai.koog.agents.core.tools.annotations.LLMDescription +import kotlinx.serialization.Serializable +import xyz.block.trailblaze.revyl.RevylCliClient +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.util.Console + +/** + * Types text into an input field on the Revyl cloud device. + * + * Optionally targets a specific field by natural language description. + * When [clearFirst] is true, the field is cleared before typing. + */ +@Serializable +@TrailblazeToolClass("revyl_type") +@LLMDescription( + "Type text into an input field. Optionally specify a target field " + + "(e.g. 'email field', 'password input').", +) +class RevylNativeTypeTool( + @param:LLMDescription("The text to type into the field.") + val text: String, + @param:LLMDescription("Optional target field, described in natural language.") + val target: String = "", + @param:LLMDescription("If true, clear the field before typing.") + val clearFirst: Boolean = false, + override val reasoning: String? = null, +) : RevylExecutableTool() { + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + val desc = if (target.isNotBlank()) "into '$target'" else "into focused field" + Console.log("### Typing '$text' $desc") + val result = revylClient.typeText(text, target.ifBlank { null }, clearFirst) + val feedback = "Typed '$text' $desc at (${result.x}, ${result.y})" + Console.log("### $feedback") + return TrailblazeToolResult.Success(message = feedback) + } +} diff --git a/trailblaze-revyl/src/test/java/xyz/block/trailblaze/revyl/RevylToolAgentTest.kt b/trailblaze-revyl/src/test/java/xyz/block/trailblaze/revyl/RevylToolAgentTest.kt new file mode 100644 index 00000000..8bdfd62f --- /dev/null +++ b/trailblaze-revyl/src/test/java/xyz/block/trailblaze/revyl/RevylToolAgentTest.kt @@ -0,0 +1,214 @@ +package xyz.block.trailblaze.revyl + +import assertk.assertThat +import assertk.assertions.hasSize +import assertk.assertions.isEqualTo +import assertk.assertions.isInstanceOf +import assertk.assertions.isTrue +import kotlinx.serialization.Serializable +import kotlinx.serialization.Transient +import org.junit.Test +import xyz.block.trailblaze.devices.TrailblazeDevicePlatform +import xyz.block.trailblaze.devices.TrailblazeDriverType +import xyz.block.trailblaze.revyl.tools.RevylExecutableTool +import xyz.block.trailblaze.toolcalls.TrailblazeTool +import xyz.block.trailblaze.toolcalls.TrailblazeToolClass +import xyz.block.trailblaze.toolcalls.TrailblazeToolExecutionContext +import xyz.block.trailblaze.toolcalls.TrailblazeToolResult +import xyz.block.trailblaze.toolcalls.commands.BooleanAssertionTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.ObjectiveStatusTrailblazeTool +import xyz.block.trailblaze.toolcalls.commands.Status +import xyz.block.trailblaze.toolcalls.commands.StringEvaluationTrailblazeTool +import xyz.block.trailblaze.utils.ElementComparator +import kotlin.test.assertContains + +/** + * Unit tests for [RevylToolAgent] dispatch logic, error handling, + * and platform-specific context construction. + * + * Uses a stub [RevylExecutableTool] to avoid needing a real Revyl CLI + * binary or cloud device. The [RevylCliClient] is constructed with + * `/bin/echo` as the binary override so its init-time availability + * check passes without the real CLI installed. + */ +class RevylToolAgentTest { + + private val dummyClient = RevylCliClient("/bin/echo") + + private val noOpComparator = object : ElementComparator { + override fun getElementValue(prompt: String): String? = null + override fun evaluateBoolean(statement: String) = + BooleanAssertionTrailblazeTool(reason = statement, result = true) + override fun evaluateString(query: String) = + StringEvaluationTrailblazeTool(reason = query, result = "") + override fun extractNumberFromString(input: String): Double? = null + } + + // ── Stub tool that records execution and returns a configurable result ── + + @Serializable + @TrailblazeToolClass("stub_revyl_tool") + private class StubRevylTool( + @Transient private val result: TrailblazeToolResult = TrailblazeToolResult.Success(), + @Transient private val onExecute: ((TrailblazeToolExecutionContext) -> Unit)? = null, + ) : RevylExecutableTool() { + @Transient var executed = false + override val reasoning: String? = null + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + executed = true + onExecute?.invoke(context) + return result + } + } + + @TrailblazeToolClass("stub_throwing_tool") + private class ThrowingRevylTool( + private val exception: Exception = RuntimeException("boom"), + ) : RevylExecutableTool() { + override val reasoning: String? = null + + override suspend fun executeWithRevyl( + revylClient: RevylCliClient, + context: TrailblazeToolExecutionContext, + ): TrailblazeToolResult { + throw exception + } + } + + /** A tool type that RevylToolAgent does not recognize. */ + @Serializable + @TrailblazeToolClass("unknown_tool") + private class UnsupportedTool : TrailblazeTool + + // ── Helpers ── + + private fun runTools( + agent: RevylToolAgent, + vararg tools: TrailblazeTool, + ) = agent.runTrailblazeTools( + tools = tools.toList(), + traceId = null, + screenState = null, + elementComparator = noOpComparator, + screenStateProvider = null, + ) + + // ── Tests ── + + @Test + fun `dispatches RevylExecutableTool and returns Success`() { + val agent = RevylToolAgent(dummyClient, "android") + val tool = StubRevylTool() + + val result = runTools(agent, tool) + + assertThat(tool.executed).isTrue() + assertThat(result.result).isInstanceOf(TrailblazeToolResult.Success::class) + assertThat(result.executedTools).hasSize(1) + } + + @Test + fun `returns Success for ObjectiveStatusTrailblazeTool`() { + val agent = RevylToolAgent(dummyClient, "android") + val tool = ObjectiveStatusTrailblazeTool( + explanation = "test complete", + status = Status.COMPLETED, + ) + + val result = runTools(agent, tool) + + assertThat(result.result).isInstanceOf(TrailblazeToolResult.Success::class) + assertThat(result.executedTools).hasSize(1) + } + + @Test + fun `returns UnknownTrailblazeTool for unsupported tool type`() { + val agent = RevylToolAgent(dummyClient, "android") + val tool = UnsupportedTool() + + val result = runTools(agent, tool) + + assertThat(result.result).isInstanceOf(TrailblazeToolResult.Error.UnknownTrailblazeTool::class) + } + + @Test + fun `stops execution on first tool failure`() { + val agent = RevylToolAgent(dummyClient, "android") + val failingTool = StubRevylTool( + result = TrailblazeToolResult.Error.ExceptionThrown( + errorMessage = "simulated failure", + command = StubRevylTool(), + stackTrace = "", + ), + ) + val secondTool = StubRevylTool() + + val result = runTools(agent, failingTool, secondTool) + + assertThat(failingTool.executed).isTrue() + assertThat(secondTool.executed).isEqualTo(false) + assertThat(result.executedTools).hasSize(1) + assertThat(result.result).isInstanceOf(TrailblazeToolResult.Error::class) + } + + @Test + fun `catches exception and returns ExceptionThrown`() { + val agent = RevylToolAgent(dummyClient, "android") + val tool = ThrowingRevylTool(RuntimeException("something broke")) + + val result = runTools(agent, tool) + + assertThat(result.result).isInstanceOf(TrailblazeToolResult.Error.ExceptionThrown::class) + val error = result.result as TrailblazeToolResult.Error.ExceptionThrown + assertContains(error.errorMessage, "something broke") + } + + @Test + fun `builds context with IOS platform when platform is ios`() { + val agent = RevylToolAgent(dummyClient, "ios") + var capturedContext: TrailblazeToolExecutionContext? = null + val tool = StubRevylTool(onExecute = { capturedContext = it }) + + runTools(agent, tool) + + val info = capturedContext!!.trailblazeDeviceInfo + assertThat(info.trailblazeDriverType).isEqualTo(TrailblazeDriverType.REVYL_IOS) + assertThat(info.trailblazeDeviceId.trailblazeDevicePlatform) + .isEqualTo(TrailblazeDevicePlatform.IOS) + } + + @Test + fun `builds context with ANDROID platform when platform is android`() { + val agent = RevylToolAgent(dummyClient, "android") + var capturedContext: TrailblazeToolExecutionContext? = null + val tool = StubRevylTool(onExecute = { capturedContext = it }) + + runTools(agent, tool) + + val info = capturedContext!!.trailblazeDeviceInfo + assertThat(info.trailblazeDriverType).isEqualTo(TrailblazeDriverType.REVYL_ANDROID) + assertThat(info.trailblazeDeviceId.trailblazeDevicePlatform) + .isEqualTo(TrailblazeDevicePlatform.ANDROID) + } + + @Test + fun `executes multiple tools sequentially on success`() { + val agent = RevylToolAgent(dummyClient, "android") + val tool1 = StubRevylTool() + val tool2 = StubRevylTool() + val tool3 = StubRevylTool() + + val result = runTools(agent, tool1, tool2, tool3) + + assertThat(tool1.executed).isTrue() + assertThat(tool2.executed).isTrue() + assertThat(tool3.executed).isTrue() + assertThat(result.executedTools).hasSize(3) + assertThat(result.result).isInstanceOf(TrailblazeToolResult.Success::class) + assertThat(result.inputTools).hasSize(3) + } +} diff --git a/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/ExternalLinkUtils.kt b/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/ExternalLinkUtils.kt new file mode 100644 index 00000000..372353f0 --- /dev/null +++ b/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/ExternalLinkUtils.kt @@ -0,0 +1,34 @@ +package xyz.block.trailblaze.ui.tabs.session + +import xyz.block.trailblaze.devices.TrailblazeDeviceInfo + +/** + * A clickable link derived from device metadata. + * + * @property label Human-readable display text for the link. + * @property url Fully-qualified URL to open. + */ +data class ExternalLink(val label: String, val url: String) + +/** + * Extracts external links from device metadata using the `*_url` key convention. + * + * For each metadata entry whose key ends with `_url` and whose value is non-blank, + * a link is produced. The label is resolved from a sibling `*_label` key if present, + * otherwise auto-generated from the key prefix (e.g. `revyl_viewer_url` becomes + * "Open Revyl viewer"). + * + * @param deviceInfo Device info containing the metadata map. Returns empty list if null. + * @return Ordered list of external links. + */ +fun extractExternalLinks(deviceInfo: TrailblazeDeviceInfo?): List { + val metadata = deviceInfo?.metadata.orEmpty() + return metadata.entries + .filter { it.key.endsWith("_url") && it.value.isNotBlank() } + .map { (key, url) -> + val prefix = key.removeSuffix("_url") + val label = metadata["${prefix}_label"] + ?: "Open ${prefix.replace('_', ' ').replaceFirstChar { c -> c.uppercase() }}" + ExternalLink(label, url) + } +} diff --git a/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/SessionDetailHeader.kt b/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/SessionDetailHeader.kt index 1796fc8d..3093e038 100644 --- a/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/SessionDetailHeader.kt +++ b/trailblaze-ui/src/commonMain/kotlin/xyz/block/trailblaze/ui/tabs/session/SessionDetailHeader.kt @@ -38,6 +38,7 @@ import androidx.compose.runtime.remember import androidx.compose.runtime.setValue import androidx.compose.ui.Alignment import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalUriHandler import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.text.style.TextOverflow import androidx.compose.ui.unit.dp @@ -157,6 +158,22 @@ internal fun SessionDetailHeader( color = MaterialTheme.colorScheme.onSurfaceVariant.copy(alpha = 0.7f), ) } + val externalLinks = extractExternalLinks(sessionDetail.session.trailblazeDeviceInfo) + if (externalLinks.isNotEmpty()) { + val uriHandler = LocalUriHandler.current + for (link in externalLinks) { + TextButton( + onClick = { uriHandler.openUri(link.url) }, + contentPadding = ButtonDefaults.TextButtonContentPadding, + ) { + Text( + text = link.label, + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.primary, + ) + } + } + } } } }