AskUI Agent Setup Guide

Prerequisites

• AskUI Shell installed
• Android Emulator installed
• More than two devices are connected
• Command adb devices shows more than two devices

Setup Steps

1. Open AskUI Shell

   askui-shell

2. Connect Android devices

   • Start two or more Android emulators or connect two or more Android devices via USB
   • Command adb devices must show more than two devices.

3. Configure AskUI Credentials

   1. Create an access token: Access Token Guide
   2. Set up your credentials: Credentials Setup Guide

Act Agent

the file act.py is the entry point for the ACT Agent. To run the ACT Agent, you can use the following command:

Start the ACT Agent

& ./run-act.ps1

Chat Server

To use the AskUI chat functionality, you need to start the chat server locally. chat.py is the entry point for the chat server.

To start the chat server, you can use the following command:

Start the Chat Server

   & ./run-chat.ps1

Chat UI

After starting the chat server, you can navigate to the chat ui hosted at https://hub.askui.com. Login and select the chat tap located at the left sidebar. You can start multiple conversations.

Agents Selection

To switch between Agents, you can click on the drop down

Agents:

• AskUI Vision Agent: The computer vision agent that can be used to interact with the computer device with the default tool set.
• AskUI Web Vision Agent: The web vision agent that uses playwright to interact with a browser.
• AskUI Android Vision Agent: Android agent that can be used to interact with the android device.
  • It can list all connected devices.
  • It can select a device by serial number.
  • It can select a devices's display by name. (Support devices with multiple displays)
  • It can save screenshots to disk. default to ./agent_workspace/
  • It can write text to disk. default to ./agent_workspace/

Example Prompt

You are tasked with executing a cross-device interaction using Android emulators. Follow the steps below precisely:

## Step-by-Step Instructions

1. **Use emulator-5554** to open a browser and navigate to 'https://www.guidgenerator.com'.
2. **Generate a new GUID** from the website.
3. **Retrieve the generated GUID** and store it temporarily.
4. Switch to **emulator-5556**, and send an SMS to the number **8974**.
5. The SMS content must be:  
`"This is the generated GUID from device 5554: <GUID_VALUE>"`  
Replace `<GUID_VALUE>` with the actual GUID you generated.
6. **Take a screenshot after each major interaction**, including:
- GUID generation on emulator-5554  
- SMS composition on emulator-5556  
- Confirmation of SMS sent
7. Once all tasks are complete, **generate a detailed report** named `cross_execution.md`. The report must include:
- Each step performed  
- The actual GUID value used  
- Descriptions of interactions and transitions between devices  
- Embedded screenshots as visual proof (markdown image tags with relative paths)

## Important Notes

- Be sequential and thorough.
- After each interaction, take a screenshot and save and include it in the report.
- Use proper markdown formatting in the report.
- Ensure screenshots are clearly named and referenced correctly.