Add accessibility API to interact with other apps

[Benjamin-Loison]

Feature description

Android API gives the ability to retrieve current structured UI content (where are texts and what are they, where are buttons...) and interact (click, type characters...) with the screen content.

Personal motivation related to Internet-less Wi-Fi required wireless debugging workaround:

This feature would unlock many use cases, notably my open-source voice assistant Benjamin_Loison/Voice_assistant which currently leverages wireless debugging to use adb exec-out uiautomator dump. The issue with wireless debugging is that, despite not using Internet, it requires a Wi-Fi for Termux to connect to the device wireless debugging itself, I planned modifying AOSP to remove this limitation, but I suspect that it is a feature, not a bug in Google's mind and they would not be open to somehow allow proceeding without Wi-Fi. Restricting such possibility to custom ROMs or now on Android versions looks too restrictive for people to enjoy it now.

Personal notes: Benjamin-Loison/android/issues/28

Reference implementation

Have you checked if the feature is accessible through the Android API? Yes, see:

https://developer.android.com/reference/android/view/accessibility/package-summary

Do you know of other open-source apps that has a similar feature as the one you want? (Provide links)

Closed-source apps:

Tasker and AutoInput are closed-source, but leverage this accessibility API to notably retrieve structured screen content. Automate is able to dump the structured screen content as XML.
I just achieved clicking with https://github.com/Nain57/Smart-AutoClicker, but not retrieving structured screen content.

RustDesk is open-source and let the ability through accessibility to click.

I plan to propose a pull request adding commands to leverage accessibility Android API, as my voice assistant is Termux based.

[Chrysaloid]

I am interested in this feature too and would like it to be available natively in Termux.

But for now I have came up with this quite good workaround that allows Termux to have capabilities of any Automate blocks but especially Inspect layout and Interact:

Communicate with Termux.zip (version for Automate free edition below)

That zip contains 2 files: Communicate with Termux.flo and server-script.js. You should import the flow file to Automate, inspect it (as you should do with every peace of code you download from the Internet), grant necessary permissions and start it. Then you should put the JS file in some folder in Termux and navigate there. You should also setup NodeJS if you don't have it. Then you can run the script with node server-script.js.

How it works:

• The flow sets up a HTTP server on http://localhost:3090/ that only accepts POST requests with JSON body and waits.
• When you run the script it:
  • sets up a HTTP server on http://localhost:3091/
  • sends a POST request to the flow's server with a JSON body of form {name: String, data: Any} and awaits the response.
• The flow immediately responds with nothing.
• The script starts awaiting a request to its server.
• The flow parses the body and extracts variables name and data:
  • name is the name of a supported event. It corresponds to the name of the block inside Automate that you want to run
  • data has different forms depending on the event and contains the parameters that will be passed to the block you want to run
• The flow runs the block and if error occurred it returns JSON of form {error: String}. If no error occurred it returns JSON of form {data: Any}.
• The JSON is then sent to the script's server.
• The script's server responds immediately with nothing and starts parsing the request only when the response has been fully sent (it is to give Automate time to terminate the fiber).
• The parsed data is then returned to the function that sent the initial POST request to the flow's server.

That way a block inside Automate'a flow becomes a line of code i.e. await toast("Hello world!");

The script and the flow can be easily extended to support calls to other Automate blocks but most of the functionalities that they provide are already available in Termux so I implemented only the necessary ones.

The script can of course be implemented in any other language that supports creating a HTTP server. I chose JS because packing the request via fetch and response via http.createServer's callback was easy thanks to the Promise class.

That creation of HTTP server on Termux is unfortunately necessary because of Automate's rules of HTTP accept and HTTP response blocks that state "the response to the request must be sent within 5 seconds". This makes it impossible to create a simpler flow that i.e. responds only when some UI element appears on the screen using the When UI layout changed proceed value of the Inspect layout block. That's why I had to use HTTP Request block to pass information back to Termux.

This approach is generally pretty fast on my Samsung S24 Ultra but it may have too much overhead for slower phones and the battery usage may be high. If you can achieve what you want with 2-3 Interact/Inspect blocks I recommend creating dedicated flow. This workaround gets better than dedicated flow the more interactions you have to perform to achieve your automation because inside a script you can see all the XPaths at once and you can add comments and changing text on a computer is much easier then moving and clicking blocks on a mobile phone. This workaround also lets you kinda bypass the Automate's free edition 30 running blocks limit.

Speaking of free edition... I have realized just now that I forgot that I was using Automate premium this whole time and I can see that the above flow uses more than 30 blocks and will not work for free edition users. That's why I created a free edition version:

Communicate with Termux free.zip

I removed utility functions used testing, removed some validation and compressed variable set blocks into one destructuring assignment but otherwise it is almost the same. It has 30 blocks so unfortunately when the flow is running no other flows can be started.

If more people would be interested, I can make it a repo where this workaround can be implemented in other languages.

License: MIT

[Benjamin-Loison]

Interesting, thanks for sharing, but it seems too heavy and closed-source such that I use it as a workaround.

[Benjamin-Loison]

@Chrysaloid enjoy #837!