AnySense is an open-source iPhone app that enables multi-sensory data collection by integrating the iPhone's sensory suite with external sensors via Bluetooth and wired interfaces, enabling both offline data collection and online streaming to robots. The app also supports on-device policy evaluation: you can load a trained CoreML policy, set a 3D goal in AR, and run the policy with AR visualization of predicted actions without a robot. We record RGB and depth videos, metric depth frames, streamed Bluetooth data (e.g. from tactile sensors) into a binary file, and timestamped pose data as a .txt file. Example streaming code for Bluetooth data can be found on AnySkin. USB streaming is supported by connecting the iPhone to your computer and using this accompanying library (forked from record3d).
AnySense data storage format:
- Streamed Bluetooth Data (.bin)
- RGB video (.mp4)
- Depth video (.mp4)
- Metric depth frames (*.bin)
- Pose Data (.txt)
-
Clone the github repository:
git clone https://github.com/NYU-robot-learning/AnySense.git -
Install XCode and open the repository in XCode.
-
In the app settings, navigate to the "Signing & Capabilities" section. Make sure the "Automatically manage signing" checkbox is checked. Add your account and set a unique bundle identifier.
-
Plug in your IOS device to your Mac, and follow instructions for trusting the computer, enabling Developer mode. You should now be able to build the app.
Questions about AnySense? Contact: raunaqbhirangi@nyu.edu
The team at Generalizable Robotics and AI Lab, NYU: Raunaq Bhirangi, Zeyu (Michael) Bian, Venkatesh Pattabiraman, Haritheja Etukuru, Mehmet Enes Erciyes, Krish Mehta, Nur Muhammad Mahi Shafiullah, Lerrel Pinto