How to segment one recording into single execution of a task

[HiFunRobo]

Topic

Data Processing Pipeline

Question

Thank you for sharing the data! I downloaded the "flagship Aria fold clothes data", but it seems that each recording consists of multiple task executions. Are there available tools to segment the data into single executions, or, do you have any recommendations on how to do that.
Looking forward to your reply!

What have you tried?

No response

[RyanPCo]

Thanks for trying it out! You're right, in our format, one Aria recording maps to one episode (a single <episode_hash>.zarr), so if you collected multiple folds in one continuous recording, they all live in one long episode. We don't currently ship a tool that automatically detects task-execution boundaries and cuts a recording into separate executions.

A couple of options depending on what you need:

1. If you just want per-execution boundaries for training/conditioning:

Use our span-based annotation format. Each episode's Zarr can hold an annotations array of {"text", "start_idx", "end_idx"} JSON entries (the annotation_v1 format). You can write one span per execution (or per sub-phase) with ZarrWriter.append_annotations:

from egomimic.rldb.zarr.zarr_writer import ZarrWriter

writer = ZarrWriter(episode_path="<episode_hash>.zarr")
writer.append_annotations(
    "annotations",
    [
        ("fold shirt", 0, 1200),
        ("fold shirt", 1200, 2350),
        # one (text, start_frame, end_frame) per execution
    ],
)

These spans are then consumed at training time (e.g. via the annotation-cutoff / action-expert datasets) to keep action chunks within an execution.

2. If you need separate episodes (separate files/rows):

You can slice the Zarr by frame range and re-emit. Read the frame window with ZarrEpisode.read({key: (start, end)}) and write a new store with ZarrWriter, giving each slice its own episode hash (and a corresponding app.episodes row).

Finding the boundaries: the recordings don't carry execution markers, so boundaries have to come from either (a) manual review of the rendered .mp4 we generate alongside each episode, (b) a simple heuristic on hand-tracking / gaze signals (e.g. detecting reset/idle gaps), or (c) using a VLM to label frames.