dataset generation

[leyuan-sun]

Could you plz explain more about how to generation dataset? As you mentioned that, "We follow RLBench’s data generation pipeline for consistency and scalability", but I cannot find procedures in the RLBench repo. Thanks!

[leyuan-sun]

I still confused about the training dataset, I find out this https://huggingface.co/datasets/hqfang/rlbench-18-tasks, could I directly use it for train the flowram??

[SanMumumu]

Thank you for your interest and question!

For details on how to generate the dataset, you can refer to the documentation here: DATA_PREPARATION_RLBENCH.md, which provides step-by-step guidance on following the RLBench pipeline.

In addition, you may also check out 3d_flowmatch_actor, where you can find a more standardized and clear implementation that could further help you understand the data preparation and training process.

Hope this helps!

[leyuan-sun]

@SanMumumu Thanks for your reply, I just directly download the training epoisodes: "Download our packaged demonstrations for training from here." then put into $YOUR_REPO_PATH/FlowRAM/data/, is it ok?

[leyuan-sun]

@SanMumumu Sry, I need help again. I have generated high resolutions 256*256 training dataset to train with precise_train.sh, but I got the error in mamba-ssm about tensor shape, i also change the imag_size in the main_trajetory.py, I have no idea right now, plz check the error:

[rank0]: Traceback (most recent call last): [rank0]: File "/home/sly/FlowRAM/main_trajectory.py", line 429, in <module> [rank0]: train_tester.main(collate_fn=traj_collate_fn) [rank0]: File "/home/sly/FlowRAM/engine.py", line 155, in main [rank0]: self.train_one_step(model, criterion, optimizer, step_id, sample) [rank0]: File "/home/sly/FlowRAM/main_trajectory.py", line 210, in train_one_step [rank0]: out, trans_loss, rot_loss, open_loss = model( [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1648, in forward [rank0]: else self._run_ddp_forward(*inputs, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1474, in _run_ddp_forward [rank0]: return self.module(*inputs, **kwargs) # type: ignore[index] [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/sly/FlowRAM/diffuser_actor/trajectory_optimization/diffuser_actor.py", line 418, in forward [rank0]: fixed_inputs = self.encode_inputs( [rank0]: File "/home/sly/FlowRAM/diffuser_actor/trajectory_optimization/diffuser_actor.py", line 98, in encode_inputs [rank0]: context_feats = self.encoder.vision_language_attention( [rank0]: File "/home/sly/FlowRAM/diffuser_actor/utils/encoder.py", line 285, in vision_language_attention [rank0]: feats, _ = self.vl_attention[0]( [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/sly/FlowRAM/diffuser_actor/utils/layers.py", line 170, in forward [rank0]: seq1_, seq2_ = layer( [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/sly/FlowRAM/diffuser_actor/utils/layers.py", line 107, in forward [rank0]: seq1b = self.cross_12(seq1, inference_params=None) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/home/sly/anaconda3/envs/flowram/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/home/sly/FlowRAM/mamba-1.1.1/mamba_ssm/modules/mamba_simple.py", line 124, in forward [rank0]: batch, seqlen, dim = hidden_states.shape [rank0]: ValueError: not enough values to unpack (expected 3, got 2)

[leyuan-sun]

I suggest you to upload one task of training dataset or check the provided code, since I made some modifications of your code to run at the above progress. Thanks!!