Unable to extract object pose

[jorou125]

Hi, I'm having difficulty extracting the object pose annotation. When I follow the documentation and run annotation_bbox_pose.py I get pose annotation which do not match with the object's position. Here is an example with object MR6D11 in scene 6.

I downloaded the dataset on the Hugging Face link provided on this repo. I ran the data preprocessing steps required before launching the annotation script. Is it possible to guide me into extracting the correct pose of objects. Thank you in advance.

[jorou125]

I did further experiment and have found that the extraction of pose is problematic on my end. I have tried on multiple scenes and here is what I found:

1. When downloading the dataset, the annotation of object pose can be found in a file such as scene6\annotation\rgb_MR6D11__pose.json for object MR6D11 in scene 6. I assumed these were the ground truth poses, but when I project the model on the image, it does not align with the object properly. I then used the appropriate version of the script for the scene (documentation from dataset mentions to use a certain commit for scenes captured at 25fps), but the results were still not satisfactory. Finally, I tried with the latest version of the annotation script, but it produced the results I have previously shown. These figures show the results for scene6, red being the annotations downloaded, green the annotations with the 25 fps script, and blue the annotations with the 100 fps script:
2. I tried with scenes at 100 fps with the latest version of annotation scripts. During the annotation, you use pyrender to project the object on the image, but the image shown shows no object (for example, in scene 75 with object 13). It seems that the pose information is not in the correct referential, as the translation according to z axis is positive while pyrender seems to use the negative z axis for objects in front of camera. I tried converting the pose annotation to be able to project the object, but there still exist a certain offset to the object on the image. Is it within the expected margin for your dataset's accuracy or is it linked to the rendering problems?

Thank you in advance for your support, MTevent seems to be a great dataset and I hope to be able to use it to its full extent!

[shrutarv]

Hi there,
Sorry for the delayed response. I am out of office till 30 Nov.
Answer 1: The annotations using the 25 fps script should be used for scene1 - scene25. Actually, the GT annotations shown in red should be the same as those you get when executing the 25 fps script (green box). There looks to be a significant difference between the red and green boxes. Maybe, it is better if you execute the 25 fps script and save the annotations and use those annotations. The 100 fps script is only for scenes 26-100 and should not be used in this case.

Answer 2: This offset is within the permissible limit. The reason being: as the distance of the object from the camera increases the error will increase if the baseline between the cameras is small. Here the distance varies from 1- 10m, which is challenging. So you could go ahead with these annotations. But, I will still take a look at the rendering code.

Hope, I was able to help you. Please let me know if you face any issues.

[jorou125]

Hi,

Thanks again for your support and for providing this dataset.

I've continued experimenting with MTevent and created some visualizations to better understand the pose annotations. Specifically, I tested scenes 6 and 75 using the corresponding versions of the annotation script. While reviewing the projected poses on the RGB frames, I noticed that in several instances, the alignment between the projected object poses and the actual object appearances seems to be off by a noticeable margin.

Here are a couple of examples that illustrate the issue:

Scene 6, Object MR6D11, and 25 fps
https://github.com/user-attachments/assets/c1ccd4f3-feda-406e-8ec3-d156522404a6

Scene 75, Object MR6D13, and 100 fps
https://github.com/user-attachments/assets/fe1f27d1-cfa4-4145-8813-4c5e536f88fa

I understand that some level of projection error can be expected, but in these cases, the discrepancy seems a bit larger than anticipated. I just wanted to check whether this might be related to the annotation script, or if there are known limitations or calibration nuances I should be aware of.

Thanks again for your work on this dataset. Looking forward to your insights!

[jorou125]

Hi @shrutarv,

I hope you are doing well. I was wondering if you have had the time to look at the videos I've shared previously. I would like to have your opinion on the matter. Furthermore, I would also be interested to know if you've had time to look at the rendering code, as you mentioned before.

Thank you in advance for your help and time!

[shrutarv]

Hi @jorou125 ,

Sorry for the delayed response. The videos you shared are for rgb cameras. Those videos typically show the problem of using a rgb camera to annotate fast or jerky motion. The annotations for RGB camera are delayed and thus inaccurate. However, the event camera annotations for the same scene are much better and within the usable limit. The main idea behind this dataset is to create data which can be used for development of networks tailored for event camera data.

The rgb annotations and videos are given just to compare the event and the rgb cameras. These annotations would be less accurate than event camera annotations due to the fast and jerky motions performed during the data collection.