Implement DTW-MJE and nDTW-MJE (from Ham2Pose)

[cleong110]

Zifan said to go ahead and move his code over here, just making an issue to track progress.

[cleong110]

Possibly something for Future Work, but fastdtw could potentially be replaced with https://pypi.org/project/dtw-python/, which seems more up to date.

Also, dtw-python does, apparently include within it the capability to implement the FastDTW algorithm, the paper/documentation for dtw-python cites FastDTW

Having the ability to graph/visualize alignments would also be helpful.

[cleong110]

As I'm working on this, I note that Ham2Pose metrics had a number of preprocessing steps they did, and am considering what if anything needs updating.

• Normalize: they use pose_format.utils.generic.pose_normalization_info, is that still correct? pose-format readme suggests using normalize_distribution(). See pose.normalize() - would it be better for PoseNormalizationInfo to be optional? pose#140
• Focus: no changes needed I think.
• Hide the legs - presumably I should switch to the pose-format implementation.
• Hide low conf - looks fine.
• Trim leading/trailing frames with only zeros, almost no face, or no hands: I don't really know what all these specific indices/numbers are, or if this implementation will still work.

[AmitMY]

• pose_normalization_info is correct. it normalizes by the shoulders.
• no need to focus imo
• ideally yes, we have a utility
• hide low conf - i'll be against, especially in the reference (one could do this preprocessing by themselves, while this library is generic)
• in that implementation, this was because the hands are out-of-frame in the beginning or end, and cropping them was easier to experiment with. i think it is unnecessary (especially for ML outputs)

[J22Melody]

My current usage at https://github.com/J22Melody/iict-eval-private/blob/text2pose/metrics/metrics.py#L94

I believe some questions can be made a function parameter/option, we just need to decide the default behavior.

1. I currently normalize the poses before calling compare_poses, but I think it would be useful to have it inside the function, default yes
2. one more thing I do is to reduce the pose2 (reference) to the set of keypoints of pose1 (hypothesis), assuming pose2 has full keypoints but pose1, a model output might only has subset. It is useful, but maybe default no, and raise an error if the two poses do not match.
3. no need for focus
4. hide the legs, yes, better be an option with default yes
5. hide low conf, I am using it to maximally mimic ham2pose, but maybe better an option with default no
6. should be unnecessary, and the current implementation is tailored for openpose

[J22Melody]

Re:dtw-python, I am not familiar, but maybe the DTW implementation can be chosen as an option as well. @cleong110 what are the benefits there you can see? Need some benchmarking I guess? (future work)

[J22Melody]

Having the ability to graph/visualize alignments would also be helpful.

This is very nice but maybe beyond the purpose of the metrics utility.

It reminds me one possible limitation of the current Ham2Pose DTW is that it calculates point-wise DTW over time separately and then do a weighted average, which now I just set to all 1 for every mediapipe keypoint. This is for sure not optimal especially when you are comparing two poses with dense face keypoints. Perhaps it is better to get a frame-wise distance for the whole pose and then do DTW, then also if there is a visualization of the alignment over time, it would be super informative.

[cleong110]

Re:dtw-python, I am not familiar, but maybe the DTW implementation can be chosen as an option as well. @cleong110 what are the benefits there you can see? Need some benchmarking I guess? (future work)

Re: DTW-Python, I think the main things for me are (1) that it is being actively maintained, where fastDTW seems to be accumulating issues over the years and (2) dtw-python seems to be pretty full-featured, letting you play around in detail with components of the DTW process like windowing, step patterns, and so forth.

I'm not really sure about performance, it would need benchmarking indeed.

But yeah it's not a high priority, future work for sure.

[cleong110]

Regarding pose_hide_legs, do we want to drop the POSE_WORLD_LANDMARKS as well?

[J22Melody]

Regarding pose_hide_legs, do we want to drop the POSE_WORLD_LANDMARKS as well?

sounds fine, never used it.

[cleong110]

Could use a bit of advice on implementation details: thoughts here: #4 (comment)

[cleong110]

Closed as of #19. The Example script there has DTW-MJE