Thoughts on potential metrics #13
Description
Jotting down some thoughts: Generalizing from Ham2Pose, what are the possible decision points in building a metric, when we think of these as sequences of keypoints for each frame, and we want to do sequence comparisons? Based on thoughts I had while implementing #4
Options for what to do with different body shapes/sizes:
- normalize, typically by shoulders
- Other rotations/rescaling, like Procrustes alignment
Joint selection, maybe we only want to look at certain ones:
- drop legs
- get only the face contours.
Options for what to with sequence length mismatch:
- pad with zeros
- dynamic time warping
- some other sort of interpolation algorithm to try and infer missing points?
Options for dealing with joint mismatch, e.g. missing wrist point
- drop missing points in the hypothesis or reference
- Use a visibility flag, as in OKS: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/13040/3016568/YOLO-based-GNN-for-multi-person-pose-estimation/10.1117/12.3016568.full
Options for comparing two XYZ points
- euclidean distance
- manhattan/cityblock
- All the other options from https://docs.scipy.org/doc/scipy-1.15.0/reference/spatial.distance.html
Other possibilities include:
- don't look at points, look at the shapes they make. https://www.semanticscholar.org/reader/99ec8035b8e0f1cebdd4853ac80338498c88d6bc
- create a 3D shape of the trajectory, maybe with Open3D?
- basically any pose estimation metric could be a candidate: https://doi.org/10.1145/3524497
- algorithms designed for trajectories like Absolute Trajectory Error: https://gereon-t.github.io/trajectopy/Python_Package_Usage/
- more trajectory algorithms: https://www.mathworks.com/help/vision/ref/comparetrajectories.html