Add mapping tester reference results

[fsimonis]

Main changes of this PR

This PR adds a mapping tester compare script and reference results.

The script checks:

• Existing columns
• Equal test cases
• Valid time and memory data (>0) as comparing is difficult across machines
• Approx identical mapping results

Closes #208

Author's checklist

• I used the pre-commit hook and used pre-commit run --all to apply all available hooks.
• I added a test to cover the proposed changes in our test suite.
• I updated the documentation in docs/README.md.
• I added a changelog entry in ./changelog-entries/ (create if necessary).
• I updated potential breaking changes in the tutorial precice/tutorials/aste-turbine.

[davidscn]

That's a great step for more robust testing. Currently, tests are failing.

[fsimonis]

I rewrote the compare file to be actually helpful now.

[fsimonis]

The results are still strange.

When running ctest -R mapping-tester locally with preCICE develop, I get for the NP mapping:

With clang (both) release

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.014321 avg: 0.000882676 var: 2.58172e-06 cnt: 1729
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0137388 avg: 0.000680923 var: 2.41566e-06 cnt: 1729

With gcc (preCICE) release I get

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.014321 avg: 0.000882676 var: 2.58172e-06 cnt: 1729
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0137388 avg: 0.000680923 var: 2.41566e-06 cnt: 1729

With gcc (both) release I get

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.014321 avg: 0.000882676 var: 2.58172e-06 cnt: 1729
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0137388 avg: 0.000680923 var: 2.41566e-06 cnt: 1729

In the CI, we get

Clang debug

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088252 var: 2.69063e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000699396 var: 2.59288e-06 cnt: 1715

Clang release

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088171 var: 2.69091e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000701228 var: 2.61619e-06 cnt: 1715

GCC Debug

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088171 var: 2.69091e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000701228 var: 2.61619e-06 cnt: 1715

GCC Release

[preCICE:B:0] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132921 avg: 0.00088171 var: 2.69091e-06 cnt: 1743
[preCICE:B:1] (mapping::NearestNeighborProjectionMapping in computeMapping) INFO : Mapping distance min:0 max:0.0132319 avg: 0.000701228 var: 2.61619e-06 cnt: 1715

So, there seems to be enough variation in the nearest-projection to result in failing tests.

The big question is why are these projection results different.
Also, why are there still different amount of vertices in the mappings. Is the partitioning still off?

---

Turned out that the Metis partitioning is also non-deterministic.
Even if setting the seed manually, different versions return different results. Given that NP is hypersensitive to the partitioning, we decided to remove it from the parallel tests and use something else instead.

[fsimonis]

The NP mappings are indeed different between versions 1.86 and 1.87 of boost.

[davidscn]

The NP mappings are indeed different between versions 1.86 and 1.87 of boost.

The solution should be unique, or am I missing something? This sounds like a bug either in preCICE or boost geometry. The release notes of boost 1.87 do not mention anything which could sound related.

[fsimonis]

So, the mappings map to different elements, but the effective difference in distance is tiny. I think this happens when a point is located on the edge between two triangles.

Here I print out the polation elements and weights and then compute the projection point for the old and new boost.
It is possible that the order in which the elements for the nearest-projection are considered changed, leading to different elements to be considered first.

Given that our input functions are spacially dependent, this shouldn't lead to such a big difference in the output. Or could the different points impact the result this extremely in the output of ASTE?

97 different mappings

Diff for 43 [0.031755  0.0892496 0.0909088] -> [0.031755  0.0892496 0.0909088] diff 6.938893903907228e-18
Diff for 54 [0.0910679 0.0459595 0.0272727] -> [0.0910679 0.0459595 0.0272727] diff 0.0
Diff for 74 [0.0512694 0.0345959 0.272726 ] -> [0.0512694 0.0345959 0.272726 ] diff 6.938893903907228e-18
Diff for 331 [0.03890349 0.0902521  0.01557049] -> [0.03890349 0.0902521  0.01557049] diff 0.0
Diff for 340 [0.05321172 0.09000761 0.07610572] -> [0.05321172 0.09000761 0.07610572] diff 0.0
Diff for 341 [0.0532904  0.09000816 0.01482465] -> [0.0532904  0.09000816 0.01482465] diff 1.734723475976807e-18
Diff for 357 [0.02618996 0.08602949 0.00877795] -> [0.02618996 0.08602949 0.00877795] diff 0.0
Diff for 358 [0.06350063 0.08637626 0.07989305] -> [0.06350063 0.08637626 0.07989305] diff 1.962615573354719e-17
Diff for 359 [0.06378877 0.0873704  0.01118595] -> [0.06378877 0.0873704  0.01118595] diff 0.0
Diff for 360 [0.00912367 0.07106188 0.03752179] -> [0.00912367 0.07106188 0.03752179] diff 0.0
Diff for 365 [0.08341745 0.07099416 0.01118875] -> [0.08341745 0.07099416 0.01118875] diff 1.3985787785349405e-17
Diff for 383 [0.01993756 0.08184277 0.00871922] -> [0.01993756 0.08184277 0.00871922] diff 3.469446951953614e-18
Diff for 391 [0.0463409  0.09088561 0.02071189] -> [0.0463409  0.09088561 0.02071189] diff 0.0
Diff for 405 [0.07786282 0.07664384 0.00748544] -> [0.07786282 0.07664384 0.00748544] diff 0.0
Diff for 406 [0.00172516 0.05427121 0.00847953] -> [0.00172516 0.05427121 0.00847953] diff 1.734723475976807e-18
Diff for 409 [0.08979725 0.05430727 0.00847512] -> [0.08979725 0.05430727 0.00847512] diff 0.0
Diff for 415 [0.08683562 0.06205872 0.02275029] -> [0.08683562 0.06205872 0.02275029] diff 1.4304896245381992e-17
Diff for 419 [0.07130562 0.08264604 0.07449454] -> [0.07130562 0.08264604 0.07449454] diff 1.3877787807814457e-17
Diff for 420 [0.07107596 0.08264124 0.01653888] -> [0.07107596 0.08264124 0.01653888] diff 0.0
Diff for 423 [0.02314756 0.08389745 0.01666558] -> [0.02314756 0.08389745 0.01666558] diff 4.9065389333867974e-18
Diff for 440 [0.0584609  0.08732944 0.07103828] -> [0.0584609  0.08732944 0.07103828] diff 1.3877787807814457e-17
Diff for 444 [0.08691084 0.06366636 0.00781267] -> [0.08691084 0.06366636 0.00781267] diff 1.97026712507607e-17
Diff for 567 [0.01831766 0.04207775 0.86050046] -> [0.01831766 0.04207775 0.86050046] diff 0.0
Diff for 623 [0.00988937 0.03617199 0.28819549] -> [0.00988937 0.03617199 0.28819549] diff 0.0
Diff for 773 [0.00671197 0.04212451 0.5846063 ] -> [0.00671197 0.04212451 0.5846063 ] diff 0.0
Diff for 853 [0.0589451  0.03582975 0.43449593] -> [0.0589451  0.03582975 0.43449593] diff 0.0
Diff for 1260 [0.00614159 0.04420755 0.73325251] -> [0.00614159 0.04420755 0.73325251] diff 0.0
Diff for 1590 [0.0148691  0.04964951 0.41585379] -> [0.0148691  0.04964951 0.41585379] diff 6.938893903907228e-18
Diff for 1714 [0.0116469  0.04728232 0.79225595] -> [0.0116469  0.04728232 0.79225595] diff 0.0
Diff for 1753 [0.00753731 0.04997131 0.58459527] -> [0.00753731 0.04997131 0.58459527] diff 0.0
Diff for 1760 [0.00611012 0.04863093 0.44757775] -> [0.00611012 0.04863093 0.44757775] diff 0.0
Diff for 1826 [0.02168015 0.05346873 0.35561084] -> [0.02168015 0.05346873 0.35561084] diff 0.0
Diff for 1897 [0.00747925 0.04780511 0.67721435] -> [0.00747925 0.04780511 0.67721435] diff 0.0
Diff for 1951 [0.00548251 0.04833122 0.41518463] -> [0.00548251 0.04833122 0.41518463] diff 8.673617379884035e-19
Diff for 2165 [0.02437463 0.0504875  0.41859023] -> [0.02437463 0.0504875  0.41859023] diff 0.0
Diff for 2241 [0.00656444 0.04773148 0.73337781] -> [0.00656444 0.04773148 0.73337781] diff 0.0
Diff for 2298 [0.0043105  0.04778281 0.50679067] -> [0.0043105  0.04778281 0.50679067] diff 8.673617379884035e-19
Diff for 2424 [0.02789976 0.06928756 0.15753992] -> [0.02789976 0.06928756 0.15753992] diff 3.122502256758253e-17
Diff for 2455 [0.0099326  0.06322157 0.14285575] -> [0.0099326  0.06322157 0.14285575] diff 1.734723475976807e-18
Diff for 2460 [0.06792024 0.05329357 0.25234925] -> [0.06792024 0.05329357 0.25234925] diff 0.0
Diff for 2478 [0.04280791 0.08663861 0.10738089] -> [0.04280791 0.08663861 0.10738089] diff 0.0
Diff for 2515 [0.14210568 0.04705542 0.1785666 ] -> [0.14210568 0.04705542 0.1785666 ] diff 0.0
Diff for 2522 [0.02131924 0.07008853 0.14618993] -> [0.02131924 0.07008853 0.14618993] diff 3.1031676915590914e-17
Diff for 2546 [0.0922402  0.04834609 0.26126209] -> [0.0922402  0.04834609 0.26126209] diff 1.5515838457795457e-17
Diff for 2604 [0.13817501 0.04109845 0.26333141] -> [0.13817501 0.04109845 0.26333141] diff 0.0
Diff for 2618 [0.02061306 0.07121113 0.13596627] -> [0.02061306 0.07121113 0.13596627] diff 0.0
Diff for 2648 [0.11134667 0.04712509 0.25437413] -> [0.11134667 0.04712509 0.25437413] diff 0.0
Diff for 2667 [0.02919213 0.07456837 0.14235567] -> [0.02919213 0.07456837 0.14235567] diff 0.0
Diff for 2673 [0.00390707 0.05121262 0.19047989] -> [0.00390707 0.05121262 0.19047989] diff 0.0
Diff for 2681 [0.08225485 0.07050358 0.11610616] -> [0.08225485 0.07050358 0.11610616] diff 1.962615573354719e-17
Diff for 2704 [0.02819699 0.07165435 0.15035807] -> [0.02819699 0.07165435 0.15035807] diff 0.0
Diff for 2706 [0.04762203 0.07250641 0.16250567] -> [0.04762203 0.07250641 0.16250567] diff 0.0
Diff for 2725 [0.02080786 0.06706447 0.15519056] -> [0.02080786 0.06706447 0.15519056] diff 0.0
Diff for 2739 [0.00342845 0.05127787 0.17228183] -> [0.00342845 0.05127787 0.17228183] diff 2.7755575615628914e-17
Diff for 2764 [0.11696258 0.04694698 0.15010603] -> [0.11696258 0.04694698 0.15010603] diff 6.938893903907228e-18
Diff for 2776 [0.03372152 0.07182028 0.15720216] -> [0.03372152 0.07182028 0.15720216] diff 2.8609792490763984e-17
Diff for 2777 [0.03253129 0.07940229 0.12751047] -> [0.03253129 0.07940229 0.12751047] diff 3.179800655392251e-17
Diff for 2870 [0.10657692 0.02933997 0.15113578] -> [0.10657692 0.02933997 0.15113578] diff 0.0
Diff for 2888 [0.06086046 0.0191452  0.18185132] -> [0.06086046 0.0191452  0.18185132] diff 2.7755575615628914e-17
Diff for 2899 [0.14752535 0.03398834 0.18289833] -> [0.14752535 0.03398834 0.18289833] diff 2.7755575615628914e-17
Diff for 2900 [0.05343157 0.03284744 0.25893433] -> [0.05343157 0.03284744 0.25893433] diff 0.0
Diff for 2930 [0.09212792 0.03524083 0.11615036] -> [0.09212792 0.03524083 0.11615036] diff 6.938893903907228e-18
Diff for 2938 [0.00547898 0.04060849 0.22066274] -> [0.00547898 0.04060849 0.22066274] diff 0.0
Diff for 2942 [0.06868673 0.03298752 0.26151751] -> [0.06868673 0.03298752 0.26151751] diff 1.3877787807814457e-17
Diff for 2991 [0.10587044 0.02380839 0.15849539] -> [0.10587044 0.02380839 0.15849539] diff 3.469446951953614e-18
Diff for 3003 [0.00264032 0.03751523 0.1117432 ] -> [0.00264032 0.03751523 0.1117432 ] diff 1.3877787807814457e-17
Diff for 3011 [0.13837023 0.03729602 0.17047996] -> [0.13837023 0.03729602 0.17047996] diff 2.8609792490763984e-17
Diff for 3014 [0.08961514 0.03844787 0.09909619] -> [0.08961514 0.03844787 0.09909619] diff 2.0816681711721685e-17
Diff for 3019 [0.06818599 0.01939514 0.18490653] -> [0.06818599 0.01939514 0.18490653] diff 0.0
Diff for 3028 [0.06066809 0.02128966 0.19048013] -> [0.06066809 0.02128966 0.19048013] diff 6.938893903907228e-18
Diff for 3041 [0.05706328 0.03357908 0.2646014 ] -> [0.05706328 0.03357908 0.2646014 ] diff 0.0
Diff for 3059 [0.05130423 0.01880416 0.17701843] -> [0.05130423 0.01880416 0.17701843] diff 3.469446951953614e-18
Diff for 3069 [0.07140464 0.03188875 0.25384761] -> [0.07140464 0.03188875 0.25384761] diff 0.0
Diff for 3083 [0.07462764 0.0123213  0.13290129] -> [0.07462764 0.0123213  0.13290129] diff 0.0
Diff for 3102 [0.14636833 0.03747274 0.17517573] -> [0.14636833 0.03747274 0.17517573] diff 0.0
Diff for 3107 [0.10217736 0.02724643 0.14549859] -> [0.10217736 0.02724643 0.14549859] diff 0.0
Diff for 3110 [0.1128545  0.02856487 0.15885248] -> [0.1128545  0.02856487 0.15885248] diff 3.469446951953614e-18
Diff for 3133 [0.09154206 0.03900084 0.10636968] -> [0.09154206 0.03900084 0.10636968] diff 1.3877787807814457e-17
Diff for 3176 [0.08549466 0.02016327 0.12577194] -> [0.08549466 0.02016327 0.12577194] diff 3.122502256758253e-17
Diff for 3199 [0.05964699 0.03253331 0.25730115] -> [0.05964699 0.03253331 0.25730115] diff 0.0
Diff for 3227 [0.05229692 0.0017486  0.01556087] -> [0.05229692 0.0017486  0.01556087] diff 2.168404344971009e-19
Diff for 3230 [0.00276889 0.03210614 0.0149353 ] -> [0.00276889 0.03210614 0.0149353 ] diff 7.152448122690996e-18
Diff for 3232 [0.08845822 0.03210614 0.0149353 ] -> [0.08845822 0.03210614 0.0149353 ] diff 6.938893903907228e-18
Diff for 3233 [0.07684784 0.01360584 0.01551274] -> [0.07684784 0.01360584 0.01551274] diff 0.0
Diff for 3235 [0.01437927 0.01360584 0.01551274] -> [0.01437927 0.01360584 0.01551274] diff 0.0
Diff for 3238 [0.0380434  0.00199351 0.01477338] -> [0.0380434  0.00199351 0.01477338] diff 7.165583942981724e-18
Diff for 3253 [0.06507855 0.00581193 0.0087868 ] -> [0.06507855 0.00581193 0.0087868 ] diff 0.0
Diff for 3256 [0.02772154 0.00547702 0.01103672] -> [0.02772154 0.00547702 0.01103672] diff 3.576224061345498e-18
Diff for 3260 [0.08276953 0.02140194 0.01110925] -> [0.08276953 0.02140194 0.01110925] diff 1.3877787807814457e-17
Diff for 3261 [0.00845757 0.02140195 0.01110925] -> [0.00845757 0.02140195 0.01110925] diff 3.469446951953614e-18
Diff for 3279 [0.07130201 0.01005897 0.00871969] -> [0.07130201 0.01005897 0.00871969] diff 0.0
Diff for 3287 [0.0448815  0.00100162 0.02070964] -> [0.0448815  0.00100162 0.02070964] diff 0.0
Diff for 3298 [0.08903192 0.03881409 0.05091484] -> [0.08903192 0.03881409 0.05091484] diff 0.0
Diff for 3303 [0.08950909 0.03768465 0.00847584] -> [0.08950909 0.03768465 0.00847584] diff 0.0
Diff for 3305 [0.00171801 0.03768464 0.00847584] -> [0.00171801 0.03768464 0.00847584] diff 6.938893903907228e-18
Diff for 3307 [0.05537713 0.00205687 0.00817447] -> [0.05537713 0.00205687 0.00817447] diff 0.0
Diff for 3371 [0.05090186 0.00199714 0.02223512] -> [0.05090186 0.00199714 0.02223512] diff 0.0

[fsimonis]

I was able to trace this down to the NP mapping itself projecting to worse elements on macOS-latest.
So, I disabled a test for now on macOS, opened a bug report in the preCICE repository precice/precice#2185, and a PR to add the test again #223,

[fsimonis]

Merging now and rebasing the WIP RPs