calculate failure rate using total number of test cases

[schoekek]

The test test_fit_single_parameter_with_noise_two_percent() is intended to verify that no more than 5% of model fits fail when 2% random noise is added to the synthetic data.

The previous implementation calculated the failure rate as:

len(faillist) / len(succlist)

This uses only the number of successful fits as the denominator, which makes the threshold slightly stricter than intended. For example, 5 failures and 95 successes result in a value of 5/95 = 5.26%, even though the actual failure rate is exactly 5% (5 out of 100 total runs).

This patch changes the calculation to:

len(faillist) / (len(succlist) + len(faillist))

Using the total number of test cases as the denominator correctly reflects the proportion of failed fits and matches the stated 5% acceptance criterion.

This also makes the test less sensitive to small architecture-dependent numerical differences in the fitting routines, which can otherwise lead to spurious failures on platforms such as ppc64el.

[paulmueller]

Thanks for the PR!

Is this an issue you ran into as a debian maintainer?

I believe the main reason the test fails spuriously is because the noise is not deterministic. Since you already went through the effort of writing a very thoughtout PR, could I encourage you to also change the current use of np.random.random in ...

PyCorrFit/tests/test_fit_models.py

Line 33 in c91f274

anoise = (np.random.random(data.shape[0])-.5)*deltanoise

... such that before each call the random seed is set according to https://numpy.org/doc/stable/reference/random/generator.html#numpy.random.Generator ?

It should look similar to this

import numpy as np


def fit_single_parameter(...)
    [...]
    rng = np.random.default_rng(seed=42)
    anoise = (rng.random(data.shape[0])-.5)*deltanoise

[schoekek]

Yes, this is an issue with the tests within the Debian build/test environment.
I am not a programmer, but my patch at least allows the tests to pass for now.
If you have a better solution and would like to implement it, please feel free to do so; I can then remove my patch from Debian again during the next update.