WHL: enable Limited API-compliant (abi3) wheels

[neutrinoceros]

I initially just wanted to experiment with this and see how far from a working state we were... turns out the existing code is pretty close already, so only a couple tweaks were needed.

PyTuple_GET_* functions are normally faster, as they skip certain checks that equivalent Limited API functions will run, so there could be a case to be made that we should still use them by default, but I'm not convinced that the impact is even measurable (I'll run the benchmarks to double check before I undraft though).

[neutrinoceros]

This appears to work as expected in CI.
I do seem to get slightly worse performance on some benchmarks though. It still seems worth it to me but maybe I can make the default build use version-specific APIs, and only enable abi3 as an opt in (enabled in ci)

[rgommers]

make the default build use version-specific APIs, and only enable abi3 as an opt in (enabled in ci)

That's the way I'm setting it up for PyWavelets too, and is generally the right strategy I believe. For distros for example, they always only compile for a single Python version, so there are no upsides to using the Stable ABI. Same for end users building from source. Wheel distribution is kinda the odd exception.

[rgommers]

Can you post the benchmark results?

[neutrinoceros]

Ah, there are already lost to my terminal history unfortunately, but making abi3 an opt-in was trivial enough

[neutrinoceros]

this is needed for malloc and free. I suppose we got away without this include so far because Python.h must include it already, but only conditionally ?

[neutrinoceros]

opening for review now. The last commit is temporary and intended to be trimmed before merge, but it helps seeing relevant CI in action

[neutrinoceros]

I reran the benchmarks (this time on linux + x86_64)

master:

Bottleneck performance benchmark
    Bottleneck 1.6.0.post0.dev38; Numpy 2.4.2
    Speed is NumPy time divided by Bottleneck time
    NaN means approx one-fifth NaNs; float64 used

              no NaN     no NaN      NaN       no NaN      NaN    
               (100,)  (1000,1000)(1000,1000)(1000,1000)(1000,1000)
               axis=0     axis=0     axis=0     axis=1     axis=1  
nansum         21.3        0.8        2.1        1.7        2.2
nanmean        93.8        1.2        2.1        1.7        2.4
nanstd         91.6        1.6        2.3        2.0        2.6
nanvar         89.4        1.6        2.1        2.0        2.3
nanmin         20.7        0.4        0.5        0.6        0.7
nanmax         21.0        0.4        0.5        0.6        0.7
median         74.7        1.3        6.8        1.1        6.6
nanmedian      74.1        3.1        3.6        3.0        3.5
ss              7.0        0.8        0.8        1.0        1.0
nanargmin      43.3        1.8        3.8        1.4        4.8
nanargmax      43.6        2.9        6.5        2.2        5.0
anynan          8.0        0.4       35.7        0.7       27.5
allnan         11.4      154.3      100.4      139.8       86.5
rankdata       20.5        2.5        1.7        3.1        1.9
nanrankdata    23.3        2.9        1.4        2.9        1.4
partition       2.3        0.4        0.7        0.2        0.5
argpartition    4.2        0.6        1.4        0.5        1.6
replace         7.4        1.8        1.8        1.8        1.9
push         1513.1        9.9        9.6       21.4       14.0
move_sum     1546.1       48.0      179.2      249.5      238.7
move_mean    3753.6      133.1      187.0      330.0      264.8
move_std     5822.4      117.6      256.7      250.0      362.0
move_var     6180.6      146.7      297.3      299.5      412.7
move_min      878.3        5.7        5.6       12.6       16.5
move_max      857.8        5.4        5.3       12.1       15.9
move_argmin  1815.1       37.8       75.3       43.6       87.8
move_argmax  1843.1       39.1       76.6       43.3       90.6
move_median  1199.6       90.7      102.3       95.5      103.8
move_rank     584.6        1.0        1.2        1.5        1.7

this branch:

Bottleneck performance benchmark
    Bottleneck 1.6.0.post0.dev38; Numpy 2.4.2
    Speed is NumPy time divided by Bottleneck time
    NaN means approx one-fifth NaNs; float64 used

              no NaN     no NaN      NaN       no NaN      NaN    
               (100,)  (1000,1000)(1000,1000)(1000,1000)(1000,1000)
               axis=0     axis=0     axis=0     axis=1     axis=1  
nansum         22.8        0.9        1.9        1.2        2.1
nanmean        67.7        1.4        2.2        1.6        2.5
nanstd         84.1        2.0        2.3        2.0        2.6
nanvar        115.1        1.4        2.3        2.0        2.6
nanmin         18.8        0.3        0.4        0.4        0.5
nanmax         19.2        0.3        0.4        0.4        0.5
median         70.8        1.2        5.6        1.1        6.5
nanmedian      69.7        3.3        4.3        3.5        3.8
ss              7.1        0.8        0.8        1.0        1.0
nanargmin      54.3        2.4        5.1        1.3        4.5
nanargmax      53.5        3.0        6.3        1.7        5.7
anynan          6.5        0.4       37.8        0.5       35.8
allnan         10.1      153.3      105.8      138.2       98.3
rankdata       21.3        2.5        1.9        3.1        2.0
nanrankdata    24.4        2.9        1.5        3.6        1.6
partition       2.4        0.4        0.7        0.2        0.5
argpartition    4.0        0.6        1.3        0.5        1.5
replace         7.1        1.8        1.8        1.8        1.8
push         1565.4        9.6        9.5       20.1       13.8
move_sum     1637.4       70.6      171.5      248.3      240.7
move_mean    3939.6      123.2      193.0      339.6      262.7
move_std     6360.4      120.0      251.0      255.3      361.8
move_var     6540.3      143.3      302.4      302.0      428.5
move_min      931.7        6.7        6.7       13.7       16.2
move_max      998.1        5.5        5.6       12.6       a16.5
move_argmin  1923.1       39.6       76.7       43.7       88.3
move_argmax  2089.7       39.5       76.3       44.0       91.0
move_median  1275.1       95.6      102.6       91.9      103.3
move_rank     596.3        1.1        1.3        1.6        1.8

[neutrinoceros]

not sure any of these diffs are significant, but I think it's still worth making abi3 an opt-in so we can grow this space without worrying about maximising performance. I expect this will come in handy later this year when/if some version of abi3 makes it into Python 3.15t (see PEPs 803 and 809)

[rgommers]

not sure any of these diffs are significant,

It's pretty hard to tell, using asv compare is usually more informative. It looks like some of the faster-running benchmarks are slightly slower, but not much (as expected).

I'll do a quick pass on review now.

[rgommers]

The opt-in approach here looks good to me. The Stable ABI is typically irrelevant for distros as well as end users building from source, so opt-in seems better as the default in general.

A couple of initial comments.

[rgommers]

This should not be necessary, NPY_TARGET_VERSION isn't useful to use except for targeting newer API than the default.

[neutrinoceros]

good call, I'll remove this

[rgommers]

Centralizing NPY_NO_DEPRECATED_API seems like a useful refactoring either way indeed.

[rgommers]

This doesn't look quite right, I think we should always target the lowest-supported Python version, rather than the version that's running the build. Or do you plan to enforce (via code comment?) to handle this in the CI workflow file?

[neutrinoceros]

The problem is that when targeting a version that's older than the running interpreter, there are no guarantees that the resulting binaries will actually be backward compatible. Luckily, because cibuildwheel already runs targets in increasing Python version order, this approach works perfectly: the first (oldest) Python builds a wheel and runs tests, then all the other skip the build step and only check for forward compat by running tests.

[neutrinoceros]

see https://docs.python.org/3/c-api/stable.html#limited-api-caveats