Add stats estimation accuracy to benchmarks

[gabotechs]

This PR enhances the benchmarks by comparing estimated bytes vs actual bytes in network boundaries during dynamic planninng:

• Compare sampled bytes with actual network-boundary output bytes using q-error.
• Store it per iteration in local and remote benchmarks.
• Report median q-error (QERR P50) in benchmark comparisons.

Preliminary step for #552.

---

Sample output:

=== Comparing tpch_sf10 results from branch 'stats-accuracy-benches' [prev] with 'stats-accuracy-benches' [new] ===
os:        macos
arch:      aarch64
cpu cores: 16
threads:   2 -> 2
workers:   8 -> 8
===================================================================================================================
tpch_sf10 q1: prev= 327 ms, new= 355 ms, diff=1.09 slower ✖
tpch_sf10 q2: prev= 271 ms, new= 278 ms, diff=1.03 slower ✖
tpch_sf10 q3: prev= 378 ms, new= 346 ms, diff=1.09 faster ✔
tpch_sf10 q4: prev= 132 ms, new= 129 ms, diff=1.02 faster ✔
tpch_sf10 q5: prev= 617 ms, new= 610 ms, diff=1.01 faster ✔
tpch_sf10 q6: prev= 115 ms, new= 109 ms, diff=1.06 faster ✔
tpch_sf10 q7: prev= 765 ms, new= 743 ms, diff=1.03 faster ✔
tpch_sf10 q8: prev= 781 ms, new= 889 ms, diff=1.14 slower ✖
tpch_sf10 q9: prev=1040 ms, new=1010 ms, diff=1.03 faster ✔
tpch_sf10 q10: prev= 448 ms, new= 455 ms, diff=1.02 slower ✖
tpch_sf10 q11: prev= 132 ms, new= 121 ms, diff=1.09 faster ✔
tpch_sf10 q12: prev= 200 ms, new= 205 ms, diff=1.02 slower ✖
tpch_sf10 q13: prev= 233 ms, new= 232 ms, diff=1.00 faster ✔
tpch_sf10 q14: prev= 158 ms, new= 171 ms, diff=1.08 slower ✖
tpch_sf10 q15: prev= 262 ms, new= 241 ms, diff=1.09 faster ✔
tpch_sf10 q16: prev=  92 ms, new=  94 ms, diff=1.02 slower ✖
tpch_sf10 q17: prev= 833 ms, new= 816 ms, diff=1.02 faster ✔
tpch_sf10 q18: prev= 917 ms, new= 946 ms, diff=1.03 slower ✖
tpch_sf10 q19: prev= 187 ms, new= 199 ms, diff=1.06 slower ✖
tpch_sf10 q20: prev= 422 ms, new= 420 ms, diff=1.00 faster ✔
tpch_sf10 q21: prev=1044 ms, new=1022 ms, diff=1.02 faster ✔
tpch_sf10 q22: prev= 101 ms, new= 103 ms, diff=1.02 slower ✖
   TOTAL: prev=9455 ms, new=9494 ms, diff=1.00 slower ✖
QERR P50: prev=n/a, new=19.47x

[gabotechs]

Now these two are always enabled. I don't think we want them disabled at any moment, as they are pretty stable now.

[asolimando]

Unless tomorrow we want to follow Spark example and have a global AQE knob to turn it off, at that point we can avoid the overhead as there is no consumer. But we can probably do this later

[asolimando]

We would probably have the same effect by not inserting SampleExec nodes, so it's probably enough to do that tomorrow to disable AQE

[asolimando]

LGTM, great work!

[asolimando]

I think bytes will be the dominating factor in most cases, but we estimate row count too, I am OK to start with bytes only but if we see that this doesn't correlate with performance benchmarks, we might need to track row counts similarly.

Similarly for median, it's more stable than avg, and I think it's what we need to track, but we might need to add other percentile values possibly if we see that median is not enough.

[gabotechs]

I think bytes will be the dominating factor in most cases, but we estimate row count too

Yeah, in fact, for future optimizations, rows will start being relevant, as that's what DF optimizer uses for several things.

As today we don't do those optimizations dynamically, and we just care about bytes for sizing the stages, bytes is the only thing that actually use.

When we start doing the other optimizations, I think we should start taking into account output rows as well.

[gabotechs]

@asolimando I'm not sure if median is the best here... it has the chance of swallowing very bad estimations in cases there are enough straight-forward stages in the plan to even that out.

I'm thinking that avg might be a bit better. WDYT?

[gabotechs]

I'm getting some more meaningful metrics with a mean rather than median here, I'm going to go for it.

[asolimando]

I briefly touched upon this here and I agree that median, being robust to outliers, will "absord" degenerate cases that might result in individual queries to fail.

Being robust to outliers is a good property for this type of measures, avg would suffer from the opposite problem, it won't provide a clear signal for the "regular" case and conflate the two behaviors.

To track also outliers we need to complement median with P90 or P95, exactly as https://vldb.org/pvldb/vol9/p204-leis.pdf does (look at Table 1 page 4 for an example).

Median will already provide a good signal for most cases but as we move forward and refine the approach we will surely need more to avoid regressions for extreme cases.

If that's easy enough to add now, it would be safer, if you don't feel like adding it now, we can wait and see if better q-error median values positively correlates with improvements in benchmark results, otherwise we need extra signals.

[gabotechs]

🤔 this is a good call... it's not trivial to do, but it's worth it, because I'm seeing that P50 is swallowing pretty wild overestimation, just because there's a few stages that are trivial to estimate (because 100% of the data was sampled for example)

[gabotechs]

Even if it's not super trivial, I'm going to go with P95 additionally, even if that will bloat a bit this PR.

[gabotechs]

Done, P50 and P95 looking very good now, thanks for the suggestion!