first method in network performance test takes longer

[acostarelli]

I discovered by accident that running the same test with the same refinements multiple times, the first one iteration twice as long.

I ran Bin2 + solver SOS with refinement = 4 on the lossy network problem.

====================================================================================================
MIP Bilinear Approximations (HiGHS)
  Refinement = num_segments for SOS2 methods, depth for Sawtooth
====================================================================================================
Method             Ref   Vars Constrs   Bins    Objective    Gap(%)  Mean Resid  Max Resid  Time(s)
----------------------------------------------------------------------------------------------------
Bin2+sSOS            2    190     495      0     0.964218        -    1.52e-01   4.48e+01   3.0377

Bin2+Saw             2    190     495      0     0.964218        -    1.52e-01   4.48e+01   1.1425

HybS+sSOS            2    190     495      0     0.964218        -    1.52e-01   4.48e+01   1.1417

HybS+Saw             2    190     495      0     0.964218        -    1.52e-01   4.48e+01   1.1407

====================================================================================================

The methods and ref columns do not reflect the actual methods and refinements.

[luke-kiernan]

Looks like typical Julia stuff: search TTFX ("time to first execution"). The first call triggers compilation: parsing, type inference, LLVM codegen, etc. Subsequent calls reuse those products, so they're faster. That's why our performance script runs things twice: first time to measure compilation, second time to measure actual runtime.

[acostarelli]

Yeah this issue might be resolved? We added in running a small network test before the main run on local and CI/CD environments that solves this. But when running on kestrel, each method gets run in a separate Julia instance so it's not doing this pre-run. Yet the build times still look weird, with smaller MIP refinements taking longer to build in some cases.

[luke-kiernan]

Yet the build times still look weird, with smaller MIP refinements taking longer to build in some cases.

If you figure a reasonably self-contained way to replicate this reliably, please do share. Could be some type instability problems.