Profile: add multistart optimization for profile points by Doresic · Pull Request #1712 · ICB-DCM/pyPESTO

Doresic · 2026-05-07T16:39:36Z

I added multistart optimization at each profile point.

Additional starts are sampled around the point suggested by profile_next_guess, and the original suggested start is always included as well. Failed starts are tolerated, and the best finite result is kept.

I also added warnings when profiling finds lower function values than the supplied optimum, and when non-profiled parameters hit bounds during profiling.

Also:

add profile_n_starts and profile_sampling_sigma to ProfileOptions
use multistart optimization during profile walking
extend the related profiling tests

Open question: when non-profiled parameters hit bounds during profiling, should we eventually support relaxing those bounds (of the parameters we're not profiling), or is warning-only the better behavior here? Allowing a relaxation of the bounds would make the plots actually show non-identifiable parameters when they are such. But we might risk some weird optimisation failures in some unusual cases.

Should be merged only after #1711 -- contains all changes of that PR currently, too.

codecov-commenter · 2026-05-07T16:44:03Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 87.75510% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.37%. Comparing base (7255fc7) to head (0f84792).

Files with missing lines	Patch %	Lines
pypesto/profile/profile.py	72.72%	3 Missing ⚠️
pypesto/profile/walk_along_profile.py	90.62%	3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1712      +/-   ##
===========================================
+ Coverage    84.35%   84.37%   +0.01%     
===========================================
  Files          164      164              
  Lines        14752    14799      +47     
===========================================
+ Hits         12444    12486      +42     
- Misses        2308     2313       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…filing

We were changing the step size of profiling parameter to 0 by accident.

…filing

- explicit absolute/relative profile step-size options - deprecated old absolute step-size option names - per-parameter step-size family resolution from `ub - lb` - finite-bound validation before profiling - single upfront step-size resolution before profile walking - resolved step sizes passed through profiling tasks/proposals - lightweight many-steps precheck warning/error mode - focused profile tests for options, resolution, precheck

…filing

PaulJonasJost

Regarding open question: I am strongly against loosening bounds. They are there for a reason, even if they are set uninformed. Thus they should not be just changed, especially not mid workflow. Other view on this is that if I loosen the bounds, I do not guarantee at all anymore, that my optimum is actually an optimum, e.g. if in the optimisation we already hit the same bound.

PaulJonasJost · 2026-06-02T08:06:44Z

        reg_order: int = 4,
        adaptive_target_scaling_factor: float = 1.5,
        whole_path: bool = False,
+        profile_n_starts: int = 6,


Wouldn`t keeping the default at one or two be better? In case everything works it is 1/6 (1/3)of the overall work and otherwise depending on how fatally it fails, we can add starts?

True
This is the design choice of the default being either

robust according to the benchmark we ran

initial non-robust run but faster --> in models with non-identifiability will probably show many jumps.

Having less starts had a large impact on the amount of jumps the profiles showed for the benchmark models I tested.

Based on previous discussions we had in team meetings, it seemed that the default we would like to have would be the robust one -- so all settings are following that -> order 0 (next PR), medium step size (previous PR), 6 multistarts.

Would you vote for the efficient version rather the robust one?

PaulJonasJost · 2026-06-02T08:09:02Z

+        logger.warning(
+            "Profiling found lower function values than the supplied "
+            f"optimization result (fval = {global_opt:.6g}) for:\n"
+            f"{par_lines}\n"
+            "This means profiling was started from a suboptimal point, so the "
+            "profile ratios and confidence thresholds are not anchored to the "
+            "true global optimum. Re-optimization of the model is suggested. "
+            "The better profile points can be included as startpoints "
+            "through `x_guesses` of the optimization problem."
+        )


If I get the message right it is notifying me when I find a point actually better then the best found point from optimization? Like the message, but I think it should also include how to get the best points out if it includes the x_guesses hint

PaulJonasJost · 2026-06-02T08:09:31Z

+            problem=problem,
+            x0=startpoint,
+            id=str(0),
+            optimize_options=OptimizeOptions(allow_failed_starts=True),


why allow failed starts?

Actually in this one I tend to understand (even though it should be impossible. But In 76 it seems more weird.

I'd also stick to False. Maybe make it configurable.
In my opinion, Objective.__call__ should never raise when called with admissible parameters. If it does, this is most likely a problem with the Objective and we should not cover that up.

True
I put it there with something else in mind that doesn't apply. Will change to False

PaulJonasJost · 2026-06-02T08:12:18Z

+
+        for k, j_par in enumerate(problem.x_free_indices):
+            x_j = optimizer_result.x[j_par]
+            lb_j = problem.lb[k]
+            ub_j = problem.ub[k]
+            at_lb = abs(x_j - lb_j) <= 1e-8
+            at_ub = abs(x_j - ub_j) <= 1e-8
+            if (at_lb or at_ub) and j_par not in warned_at_bound:
+                warned_at_bound.add(j_par)
+                bound_val = lb_j if at_lb else ub_j
+                logger.warning(
+                    f"Parameter '{problem.x_names[j_par]}' hit its "
+                    f"{'lower' if at_lb else 'upper'} bound "
+                    f"({bound_val:.4g}) while profiling "
+                    f"'{problem.x_names[i_par]}'. "
+                    "The profile may be constrained near this region."
+                )
+


Not sure how informative that actually is, as it should become very apparent in the 2d plots right? Would potentially remove.

True
Still wanted to add it as Jan mentioned it was in MATLAB previously, and it might just indicate something "might be happening" to users.
I don't think it hurts, as we also send it only once ever per parameter. So logs won't get cluttered with it.

PaulJonasJost · 2026-06-02T08:12:59Z

        assert not precheck_warnings


+def test_profile_multistart_optimize_uses_best_start(monkeypatch):


what does monkeyxpatch do?

dweindl

Nice!

dweindl · 2026-06-02T09:48:22Z

+        additional optimization startpoints from a Gaussian centered at the
+        point suggested by `profile_next_guess`, in the reduced parameter
+        space. The profiled parameter itself is already fixed and is therefore
+        not perturbed.


Suggested change

not perturbed.

not perturbed. Only relevant if `profile_n_starts > 1`.

Or similar.

dweindl · 2026-06-02T09:53:48Z

        adaptive_target_scaling_factor: float = 1.5,
        whole_path: bool = False,
+        profile_n_starts: int = 6,
+        profile_sampling_sigma: float = 0.01,


Since this is an absolute sigma, it might be nice to support per-parameter settings, but I guess this can also easily be extended later.

dweindl · 2026-06-02T10:00:41Z

+        better_optima.append((problem.x_names[i_par], best_fval))
+    if better_optima:
+        par_lines = "\n".join(
+            f"  {name}: best fval = {fval:.6g}" for name, fval in better_optima


With the isclose thresholds above, this may display identical values, which might be confusing.

dweindl · 2026-06-02T10:16:33Z

+            problem=problem,
+            x0=startpoint,
+            id=str(0),
+            optimize_options=OptimizeOptions(allow_failed_starts=True),


I'd also stick to False. Maybe make it configurable.
In my opinion, Objective.__call__ should never raise when called with admissible parameters. If it does, this is most likely a problem with the Objective and we should not cover that up.

Doresic added 2 commits May 7, 2026 15:48

Improve profiling step-size defaults and step-size robustness

3a8dce3

Profile: add multistart optimization for profiling

3ef1b13

Doresic added 4 commits May 8, 2026 14:24

Fix extrapolation trust-region not using resolved

d943288

Merge branch 'profile-step_size_changes' into profile-multi_start_pro…

3122fb0

…filing

Fix small bug

12e20eb

We were changing the step size of profiling parameter to 0 by accident.

Merge branch 'profile-step_size_changes' into profile-multi_start_pro…

d5fa6c8

…filing

Doresic changed the base branch from develop to profile-step_size_changes May 13, 2026 12:10

Doresic added 5 commits May 18, 2026 14:50

Merge branch 'profile-step_size_changes' into profile-multi_start_pro…

c95dc30

…filing

Change relative defaults

0440810

Merge branch 'develop' into profile-step_size_changes

2bd0854

Merge branch 'profile-step_size_changes' into profile-multi_start_pro…

640094b

…filing

Doresic marked this pull request as ready for review May 29, 2026 08:38

Doresic requested a review from PaulJonasJost as a code owner May 29, 2026 08:38

Base automatically changed from profile-step_size_changes to develop May 29, 2026 12:03

Merge branch 'develop' into profile-multi_start_profiling

0f84792

PaulJonasJost approved these changes Jun 2, 2026

View reviewed changes

dweindl reviewed Jun 2, 2026

View reviewed changes

		assert not precheck_warnings


		def test_profile_multistart_optimize_uses_best_start(monkeypatch):

	not perturbed.
	not perturbed. Only relevant if `profile_n_starts > 1`.

Conversation

Doresic commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

PaulJonasJost left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dweindl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Doresic commented May 7, 2026 •

edited

Loading

codecov-commenter commented May 7, 2026 •

edited

Loading