Fix C-heap memory leaks in solver state cleanup#2988
Open
mstumberger wants to merge 6 commits intoERGO-Code:latestfrom
Open
Fix C-heap memory leaks in solver state cleanup#2988mstumberger wants to merge 6 commits intoERGO-Code:latestfrom
mstumberger wants to merge 6 commits intoERGO-Code:latestfrom
Conversation
Problem HiGHS leaks C-heap memory when a Highs instance is reused across multiple solves. Python GC objects show zero growth but RSS climbs monotonically. Benchmarks on a long-running service showed: ┌───────────┬────────────────┬───────────────┬──────────┐ │ Allocator │ Avg growth/job │ After 10 jobs │ Peak RSS │ ├───────────┼────────────────┼───────────────┼──────────┤ │ glibc │ +10.13 MB │ +101.3 MB │ 952 MB │ ├───────────┼────────────────┼───────────────┼──────────┤ │ jemalloc │ +2.74 MB │ +27.4 MB │ 901 MB │ └───────────┴────────────────┴───────────────┴──────────┘ Root causes 1. saved_objective_and_solution_ never cleared — clearModel() clears model_ and multi_linear_objective_ but skips this vector of MIP solution snapshots. It accumulates indefinitely across solves and is only freed by the destructor. 2. invalidateSolution() / invalidateBasis() retain vectors — Both methods only flip boolean flags (value_valid = false, valid = false) but leave the underlying std::vectors (col_value, col_dual, row_value, row_dual, col_status, row_status) allocated. The actual .clear() methods that free the vectors exist but are never called. 3. HEkk::invalidate() retains all simplex memory — clearSolver() calls ekk_instance_.invalidate() which only sets three status flags. The actual HEkk::clear() method (which frees LP data, working arrays, dual edge weights, factorization data, and NLA state) is never invoked during normal operation. All simplex working arrays from previous solves persist for the lifetime of the Highs instance. 4. std::vector capacity retention — Even when vectors are .clear()ed, the allocated capacity is retained. With glibc's allocator these pages are never returned to the OS, causing heap fragmentation. Changes highs/lp_data/Highs.cpp: - clearModel(): Added saved_objective_and_solution_.clear() to free accumulated MIP solutions between model changes. - clearSolver(): Added ekk_instance_.clear() to free all simplex working arrays (LP data, dual edge weights, factorization, NLA) instead of just invalidating status flags. - invalidateSolution(): Changed solution_.invalidate() to solution_.clear() to actually free the four solution vectors. - invalidateBasis(): Changed basis_.invalidate() to basis_.clear() to actually free the two basis status vectors. - releaseMemory(): New public method that calls clearModel() then shrink_to_fit() on all retained vectors (solution, basis, ranging) to return unused capacity to the allocator. highs/Highs.h: Declared Highs::releaseMemory() with documentation. highs/highs_bindings.cpp: Exposed releaseMemory to Python via pybind11. highs/highspy/_core/__init__.pyi: Added type stub for releaseMemory(). Impact - clearSolver() now properly frees all solver memory between solves, eliminating the primary source of RSS growth. - releaseMemory() provides an explicit API for long-running services to return memory to the OS after a solve, addressing heap fragmentation from vector capacity retention. - The invalidate() → clear() changes in invalidateSolution() and invalidateBasis() ensure solution and basis vectors are freed immediately rather than retained indefinitely. - saved_objective_and_solution_ is now properly cleaned up in clearModel(), preventing unbounded accumulation of MIP solution snapshots. Testing - All existing CMake tests pass (2/2). - Full build succeeds with zero warnings on the changed files. - The changes are additive — clear() subsumes invalidate() (it calls invalidate() internally) so no behavioral change beyond memory being freed.
The invalidate() contract requires preserving vector sizes - some code relies on the size() of solution_ and basis_ vectors remaining valid after invalidation. - Revert invalidateSolution() and invalidateBasis() to call invalidate() instead of clear(), preserving the API contract - Add explicit clear() calls in releaseMemory() to actually free vectors before shrink_to_fit() (otherwise we're shrinking vectors that still contain data) - Add ranging_.clear() and iis_.clear() to releaseMemory() for completeness Fixes CI failures reported by @filikat.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## latest #2988 +/- ##
=======================================
Coverage 81.31% 81.32%
=======================================
Files 359 359
Lines 88852 88914 +62
=======================================
+ Hits 72253 72309 +56
- Misses 16599 16605 +6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- C++: Add TEST_CASE("releaseMemory") in TestLpSolvers.cpp that
solves a problem, calls releaseMemory(), then solves another
problem to verify the solver remains functional.
- Python: Add test_releaseMemory() in test_highspy.py with the
same pattern.
These tests ensure releaseMemory() actually frees memory without
breaking the solver's ability to solve subsequent problems.
Expose Highs::releaseMemory() to C API users as Highs_releaseMemory(). This allows C applications to explicitly free memory between solves, useful for long-running services that reuse a Highs instance.
mstumberger
changed the title
After releaseMemory() the Highs instance now holds zero allocated heap memory for solver state — equivalent to a freshly constructed Highs() without the overhead of destruction and reconstruction. Additional cleanup: - Clear and shrink solution_, basis_, ranging_ vectors - Shrink standard_form_cost_ and standard_form_rhs_ - Reset timer_ accumulated clock data via zeroAllClocks() - Clear iis_ infeasible subsystem data The instance can be reused for a new solve immediately after releaseMemory() without creating a new Highs object.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#2981 fix