8343606: Test FinalizerTest.java intermittent fails Debuggee heap OOM

[sendaoYan]

Hi all,

This is clean backport PR to fix the test bug which cause test run intermittent fails or intermittent timed out. Change include below:

1. Heap limit — Run the debuggee with -Xmx256M so OOM behavior is consistent on high-memory machines.

2. Memory cleanup — Call holdAlot.clear() before System.gc() in the allocation fallback path, so memory can be reclaimed before the second System.runFinalization().

3. Synchronization fix — Replace the finalizerRun boolean with a CountDownLatch so the main thread reliably waits for the low-priority finalizer thread to finish (a plain volatile flag was not enough).

4. Safer fallback loop — Use while (finalizerDone.getCount() > 0) instead of while (true) to avoid an infinite allocation loop when the race occurred.

5. Cleanup — Remove unused code, tidy imports, improve comments, and add a diagnostic message if the fallback path exits without triggering finalization.

The commit being backported was authored by SendaoYan on 28 Apr 2026 and was reviewed by Chris Plummer.

Thanks!

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• JDK-8343606 needs maintainer approval

Error

 ⚠️ OCA signatory status must be verified

Issue

• JDK-8343606: Test FinalizerTest.java intermittent fails Debuggee heap OOM (Bug - P3)

Reviewers

• Thomas Stuefe (@tstuefe - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk11u-dev.git pull/3188/head:pull/3188
$ git checkout pull/3188

Update a local copy of the PR:
$ git checkout pull/3188
$ git pull https://git.openjdk.org/jdk11u-dev.git pull/3188/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3188

View PR using the GUI difftool:
$ git pr show -t 3188

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk11u-dev/pull/3188.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back syan! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[openjdk]

⚠️ @sendaoYan This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

[mlbridge]

Webrevs

• 01: Full - Incremental (4862ba65)
• 00: Full (d8f11ec1)

[sendaoYan]

Change has been verified locally by run the test on linux-x64.

[bridgekeeper]

@sendaoYan This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[tstuefe]

Please describe what the patch does, and why it is important to fix this in a stable backport release. How often is intermittent? Have we seen this problem in jdk 11 ci/cd tests?

[sendaoYan]

Please describe what the patch does

I have updated the PR description.

why it is important to fix this in a stable backport release. How often is intermittent?

I can reproduce this test intermittently fails or timed out by run this test 500 times on my local linux intel-8480 machine. It's clean backport, and test-fix only. I think this backport PR has no risk. The failure probability is about 30/1k on my local intel-8480 machine. The reproduce command line shows in JBS issue.

Have we seen this problem in jdk 11 ci/cd tests?

I think it's not. Maybe com/sun/jdi/FinalizerTest.java is not tier1 test.

[tstuefe]

Ok.

Please make sure the GHAs run through.

[jerboaa]

@sendaoYan Please work with relevant OCA people to get your OCA status cleared (I don't know what changed 3 weeks ago). There is no way we can integrate anything before that happens.