8350807: Certificates using MD5 algorithm that are disabled by default are incorrectly allowed in TLSv1.3 when re-enabled

[jerboaa]

Please review this fix for JDK 11u for Oracle JDK 11.0.29 parity. Product code changes are mostly clean. The patch doesn't apply cleanly because of test changes mostly:

1. Changes needed in the test, MD5NotAllowedInTLS13CertificateSignature.java to compile with JDK 11.
2. SessionTicketExtension.java not being in 11u because 8211018: Session Resumption without Server-Side State is not in 11u.
3. Two additional tests needed to explicitly use TLS 1.2 in order for them to work. They'd fail with the expected MD5 not allowed message otherwise. These tests are: PKIXExtendedTM.java and SunX509ExtendedTM.java

Testing

• :jdk_security3 and :jdk_net test groups with result outlined in [1].
• GHA

Thoughts?

[1]

==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
>> jtreg:test/jdk:jdk_security3                        799   798     1     0 <<
>> jtreg:test/jdk:jdk_net                              737   735     2     0 <<
==============================

Same failures before/after patch. Failed tests are:

jdk_net:
java/net/httpclient/ConnectTimeoutWithProxyAsync.java                             Failed. Execution failed: `main' threw exception: java.lang.Exception: failures: 48
java/net/httpclient/ConnectTimeoutWithProxySync.java                              Failed. Execution failed: `main' threw exception: java.lang.Exception: failures: 48

jdk_security3:
sun/security/ssl/X509KeyManager/PreferredKey.java                                          Failed. Execution failed: `main' threw exception: java.lang.Exception: Failed to get the preferable key aliases

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• JDK-8350807 needs maintainer approval

Issue

• JDK-8350807: Certificates using MD5 algorithm that are disabled by default are incorrectly allowed in TLSv1.3 when re-enabled (Bug - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk11u-dev.git pull/3214/head:pull/3214
$ git checkout pull/3214

Update a local copy of the PR:
$ git checkout pull/3214
$ git pull https://git.openjdk.org/jdk11u-dev.git pull/3214/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3214

View PR using the GUI difftool:
$ git pr show -t 3214

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk11u-dev/pull/3214.diff

Using Webrev

Link to Webrev Comment

[jerboaa]

@franferrax Could you please help review this? Thank you!

[bridgekeeper]

👋 Welcome back sgehwolf! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 01: Full - Incremental (32535ee1)
• 00: Full (1231835d)

[jerboaa]

Note to self: #3222 will introduce a utility for SecureRandom.nextLong(long bound) usage, which this patch should then also use.

[jerboaa]

Note to self: #3222 will introduce a utility for SecureRandom.nextLong(long bound) usage, which this patch should then also use.

I've updated the patch to use Utils.nextLong() in the new test now.