Renew star.jquery.com cert (expires 26 Jun 2026)

[Krinkle]

Previous: #77, #50, #21.
Instructions: https://github.com/jquery/infrastructure-puppet/blob/staging/doc/cdn-cert.md

---

This year is going to be different because the CAB Forum approved a decision past year to gradually reduce the maximum lifetime of all TLS certificates to 47 days by 2029.

Ryan Aslett (Linux Foundation) tells us that our current cert provider, Sectigo, has already recently lowered the maximum from 356 to 200 days and it'll reduce to 100 days next year. (see Sectigo April 2025 press release).

So we could certainly renew this the usual way one more time, with 200 days, but by next year we should really be on automation, I think. I suggest we explore it a bit now, to see where we stand, and potentially switch this year if possible.

Timeline

Date	Action
Sat, 25 Apr 2026	Annual reminder from bot in private channel, two months margin.
Thu, 30 Apr 2026	Discussion in private channel with Ryan Aslett.
Fri, 1 May 2026	Create this issue and continue discussion in public.
Fri, 8 May 2026	Created a ticket with LF IT to request new certs.
Fri, 8 May 2026	LF IT confirmed receipt of the request.
Wed, 20 May 2026	Confirm timeline with LF IT.
Thu, 21 May 2026	Certs issued and received.
Tue, 26 May 2026	Certs locally verified.
Tue, 26 May 2026	Added new cert to Fastly and activated only on r.sni for staging via code2.jquery.com (DNS for code.jquery.com uses k.sni instead).
Tue, 26 May 2026	Tested new cert, found regression in Safari 7.1 (macOS 10.9).
Wed, 27 May 2026	Fixed cert received.
Wed, 27 May 2026	Replace new cert in Fastly for r.sni staging.
Wed, 27 May 2026	Tested new cert (again), confirmed to pass our usual deep range.

[Krinkle]

Last month, I actually started experimenting with Fastly TLS already, over at #84, by placing typesense.jquery.com behind Fastly. Note that Fastly operates with a single shared list of certificates on its servers, and the certificate applied to any given request cannot be controlled by us. It uses whichever certificate matches the request, and it does not permit deploying conflicting certs. That is: You can't have two overlapping certificates on the same cluster (e.g. same domain with different expiry, or have an overlapping domain list, or e.g. custom wildcard *.jquery.com alongside Let's Encrypt for foobar.jquery.com). For example, you cannot pin code.jquery.com to the 2025 wildcard cert, while foobar uses a new Let's Encrypt foobar.jquery.com cert; unless these domains are served by an entirely different Fastly cluster.

I summarised our previous findings about Fastly clusters at https://github.com/jquery/infrastructure-puppet/blob/docs/doc/fastly.md#tls-hostnames for easy reference.

Our 2025 wildcard cert is deployed to the k.sni Fastly cluster, which offers the broadest browser/TLS compatibility, and code.jquery.com is assigned to that cluster in DNS.

The t.sni Fastly cluster is our fallback and what we usually assign to code2, to test the "next" certicicate. The t.sni cluster is similar to k.sni, lacking only the CBC algo needed for Windows 7 compat (afaik Fastly only have one cluster with that configuration).

I've assigned typesense.jquery.com to the n.sni cluster, which has the same configuration as t.sni, and not used by us yet.

Cloudflare. As part of #79 I ran a wide range of tests with the "Cloudflare Universal" certs provided via the Enterprise account we have with Cloudflare. To my surprise, this actually reaches a broader range of devices than our Sectigo + k.sni Fastly setup for the jQuery CDN. In a nut shell, at least for Enterprise customers, Clouflare does not use Let's Encrypt and instead:

• cross-signs with both Google Trust and... Sectigo!
• generates both a modern ECDSA cert and a legacy RSA cert (I believe they use connection sniffing to decide which one to serve up; allowing for MITM in theory, but this is not a concern to us).
• includes CBC algo for Windows 7 compat
• includes TLS 1.0 and TLS 1.1 (optional, we're in the process of turning this off at Should we disable TLS 1.0/1.1? #79)

The last point is why it is broader than our custom Sectigo setup on the k.sni Fastly cluster.

[ryanaslett]

The easiest path would be to switch to something Fastly offers as part of their service, as that entails the least amount of disruption/change.

Fastly offers 3 different options for ACME certs: Let's Encrypt, Certainly, and Globalsign.

The globalsign cert has potential for reaching many very old devices: https://support.globalsign.com/ca-certificates/globalsign-root-certificates -> the globalsign root R1 seems like it goes back to the 90's, but its unclear if the fastly ACME path would include something with that CA root cross signed or not. ->

Certainly, fastly's own root CA, while newer, is cross signed by older godaddy CA: https://www.fastly.com/blog/announcing-certainly-fastlys-own-tls-certification-authority)

And I believe we've already looked at Lets Encrypt, found some of its offerings to not be backwards compatible enough.

Is the policy for legacy device support documented somewhere? or is it somewhat cultural of "we should not break old working devices"? I'm curious if there is a rolling window of "X years old" or "Still in security support" or something like that.

[timmywil]

We've historically tried to look at global usage stats, rather than enforcing an arbitrary deadline. That said, we haven't checked again since last November.

[ryanaslett]

What kind of stats do you have? Browser user agent on the requests, or some form of telemetry in jquery itself? Im curious myself what y'all are seeing out there, given how long of a shadow jquery casts.

[Krinkle]

I don't think we should research or drop support for any devices within this renew cycle. We either find a setup that is effectively equivalent in its reach, or we renew and then work out a more gradual and less rushed plan in time for next year (including announcements, social comms, proactive outreach to relevant sites that seem to be have more affected users than others, encourage alternatives in the form of self-hosting, etc).

I'm optimistic that we'll find a suitable setup with Fastly, because it isn't "just Let's Encrypt". It looks quite decent and is worth trying out and comparing.

[ryanaslett]

This renew cycle is easy enough to do the same way we've been doing it, but it only buys us 200 days from June 26th when the new cert goes in.

I'm suggesting that we research the two fastly options (certainly/globalsign) now for device support to determine if they are equivalent to what we have, or if they would require dropping device support.

If they are equivalent, then we can switch to using one of those fastly options instead of using the sectigo renewal process.

If they are not, and require dropping device support, then we use the sectigo renewal process, and do the announcements+outreach mentioning that support will be dropped in January 2027 for whichever devices would no longer be supported.

I think the only other thing we may lose in this switchover, regardless of device support, is the methodology we use for replacing certs to avoid clock skew issues. The fastly ACME cert renewals may also account for that, but theres a high likelyhood they do not.

[Krinkle]

I've switched https://podcast.jquery.com from Certainly (Fastly TLS) to GlobalSign (Fastly TLS) so that we can compare these three side-by-side.

service	cluster	certificate
https://releases.jquery.com/	k.sni	*.jquery.com Sectigo (self-managed)
https://typesense.jquery.com/health	n.sni	Certainly (Fastly-managed)
https://podcast.jquery.com/	s.sni	GlobalSign (Fastly-managed)
https://stage.api.jquery.com/	(No Fastly)	Let's Encrypt (self-hosted certbot)

[Krinkle]

browser/service	Sectigo (releases.jq)	Certainly (typesense.jq)	GlobalSign (podcast.jq)	Let's Encrypt (stage.api.jq)
iOS 7 (iPad Mini simulator)	OK	Fail to connect	Fail to connect	Fail to trust (connect OK)
iOS 9.3 (iPad Mini 4 simulator)	OK	OK	OK	Fail to trust (connect OK)

I stopped testing after this because our doubt is exactly in the place we can't test outside k.sni, namely behind the connection fail due to missing algo. We know that a "Fail to trust" means connection worked but the cert isn't trusted (usually due to unknown root), which will fail the same way on the k.sni cluster. The one we care about however is "Fail to connect" which might succeed or might fail, we don't know until we deploy it to k.sni.

I've asked Fastly Support if k.sni is the only cluster with CBC algo, or whether there is another cluster where we can test this (or possibly, if there is indeed a way to test this on k.sni without conflicting with our self-managed cert).

I've created an LFIT ticket meanwhile to go ahead with the renewing the Sectigo cert.

[ryanaslett]

One option For testing the k.sni, we do own jquery.net, and it currently has no DNS records assigned to it at all. -> We could potentially add that to fastly, and set it up to point at a placeholder somewhere and explore it that way? That would eliminate any conflicts with jquery.com. Im not sure whether we can add arbitrary domains to the existing fastly account.

[Krinkle]

Right, of course. Okay, I've switched https://bugs.jqueryui.com/ to Fastly with a Fastly-managed cert from GlobalSign, deployed to the k.sni cluster.

This one doesn't need a CDN per-se, but it is an easy one to experiment with because it is already served from the miscweb server (which has a corresponding Fastly service configured), and is one of the few non-redirect sites we serve from the miscweb server that is outside *.jquery.com.

[Krinkle]

[…] I think the only other thing we may lose in this switchover, regardless of device support, is the methodology we use for replacing certs to avoid clock skew issues.

There's broadly two ways to mitigate this: Wait or backdate.

Waiting before activation is what we currently do and what large-scale site owners have historically done, even if they don't know or mean to. When certificates are manually requested, granted, sent between two or three parties, and then verified, staged, tested, and deployed; it easily takes several hours, even if you don't intentionally wait like we do.

With automation (ACME, Certbot, Let's Encrypt) most steps are skipped and the rest can happen in a matter of seconds, thus making clocks more important than ever. Fortunately, this is accounted for through the practice of backdating. I wasn't aware of backdating as a practice until I researched this today, and I'm guessing this wasn't a thing for CAs historically (nor did they need to). I found a fewanecdotal mentions, e.g. in the Let's Encrypt community forums, and confirmed in the issue tracker of their "boulder" implementation:

Let's Encrypt deliberately backdates the certificate Not Before date by 1 hour to help with client clock skew.

Let's Encrypt certificates are backdated […] to prevent issues with computers whose system clocks are off by a bit.

And while server clockskew should be less severe than client clockskew, when we're talking about only a few seconds, this becomes sensitive there as well, as mentioned in another discussion:

Let's Encrypt intentionally backdates certificates by one hour to avoid potential issues with clock skew, where a server's clock might be slightly behind the time the certificate was issued, preventing immediate use.

I couldn't find anything in the ACME RFCs at IETF about this, except in RFC 8739 (about wildcard certs) where there's an example suggesting to start rewewal attempts halfway through the cert lifetime (e.g. 45 days into a 90-day cycle) and for the issueing server to backdate this by 3 days (capped by when the first cert for this server was issued). From RFC 8739:

     "auto-renewal": {
       "start-date": "2019-01-10T00:00:00Z",
       "end-date": "2019-01-20T00:00:00Z",
       "lifetime": 345600,          // 4 days
       "lifetime-adjust": 259200    // 3 days
     }

The amount of time that needs to be subtracted from each nominal renewal date is 3 days […]

This RFC actually mentions the same Google paper (Acer 2017) that I based our practice on (jQuery docs > Wikimedia T196248 > Acer 2017):

[…] Nevertheless, this section attempts to provide reasonable suggestions for the Web use case, informed by current operational and research experience. […] 5 days cover 97.3%, 7 days cover 99.6%. […]

And for completeness, the paper also found that 1 days covers 93.3%:

In the [sample], the client clock was more than 24 hours behind in 6.7% of reports […]

Let's Encrypt

Let's look at our own self-hosted Let's Encrypt first.

• Domain: https://stage.api.jquery.com/
• notBefore: Fri, 20 Mar 2026 23:19:13 GMT
• notAfter: Thu, 18 Jun 2026 23:19:12 GMT (90 days)

wp-03.stage:/var/log/letsencrypt$ fgrep fullchain.pem letsencrypt.log | tail -n140 | less
…
2026-03-20 15:05:07:DEBUG:certbot:   /etc/letsencrypt/live/wordpress/fullchain.pem expires on 2026-04-19 (skipped)
2026-03-21 00:17:45:DEBUG:certbot:   /etc/letsencrypt/live/wordpress/fullchain.pem (success)
2026-03-21 13:21:56:DEBUG:certbot:   /etc/letsencrypt/live/wordpress/fullchain.pem expires on 2026-06-18 (skipped)
…

I'll assume for now that our server enabled it immediately upon receipt. It was renewed 30 days before expiry, and was backdated by approximately 1 hour.

Certainly (Fastly-managed)

I moved typesense.jquery.com behind Fastly on the 1st of April (ref #84), using Certainly. It's gone through a few renewals since then, currently serving:

• Domain: typesense.jquery.com
• notBefore: Tue, 21 Apr 2026 04:51:49 GMT
• notAfter: Thu, 21 May 2026 04:51:48 GMT (30 days)

Looking in the Fastly audit log, I find:

• 2026-04-21 05:50 TLS subscription certificate updated

Which suggests that they use the same 1-hour backdate. I couldn't find anywhere why Let's Encrypt settled on 1-hour specifically, but I did find various mentions of it in the IETF email archives, suggesting it was thought about and believed good enough. For example, RE: Acme - Short term certificates (20 July 2016)

The 1 hour backdate of Let's Encrypt seems to be enough to cover most clock skews. I didn't see any complaints about that specific issue yet, at least.

I didn't see anything in the Fastly audit logs to suggest that it was deployed at a different time from when it was generated (e.g. a day later). While it seems easy in theory for Certbot or Fastly to hold on to a new cert locally before activating it, I couldn't find any mention in the Certbot source code that this exists as a concept today, nor could I find any discussion or consideration of it in online in email archives and issue trackers for Let's Encrypt, Certbot, ACME clients, or the relevant IETF RFCs.

To test how GlobalSign's automation works, we'll need to wait for the first renewal. Which is one more reason to stick with our custom cert for the short-term until we've seen this cycle play out at least one there as well for comparison.

I'll run some cross-browser tests next week with the GlobalSign cert on https://bugs.jqueryui.com/.

[Krinkle]

While it seems easy in theory […] to hold on to a new cert locally before activating it, I couldn't find any mention […] or consideration of it in online […]

Okay, so coming full circle, at Wikimedia Foundation we switched Wikipedia from manual certs to automation with Acme in 2020 (T230687, using a redundant approach involving both Let's Encrypt and Google Trust Services as providers).

So what did we do? Our implemention, called acme-chief (docs), wraps the same python-acme library as Certbot, but also features a built-in option called staging_time (T213737). This is set to 7 days for end-user certificates.

[ryanaslett]

Was there a date we wanted the sectigo cert issued, or just ASAP for the new cert?

Were we able to inspect bugs.jqueryui.com to see how globalsign behaved on the k.sni cluster?