Permit HTTP redirects globally and add RFC references

[tianon]

• allow registries to redirect any request per RFC 9110 (section 15.4); status codes in this spec are those returned after redirects have been followed; clients MUST NOT forward Authorization headers across host boundaries

  • Closes Pulling a blob doesn't specify 307 as a valid response status code #397

• add 429 / Retry-After guidance per RFC 6585 (section 4) and RFC 9110 (section 10.2.3)

• note that RFC 9110 governs HTTP behaviors not otherwise specified

  • Closes Support pull/push redirect #299

• normalize all RFC citations to consistent rfc-editor.org URLs with section numbers in link text

See also:

• Add 307 as valid response for pulling blobs #398 (prior attempt at the 307 fix, closed without merging)
• spec: add missing TOOMANYREQUESTS (429) error-code #209 (added TOOMANYREQUESTS to the error table; this adds the RFC citations)
• Historical curiosity: Why did we delete half the spec, again? #443
• spec is missing details #446

[tianon]

Implementation evidence:

307:

• distribution v2.7.1 registry/storage/blobserver.go#L41 (server)
• cue-labs/oci ociregistry/ociserver/reader.go#L61 (server)
• containerd core/remotes/docker/resolver.go#L647 (client)
• regclient internal/reghttp/http.go#L803 (client)

429 / Retry-After:

• olareg olareg.go#L184 (server, emits Retry-After)
• regclient internal/reghttp/http.go#L663 (client, reads Retry-After)

[sudo-bmitch]

LGTM.

[mikebrow]

Suggested change

	Registries MAY respond to any request with a redirect per [RFC 9110 (section 15.4)](https://www.rfc-editor.org/rfc/rfc9110#section-15.4); clients MUST follow such redirects, and MUST NOT forward `Authorization` headers across host boundaries unless explicitly configured to do so.
	Registries MAY respond to any request with a redirect per [RFC 9110 (section 15.4)](https://www.rfc-editor.org/rfc/rfc9110#section-15.4); clients following such redirects MUST NOT forward `Authorization` headers across host boundaries unless explicitly configured to do so.

Don't think we should force clients to accept redirects, that feels like an optional consideration.

[tianon]

It's not "forcing", it's documenting reality.

Do you have a counter example of a client that doesn't follow redirects today? Any such client won't work against almost every single major registry, which is really compelling evidence for "this should be normative in the spec"

[mikebrow]

keyword .. "should be normative" not MUST follow.. should be noted that in some circles redirects are not normative.

Not aware of any clients that disallow redirects outside some restrictions on the number of redirects.

[mikebrow]

https://github.com/containerd/containerd/blob/main/core/remotes/docker/resolver.go#L649-L651

[tianon]

Limiting the number of redirects followed is totally reasonable and actually called out in the RFC itself:

A client SHOULD detect and intervene in cyclical redirections (i.e., "infinite" redirection loops).

Note: An earlier version of this specification recommended a maximum of five redirections ([RFC2068], Section 10.3). Content developers need to be aware that some clients might implement such a fixed limitation.

I'd be willing to soften to a SHOULD from a MUST, but removing the strong language is (IMO) not being honest about how this actually works in practice -- for example, any client that doesn't follow redirects at all will literally not work with any of the largest public registries. That's stronger ecosystem evidence than would be covered by just a recommendation.

[mikebrow]

Limiting the number of redirects followed is totally reasonable and actually called out in the RFC itself:

A client SHOULD detect and intervene in cyclical redirections (i.e., "infinite" redirection loops).

Note: An earlier version of this specification recommended a maximum of five redirections ([RFC2068], Section 10.3). Content developers need to be aware that some clients might implement such a fixed limitation.

I'd be willing to soften to a SHOULD from a MUST, but removing the strong language is (IMO) not being honest about how this actually works in practice -- for example, any client that doesn't follow redirects at all will literally not work with any of the largest public registries. That's stronger ecosystem evidence than would be covered by just a recommendation.

soften to SHOULD language SGTM

I think it would also be fair to mention the need for reasoned following of redirects to enable working with the preponderance of public registries. Another reasonable response might be to check redirect allowance config for certain regions / providers, or just to include an info/debug log of the redirects so as not to hide them from the user.. etc..