docs: document authentication for private remote stores via storage_options#4108
docs: document authentication for private remote stores via storage_options#4108deodharaditi wants to merge 2 commits into
Conversation
Add an "Authentication" subsection to the storage user guide explaining how to pass credentials (key/secret/token/endpoint_url) through storage_options for private S3 / MinIO buckets, and clarify that credentials are top-level keys, not client_kwargs entries (nesting them there triggers the cryptic aiohttp 'unexpected keyword argument secret' error). Adds a towncrier news fragment. Refs zarr-developers#2995
|
Hi, this is my first contribution. I've added an Authentication subsection to the storage guide to address #2995. Would appreciate a review when you have time! |
|
@deodharaditi thanks for this addition to the docs. There's a check in our tests that's failing -- it's designed to ensure that every code example in the docs is annotated as runnable as part of the docs build and / or checkable as part of our doctests pass. This example is probably neither, unless you want to go the extra mile to configure our test-only s3 backend to use credentials. So you need to annotate the code example with exec=false and a reason. See https://zarr.readthedocs.io/en/latest/contributing/#validating-code-blocks-exec-vs-test |
|
Thanks for the heads-up and the pointer to the contributing guide! I've added exec="false" reason="requires private S3 credentials not available in the docs build environment" to the code block in 9be3d548. Let me know if the reason string needs adjustment or if there's anything else to address. |
|
thanks! I'm going to approve this, and ask @maxrjones to review, since he has a bit more experience using zarr + s3 than I do, and he might have some additional suggestions. Great work! |
Summary
Closes #2995.
The storage user-guide only showed anonymous (
anon=True) access examples for remotestores, leaving users to guess which keys to put inside
storage_optionswhen accessinga private S3 bucket or an S3-compatible service like MinIO. A common wrong guess — nesting
credentials inside
client_kwargs— routes them to the underlyingaiohttp/aiobotocoreClientSession, which produces a crypticTypeError: ClientSession._request() got an unexpected keyword argument 'secret'.This PR adds a short Authentication subsection under
### Remote Storethat:key,secret,token, andendpoint_urlas top-level keys instorage_options(they map directly tos3fs.S3FileSystemarguments, not to the innerclient session).
!!! noteadmonition exactly whyclient_kwargsis the wrong place forcredentials (aiohttp
ClientSessiondoes not accept them), so the connection between theerror message and the fix is clear.
(env vars,
~/.aws/credentials, instance/role profile).The code example is a plain (non-executed)
pythonblock deliberately — the surroundingexamples use
exec="true"(markdown-exec runs them at build time), but we cannot connectto a private bucket in CI.
For reviewers
The main thing I'd like a second look at is whether the
s3fs.S3FileSystemAPI descriptionis accurate and complete enough. I verified the top-level keys (
key,secret,token,endpoint_url) by inspecting the installed signature ofs3fsv2026.4.0 in a local devenvironment, but I'd welcome a check from someone more familiar with the fsspec/s3fs
interaction. I'm confident the prose style matches the existing admonition usage in the doc,
and that the non-executed block won't break the CI build.
Author attestation
and can explain why each is correct.
TODO
Add unit tests and/or doctests in docstrings— N/A, docs-only changeAdd docstrings and API docs for any new/modified user-facing classes and functions— N/Adocs/user-guide/*.mdchanges/Test coverage is 100% (Codecov passes)— N/A, no code changed