Skip to content

docs: document authentication for private remote stores via storage_options#4108

Open
deodharaditi wants to merge 2 commits into
zarr-developers:mainfrom
deodharaditi:docs-issue-2995
Open

docs: document authentication for private remote stores via storage_options#4108
deodharaditi wants to merge 2 commits into
zarr-developers:mainfrom
deodharaditi:docs-issue-2995

Conversation

@deodharaditi

Copy link
Copy Markdown

Summary

Closes #2995.

The storage user-guide only showed anonymous (anon=True) access examples for remote
stores, leaving users to guess which keys to put inside storage_options when accessing
a private S3 bucket or an S3-compatible service like MinIO. A common wrong guess — nesting
credentials inside client_kwargs — routes them to the underlying aiohttp/aiobotocore
ClientSession, which produces a cryptic TypeError: ClientSession._request() got an unexpected keyword argument 'secret'.

This PR adds a short Authentication subsection under ### Remote Store that:

  1. Shows how to pass key, secret, token, and endpoint_url as top-level keys in
    storage_options (they map directly to s3fs.S3FileSystem arguments, not to the inner
    client session).
  2. Explains with a !!! note admonition exactly why client_kwargs is the wrong place for
    credentials (aiohttp ClientSession does not accept them), so the connection between the
    error message and the fix is clear.
  3. Notes that credentials can be omitted if the standard AWS auto-discovery path is used
    (env vars, ~/.aws/credentials, instance/role profile).

The code example is a plain (non-executed) python block deliberately — the surrounding
examples use exec="true" (markdown-exec runs them at build time), but we cannot connect
to a private bucket in CI.

For reviewers

The main thing I'd like a second look at is whether the s3fs.S3FileSystem API description
is accurate and complete enough. I verified the top-level keys (key, secret, token,
endpoint_url) by inspecting the installed signature of s3fs v2026.4.0 in a local dev
environment, but I'd welcome a check from someone more familiar with the fsspec/s3fs
interaction. I'm confident the prose style matches the existing admonition usage in the doc,
and that the non-executed block won't break the CI build.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change
    and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings — N/A, docs-only change
  • Add docstrings and API docs for any new/modified user-facing classes and functions — N/A
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes) — N/A, no code changed

Add an "Authentication" subsection to the storage user guide explaining how to
pass credentials (key/secret/token/endpoint_url) through storage_options for
private S3 / MinIO buckets, and clarify that credentials are top-level keys, not
client_kwargs entries (nesting them there triggers the cryptic aiohttp
'unexpected keyword argument secret' error). Adds a towncrier news fragment.

Refs zarr-developers#2995
@deodharaditi

deodharaditi commented Jun 29, 2026

Copy link
Copy Markdown
Author

Hi, this is my first contribution. I've added an Authentication subsection to the storage guide to address #2995. Would appreciate a review when you have time!

@d-v-b

d-v-b commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@deodharaditi thanks for this addition to the docs. There's a check in our tests that's failing -- it's designed to ensure that every code example in the docs is annotated as runnable as part of the docs build and / or checkable as part of our doctests pass. This example is probably neither, unless you want to go the extra mile to configure our test-only s3 backend to use credentials. So you need to annotate the code example with exec=false and a reason. See https://zarr.readthedocs.io/en/latest/contributing/#validating-code-blocks-exec-vs-test

@deodharaditi

Copy link
Copy Markdown
Author

Thanks for the heads-up and the pointer to the contributing guide! I've added exec="false" reason="requires private S3 credentials not available in the docs build environment" to the code block in 9be3d548. Let me know if the reason string needs adjustment or if there's anything else to address.

@d-v-b

d-v-b commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

thanks! I'm going to approve this, and ask @maxrjones to review, since he has a bit more experience using zarr + s3 than I do, and he might have some additional suggestions. Great work!

@d-v-b d-v-b requested a review from maxrjones June 29, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DOC: Show how to authenticate with fsspec's storage_options dict

2 participants