feat(http-transport-reqwest): allow tls options #7798
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| # opendal-http-transport-reqwest | ||
|
|
||
| Reqwest-based HTTP transport for [Apache OpenDAL](https://opendal.apache.org). | ||
|
|
||
| This crate provides `ReqwestTransport`, an implementation of OpenDAL's | ||
| `HttpTransport` trait backed by [reqwest](https://crates.io/crates/reqwest). | ||
|
|
||
| ## TLS configuration | ||
|
|
||
| When using Rustls, TLS configuration has two independent axes: | ||
|
|
||
| | Axis | What it decides | Options | | ||
| |------|----------------|---------| | ||
| | **Crypto provider** | Who performs the cryptographic operations (key exchange, symmetric ciphers, hashing) | `aws-lc-rs` (default in `rustls`/`webpki-roots`), `ring`, or any custom `CryptoProvider` | | ||
| | **Certificate verification** | How the server's TLS certificate chain is validated | Platform verifier (default in `rustls`), bundled Mozilla roots (`webpki-roots`), or custom | | ||
|
|
||
| The `native-tls` feature sidesteps both axes by delegating everything to | ||
| the OS TLS library (SChannel / Secure Transport / OpenSSL). | ||
|
|
||
| ### Feature matrix | ||
|
Member
There was a problem hiding this comment. The matrix here seems not reflect the two axes mentioned before. Where is ring and where is native certs?
Member
Author
There was a problem hiding this comment. I didn't include ring as a backend. The documentation gives information for readers to understand these concepts and crates. Do we want to provide ring as a choice for backend? Because users will have to compile their own binary. If they know that, they will be capable of building 50 line configurations. For us, what default features do we want to ship? I am only considering native certs.
|
||
|
|
||
| | Feature | Crypto provider | Certificate roots | Use when | | ||
| |---------|----------------|-------------------|----------| | ||
| | `native-tls` (default) | OS library | OS trust store | You want zero Rust-side TLS config | | ||
| | `rustls` | aws-lc-rs | Platform verifier | Pure-Rust TLS with OS trust store | | ||
| | `webpki-roots` | aws-lc-rs | Bundled Mozilla roots | Fully self-contained, no OS dependency | | ||
| | `rustls-no-provider` | **you provide** | **you provide** | BYO crypto (ring, FIPS module, etc.) | | ||
|
|
||
| ### Usage via the `opendal` facade crate | ||
|
|
||
| Most users depend on `opendal` rather than this crate directly. The facade | ||
| installs this transport when any `http-transport-reqwest-*` feature is enabled. | ||
|
|
||
| ```toml | ||
| # Default — reqwest transport with native-tls | ||
| opendal = { version = "0.57" } | ||
| ``` | ||
|
|
||
| To select a different TLS backend, disable default features and enable the | ||
| one you need: | ||
|
|
||
| ```toml | ||
| opendal = { version = "0.57", default-features = false, features = ["http-transport-reqwest-rustls"] } | ||
| ``` | ||
|
|
||
| ### Feature usage with `rustls` | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| opendal-http-transport-reqwest = { version = "0.57", default-features = false, features = ["rustls"] } | ||
| ``` | ||
|
|
||
| ```rust | ||
| use std::time::Duration; | ||
|
|
||
| use opendal_http_transport_reqwest::ReqwestTlsBackend; | ||
|
Member
There was a problem hiding this comment. I didn't expect we need to implement this. Is there a reason why we didn't just use things from reqwest?
Member
Author
There was a problem hiding this comment. This is mainly a consideration of what features do we want to compile and ship. e.g.:
Bindings will use this crate mostly so compilation size is also a concern. |
||
| use opendal_http_transport_reqwest::ReqwestTransport; | ||
| use opendal::HttpTransporter; | ||
|
|
||
| // You can configure reqwest dynamically and select a compiled TLS backend. | ||
| let tls_backend = "rustls".parse::<ReqwestTlsBackend>().unwrap(); | ||
| let transport = ReqwestTransport::builder() | ||
| .tls_backend(tls_backend) | ||
| .configure(|builder| builder.connect_timeout(Duration::from_secs(10))) | ||
| .build() | ||
| .unwrap(); | ||
| ``` | ||
|
|
||
| ### Bringing your own reqwest client | ||
|
|
||
| When you need full control over the TLS stack — custom `ClientConfig`, | ||
| client certificates, proxy settings, or connection pool tuning — build a | ||
| `reqwest::Client` yourself and wrap it: | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| opendal = { version = "0.57", default-features = false, features = [ | ||
| "services-s3", | ||
| "http-transport-reqwest-rustls-no-provider", | ||
| ] } | ||
| opendal-http-transport-reqwest = { version = "0.57", default-features = false, features = ["rustls-no-provider"] } | ||
| rustls = { version = "0.23", features = ["ring"], default-features = false } | ||
| webpki-roots = "1" | ||
| ``` | ||
|
|
||
| ```rust | ||
| use std::time::Duration; | ||
|
|
||
| use opendal::HttpTransporter; | ||
| use opendal::OperationContext; | ||
| use opendal_http_transport_reqwest::ReqwestTransport; | ||
|
|
||
| fn main() { | ||
| // 1. Configure your crypto provider and certificate roots. | ||
| let root_store = | ||
| rustls::RootCertStore::from_iter(webpki_roots::TLS_SERVER_ROOTS.iter().cloned()); | ||
|
|
||
| let tls_config = rustls::ClientConfig::builder_with_provider( | ||
| rustls::crypto::ring::default_provider().into(), | ||
| ) | ||
| .with_safe_default_protocol_versions() | ||
| .unwrap() | ||
| .with_root_certificates(root_store) | ||
| .with_no_client_auth(); | ||
|
|
||
| // 2. Build a reqwest client with your TLS config. | ||
| let client = reqwest::Client::builder() | ||
| .tls_backend_preconfigured(tls_config) | ||
| .connect_timeout(Duration::from_secs(10)) | ||
| .pool_max_idle_per_host(20) | ||
| .build() | ||
| .unwrap(); | ||
|
|
||
| // 3. Wrap it as a ReqwestTransport and attach to an operator. | ||
| let transport = HttpTransporter::new(ReqwestTransport::new(client)); | ||
|
|
||
| let op = opendal::Operator::via_iter("s3", [ | ||
| ("bucket".to_string(), "my-bucket".to_string()), | ||
| ("region".to_string(), "us-east-1".to_string()), | ||
| ]) | ||
| .expect("failed to build operator") | ||
| .with_context(OperationContext::new().with_http_transport(transport)); | ||
| } | ||
| ``` | ||
|
|
||
| This approach gives you complete ownership over TLS and client. | ||
|
|
||
| ## License and Trademarks | ||
|
|
||
| Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should keep them for deprecation