Dynamo integration Branch

[biswapanda]

Cherry pick PRs for dynamo <> fastokens integration

1. deps: reduce hf-hub's transitive dependencies #5
2. feat: make pcre2 an optional cargo feature with docs #6

Details

1. deps: reduce hf-hub's transitive dependencies: #5

Replace the default native-tls/OpenSSL backend for hf-hub with the pure-Rust rustls-tls + ureq feature set, removing a large number of platform-specific and OpenSSL-related transitive dependencies.

This pr removes native-tls and openssl-sys from the dependency tree

Related failure on dynamo side -
https://github.com/ai-dynamo/dynamo/actions/runs/23097117690/job/67091453263?pr=7387

Related comment in dynamo PR: ai-dynamo/dynamo#7387 (comment)

Details

Issue: The fastokens crate (v0.1.0) declares hf-hub = "0.4.3" with default features, which pulls in native-tls and openssl-sys. Dynamo's deny.toml explicitly bans both crates (lines 63-64), causing cargo-deny to fail in CI.

The dependency chain is:

fastokens -> hf-hub (default features) -> ureq -> native-tls -> openssl-sys

Dynamo's own hf-hub declaration uses default-features = false with rustls-tls, but Cargo's feature unification merges both — so native-tls still gets enabled.

Before:

hf-hub = "0.4.3"

After:

hf-hub = { version = "0.4.3", default-features = false, features = ["rustls-tls", "ureq"] }

This disables the default features (which include native-tls, tokio, num_cpus, etc.) and opts into the pure-Rust TLS stack (rustls + ureq) instead. The result is:

native-tls, openssl-sys, hyper-tls, tokio-native-tls, and several other platform-specific crates are completely removed from the dependency tree
The cargo-deny ban is no longer triggered
576 lines removed from Cargo.lock, 94 added — a significant dependency reduction overall

2. feat: make pcre2 an optional cargo feature with docs: PR - #6

Add [features] section to Cargo.toml with pcre2 as an optinal and default feature.

Gate all pcre2-related code in split.rs behind #[cfg(feature = pcre2)] so consumers can opt out with default-features = false.

Add feature-gated tests for both pcre2-on and fallback paths.

Document the feature flag in README.md and add CONTRIBUTING.md with build/test instructions.

Related Dynamo CI failure - https://github.com/ai-dynamo/dynamo/actions/runs/23101799392/job/67103786703

This will allow disabling strict pcre2 (libpcre2-dev) dependency -

=================================== FAILURES ===================================
_______________________ test_no_bundled_shared_libraries _______________________
tests/basic/test_wheel_contents.py:34: in test_no_bundled_shared_libraries
assert (
E AssertionError: Unexpected shared libraries bundled in ai-dynamo-runtime:
E ai_dynamo_runtime.libs/libpcre2-8-516f4c9d.so.0
E assert not ['ai_dynamo_runtime.libs/libpcre2-8-516f4c9d.so.0']
bundled_libs = ['ai_dynamo_runtime.libs/libpcre2-8-516f4c9d.so.0']
installed_files = [PackagePath('ai_dynamo_runtime-1.0.0.dist-info/INSTALLER'), PackagePath('ai_dynamo_runtime-1.0.0.dist-info/METADATA')...Path('ai_dynamo_runtime-1.0.0.dist-info/WHEEL'), PackagePath('ai_dynamo_runtime-1.0.0.dist-info/direct_url.json'), ...]