fix: migrate rquest to wreq + engineering review hardening (17 fixes)#6
Open
Zireael wants to merge 7 commits intopaperfoot:masterfrom
Open
fix: migrate rquest to wreq + engineering review hardening (17 fixes)#6Zireael wants to merge 7 commits intopaperfoot:masterfrom
Zireael wants to merge 7 commits intopaperfoot:masterfrom
Conversation
…ability Config, cache, timeout, and rejection-diagnostics hardening: - config: type-numeric writes for settings.timeout/count (hbq1), legacy quoted-numeric coercion (hbq2) - cache: skip caching all-provider-failed and degraded-empty responses (hbq3) - engine: unified timeout budget from settings.timeout (hbq5), remove special-mode literals (hbq6), provider count clamping for Brave cap (hbq7) - types/errors: structured providers_failed_detail taxonomy with cause/action/ signature fields, backward-compatible (hbq4, hbq13, hbq14) - providers: spawn_blocking extraction offload in stealth/browserless (hbq9), Exa NUM_RESULTS_EXCEEDED and Jina Cloudflare-1010/Browserless auth-mode rejection classification (hbq13) - main/logging: env-driven tracing subscriber with quiet default, structured reliability events (hbq8) - README: troubleshooting rejection diagnostics section (hbq15) - clippy cleanup: unused vars, range pattern, test module ordering - build: fix backon v1 retry callback (use .notify() on retry future)
The rquest HTTP client crate has been renamed to wreq, and the old packages will be yanked. This commit migrates all references: - Cargo.toml: rquest -> wreq v5, rquest-util -> wreq-util v2 - src/errors.rs: SearchError::Rquest -> SearchError::Wreq - src/providers/stealth.rs: imports and types updated - src/engine.rs: error variant match updated - .github/workflows/release.yml: comment updated Uses wreq v5.3.0 + wreq-util v2.2.6 (both stable), which provide the same v5 API as rquest — purely a crate rename, no behavior change. Closes paperfoot#4
Cherry-picked from andrey-golovko/search-cli fix/linux-build branch. - Remove readability crate (pulled reqwest with native-tls/OpenSSL) - Replace readability extraction with tl-based title + tag-stripping fallback - Keep spawn_blocking offload for extraction from reliability hardening PR - self_update: default-features = false to avoid native-tls
Cherry-picked from mouse-value-add/search-cli feat/you-search-provider. - New You.com provider with general search and news search - Freshness mapping, domain include/exclude filters - Auth, API status, and rate-limit error handling - Wired into engine routing, config, CLI, and docs
Bug fixes: - browserless extract_text_simple now skips <script>/<style> content - Extract/Scrape chain uses shared deadline to prevent timeout overflow - stealth provider maps HTTP errors as SearchError::Api (not Config) - finalize_response() wired into execute_search return path - retry_request .when() now also matches SearchError::Wreq errors - Cross-platform home_dir() resolves /home/zir on Unix, %USERPROFILE% on Windows - Cache write failures now log warnings instead of silent ignore DRY refactoring: - Shared augment_query() extracted to providers/mod.rs (3 copies removed) - Shared map_freshness() extracted to types.rs (2 copies removed) - Shared extract_title() extracted to providers/mod.rs (2 copies removed) - Shared epoch_days_to_date() extracted to utils.rs (2 copies removed) - execute_special refactored with try_provider/try_provider_remaining helpers (~160 lines of boilerplate eliminated) Cleanup: - Removed unused Provider::timeout() trait method + 12 provider impls - Removed build_providers() call from execute_special (avoids unused instances) Enhancements: - Cache file eviction on startup removes expired q_*.json files Test coverage: - 13 classify tests (social/news/academic/scholar/patents/people/extract/ similar/images/places/general/priority + 12 SE-focused) - 3 engine tests (normalize_url, provider_allowed) - 4 browserless extract_text_simple tests (script/style skip) - 5 you.com provider tests (JSON deserialization) - 8 cache logic tests (should_cache_query_response, path determinism) - 5 additional normalize_url edge cases 95 tests pass, 0 clippy warnings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates from
rquestto stablewreq(v5) and applies 17 engineering-review fixes covering reliability bugs, DRY refactoring, dead-code cleanup, cache eviction, and test coverage.Migration
rquest(unstable, BoringSSL conflicts) towreq(stable v5, OpenSSL-compatible). Resolves the BoringSSL vs OpenSSL linking conflict that blocked builds on some platforms.Bug Fixes (6)
browserless::extract_text_simplenow skips<script>/<style>contentproviders/browserless.rsengine.rsSearchError::Apiinstead ofSearchError::Configproviders/stealth.rsfinalize_response()wired intoexecute_searchreturn path (result_count was 0 untilrun()fixed it)engine.rsretry_request.when()now also matchesSearchError::Wreqerrorsproviders/mod.rshome_dir()— resolves$HOMEon Unix,%USERPROFILE%on Windowsconfig.rs,cache.rs,logging.rs,cli.rstracing::warn!instead of silentlet _ =cache.rsDRY Refactoring (5)
augment_query()providers/mod.rsmap_freshness()types.rsextract_title()providers/mod.rsepoch_days_to_date()utils.rs(new)try_provider()/try_provider_remaining()engine.rsCleanup (2)
Provider::timeout()trait method + 12 provider impl blocks + 11 unusedDurationimportsbuild_providers()call fromexecute_special— now only constructs needed providers per modeEnhancements (1)
q_*.jsonfiles, removes expired (≥5 min old) and unparseable entriesTest Coverage
59 unit tests + 36 integration tests = 95 total (was 46+36=82)
classify.rsengine.rsnormalize_urledge cases (trailing slash, query params, fragments),provider_allowedwith/without filterbrowserless.rsextract_text_simplescript/style skipping, visible text preservationyou.rsYouResponsedeserialization (hits-only, news-only, empty, optional fields)cache.rsshould_cache_query_response(success, all_providers_failed, degraded-empty, empty-no-failures), path determinism, mode sensitivity, case insensitivity, q_ prefixVerification