Add user agent metrics and logs

[rdettai-sk]

Description

Add user agent to metrics and logs.

For now, user agent is only propagated to the root search.

This PR also adds a request log to help troubleshoot.

How was this PR tested?

I tested that logs are ok in integrations tests.

[Copilot]

Pull request overview

This PR propagates the incoming HTTP User-Agent through REST/ES search endpoints into the root search request, and extends logging/metrics to include user-agent context for troubleshooting and observability.

Changes:

• Add user_agent to SearchRequest (proto + generated code) and propagate it from REST/Elasticsearch-compatible handlers into root_search.
• Add root-search metrics labeling by (normalized) user-agent and include user-agent in root-search logs.
• Replace warp log-based HTTP metrics with a Tower layer that also logs each request and counts cancelled requests.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
quickwit/quickwit-serve/src/search_api/rest_handler.rs	Extracts User-Agent header for /search routes and forwards it into SearchRequest; updates route tests accordingly.
quickwit/quickwit-serve/src/rest.rs	Introduces a Tower layer to record HTTP metrics (including cancellations) and adds a per-request log line.
quickwit/quickwit-serve/src/metrics.rs	Changes serve-side HTTP metrics label dimensions to match the new Tower layer labeling.
quickwit/quickwit-serve/src/elasticsearch_api/rest_handler.rs	Threads user_agent through ES-compatible count/search/msearch request building into SearchRequest.
quickwit/quickwit-serve/src/elasticsearch_api/filter.rs	Extracts User-Agent header for ES-compatible routes.
quickwit/quickwit-serve/Cargo.toml	Adds pin-project dependency for the new Tower metrics future with pinned drop.
quickwit/quickwit-search/src/root.rs	Propagates user-agent through scroll simplification, adds it to root-search logs, and passes it into metrics trackers.
quickwit/quickwit-search/src/metrics_trackers.rs	Adds user-agent to root-search metrics labels and introduces normalize_user_agent bucketing.
quickwit/quickwit-search/src/metrics.rs	Expands root-search metric vectors to include user_agent as a label.
quickwit/quickwit-rest-client/src/rest_client.rs	Sets a default User-Agent header (qw-rest-client) on outgoing requests.
quickwit/quickwit-proto/src/codegen/quickwit/quickwit.search.rs	Adds user_agent field to generated SearchRequest.
quickwit/quickwit-proto/protos/quickwit/search.proto	Adds optional user_agent field to SearchRequest proto.
quickwit/quickwit-cli/src/tool.rs	Updates call site to pass None for the new user_agent parameter.
quickwit/Cargo.lock	Locks pin-project dependency addition.

Comments suppressed due to low confidence (1)

quickwit/quickwit-serve/src/search_api/rest_handler.rs:547

• New user-agent extraction isn’t asserted in these route tests (it’s currently ignored as _). Since this PR adds UA propagation, consider adding a test that sets a User-Agent header and asserts the extracted third tuple element is Some(...) (and/or that it is forwarded into the built SearchRequest).

    #[tokio::test]
    async fn test_rest_search_api_route_post() {
        let rest_search_api_filter = search_post_filter();
        let (indexes, req, _) = warp::test::request()
            .method("POST")
            .path("/quickwit-demo-index/search")
            .json(&true)
            .body(r#"{"query": "*", "max_hits":10, "aggs": {"range":[]} }"#)
            .filter(&rest_search_api_filter)
            .await

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.