Simple test with curl

[matuskosut]

Hi,

is it possible to do end to end testing purely with curl? (Similar to this setup https://github.com/prometheus-community/PushProx/blob/master/end-to-end-test.sh)

For testing I run node-exporter, proxy and client on the same machine:

pushprox-proxy --web.listen-address=":8080" --log.level=debug

pushprox-client --fqdn test.local --proxy-url "http://192.168.1.10:8080/"

prometheus node exporter running on 192.168.1.10:9100

I assume that if this is supposed to work I need to use proxy parameter in some way.
I tried to run it like this:

curl --proxy http://test.local:8080 http://test.local:9100/metrics

Not sure what am I doing wrong, but all I get is errors so far (strconv.ParseFloat: parsing "": invalid syntax).

Proxy log:

ts=2024-04-17T15:06:15.895Z caller=coordinator.go:115 level=info msg=DoScrape scrape_id=cefb3ed1-3d68-4f78-9ab8-084c469c56a8 url=http://test.local:9100/metrics
ts=2024-04-17T15:06:15.895Z caller=main.go:154 level=info msg="Responded to /poll" url=http://test.local:9100/metrics scrape_id=cefb3ed1-3d68-4f78-9ab8-084c469c56a8
ts=2024-04-17T15:06:15.897Z caller=coordinator.go:136 level=info msg=WaitForScrapeInstruction fqdn=test.local
ts=2024-04-17T15:06:15.898Z caller=main.go:135 level=info msg="Got /push" scrape_id=cefb3ed1-3d68-4f78-9ab8-084c469c56a8
ts=2024-04-17T15:06:15.898Z caller=coordinator.go:168 level=info msg=ScrapeResult scrape_id=cefb3ed1-3d68-4f78-9ab8-084c469c56a8

Client log:

ts=2024-04-17T15:07:43.770Z caller=main.go:206 level=info msg="Got scrape request" scrape_id=ab8c0768-2d88-47e0-a7a6-7fc7d5a43529 url=http://test.local:9100/metrics
ts=2024-04-17T15:07:43.770Z caller=main.go:96 level=error err="strconv.ParseFloat: parsing \"\": invalid syntax"
ts=2024-04-17T15:07:43.772Z caller=main.go:108 level=info msg="Pushed failed scrape response"

Is this something that should work? Or is Prometheus proxy request impossible to simulate with curl?

[peekjef72]

It should work like a charm just add the header "X-Prometheus-Scrape-Timeout-Seconds"

curl -v --proxy http://test.local:8080 \
    -H "X-Prometheus-Scrape-Timeout-Seconds: 5" \
    http://test.local:9100/

I had always considered it a bug that the proxy requires this header (there is no default and no test cases if the header is not set), but we get used to it!

[peekjef72]

I had always considered it a bug that the proxy requires this header (there is no default and no test cases if the header is not set), but we get used to it!

In fact, after digging a little deeper into the scenario, it is not directly the pushprox_proxy which responds with the error message but the node_exporter: the node_exporter also expects the "X-Prometheus-Scrape-Timeout-Seconds" header and as it is not present and the case is obviously not foreseen, this triggers an error. The absence is foreseen in the pushprox and a value is set (--scrape.default-timeout=15s)
Maybe a fix should be to force the header with the default value when it is not provided by the original query.
I will try to make a PR with that solution.

[matuskosut]

@peekjef72 that quick fix would probably be quite an improvement here.

What is most confusing is that calling curl http://0.0.0.0:9100/metrics responds with metrics without any issue, so one is not expecting that header to be required.