Skip to content

Conversation

@eeldaly
Copy link
Contributor

@eeldaly eeldaly commented Jan 14, 2026

What this PR does:
Add an active api tracker to print api requests in a querier on OOMKill / crash

Example log:

"These API calls didn't finish in prometheus' last run:"apis="[
{\"Method\":\"GET\",
\"Path\":\"/prometheus/api/v1/series\",
\"User-Agent\":\"PostmanRuntime/7.49.1\",
\"X-Request-ID\":\"\",
\"X-Scope-OrgID\":\"12345\",
\"end\":\"\",
\"limit\":\"100\",
\"matches\"\"metric_0metric_0metric_0m,metric_1metric_1metric_1m,metric_2metric_2metric_2m,metric_3metric_3metric\",
\"number-of-matches\":5,
\"start\":\"\",
\"timestamp_sec\":1768349758
}]"

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
Signed-off-by: Essam Eldaly <eeldaly@amazon.com>
"github.com/edsrzf/mmap-go"
)

type APITracker struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this tracker generic? Move to a better package maybe in utils. I feel it can be used in query frontend and more components as well. Maybe a generic name can be RequestsTracker. Make sure when inserting entries we can insert arbitrary entries?

entryMap["timestamp_sec"] = time.Now().Unix()
entryMap["Path"] = r.URL.Path
entryMap["Method"] = r.Method
entryMap["User-Agent"] = r.Header.Get("User-Agent")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this?

entryMap["Method"] = r.Method
entryMap["User-Agent"] = r.Header.Get("User-Agent")
entryMap["X-Scope-OrgID"] = r.Header.Get("X-Scope-OrgID")
entryMap["X-Request-ID"] = r.Header.Get("X-Request-ID")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't hardcode the request ID. It is extracted from the context

entryMap["Path"] = r.URL.Path
entryMap["Method"] = r.Method
entryMap["User-Agent"] = r.Header.Get("User-Agent")
entryMap["X-Scope-OrgID"] = r.Header.Get("X-Scope-OrgID")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract user ID from context. There is helper function to do it

entryMap["X-Request-ID"] = r.Header.Get("X-Request-ID")
entryMap["limit"] = r.URL.Query().Get("limit")
entryMap["start"] = r.URL.Query().Get("start")
entryMap["end"] = r.URL.Query().Get("end")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this work for http POST?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants