Fix stats search#3322
Open
tiborrr wants to merge 3 commits into
Open
Conversation
Split searchStats into full, general, and fields profiles so the dashboard can fetch lighter summaries. Extract field-stats queries into QItemFieldStats, add SearchStatsTest, and document the API in OpenAPI and Changelog. For large match sets, avoid loading item ids into memory and only aggregate money/numeric custom fields, which matches what the UI displays. Co-authored-by: Cursor <cursoragent@cursor.com>
Route dashboard stats boxes to general and fields profile endpoints and dedupe fetches through StatsCache so multiple boxes sharing a query only hit the backend once. Co-authored-by: Cursor <cursoragent@cursor.com>
The general stats profile still counted non-numeric custom fields via cvf.item_id IN (large item subselect), which took ~12s on ~900k items. Use a cf-first EXISTS probe instead and skip resolveStatsItemContext when searchStatsGeneral already has the item count. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this PR
Opening the dashboard was slow on my instance (~800k docs). Each box that shows search summaries triggered
POST /searchStats, and the dashboard fired several of those at once — often the same query more than once. On a database with a largecustom_field_valuetable, a single stats request could take 7–12 seconds even for a default empty query.Before — slowest query on dashboard load (
generalprofile,fieldCount):~11.7s on ~895k matching items (full table scan of
custom_field_valueagainst a hugeINsubselect).Solution
1. Split
searchStatsinto profilesIntroduce
statsProfileso callers fetch only what they need:fullgeneralfieldsPOST /sec/item/searchStats/{profile}(path > bodystatsProfile> defaultfull).general+fieldsinstead of repeated full stats;StatsCachededupes so boxes sharing a query hit the backend once per profile.POST /searchStatswithout a profile still returnsfull.2. Large-database field-stats optimizations
Three targeted fixes in
QItemFieldStats(verified on PostgreSQL with ~895k items):fieldsprofile ~12s aggregating text fields the UI never showsSearchStatsViewfilters tosum > 0anyway)general.fieldCount~12s viacvf IN (subselect)EXISTSprobe usingcustom_field_value.fieldindexAfter: fields aggregation for text-only collectives completes in <1ms;
fieldCountuses the fast EXISTS pattern instead of the 11sINsubselect.API / docs
StatsProfiletype (full|general|fields).fieldCountsemantics, deprecation note for?statsProfile=on POST.Tests
SearchStatsTest(H2): each profile omits/includes expected sections;general.fieldCountcounts non-numeric field definitions with values.Manual verification (PostgreSQL)
CI uses H2 only. On a large DB (~900k items):
/searchStats/generaland/searchStats/fields, not duplicate full stats calls.pg_stat_statements: no multi-secondCOUNT(DISTINCT cf.id) … cvf.item_id IN (SELECT …)on dashboard load.