Skip to content

feat(insights): full insight history context for investigations #460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
izadoesdev merged 1 commit into from
May 28, 2026
Merged

feat(insights): full insight history context for investigations #460

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions apps/insights/src/generation.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ import {
buildSystemPrompt,
fetchDismissedPatterns,
fetchRecentAnnotations,
fetchRecentInsightsForPrompt,
fetchInsightHistory,
formatOrgWebsitesContext,
type OrgWebsiteRow,
} from "./prompts";
Expand Down Expand Up @@ -295,10 +295,10 @@ async function analyzeWebsite(params: {

const investigationMode = enrichedSignals.length > 0;

const [annotationContext, recentInsightsBlock, siteContext, dismissedBlock] =
const [annotationContext, historyBlock, siteContext, dismissedBlock] =
await Promise.all([
fetchRecentAnnotations(params.websiteId, params.config),
fetchRecentInsightsForPrompt(
fetchInsightHistory(
params.organizationId,
params.websiteId,
params.config
Expand All @@ -321,14 +321,14 @@ async function analyzeWebsite(params: {
githubRepo: params.githubRepo,
period: params.period,
timezone: params.config.timezone,
recentInsightsBlock,
historyBlock,
annotationContext,
dismissedBlock,
orgContext,
siteContext: siteBlock,
})
: `Analyze ${params.domain} (${currentRange.from} to ${currentRange.to} vs ${previousRange.from} to ${previousRange.to}, ${params.config.timezone}). Use web_metrics with period="both" to compare periods efficiently.${siteBlock}
${orgContext}${annotationContext}${recentInsightsBlock}${dismissedBlock}`;
${orgContext}${annotationContext}${historyBlock}${dismissedBlock}`;

const { tools: analyticsTools } = createInsightsAgentTools({
websiteId: params.websiteId,
Expand Down
53 changes: 45 additions & 8 deletions apps/insights/src/prompts.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ export async function fetchDismissedPatterns(
return `\n\nInsights users marked as NOT helpful (avoid similar narratives):\n${lines.join("\n")}`;
}

export async function fetchRecentInsightsForPrompt(
export async function fetchInsightHistory(
organizationId: string,
websiteId: string,
config: InsightGenerationConfigSnapshot
Expand All @@ -95,8 +95,14 @@ export async function fetchRecentInsightsForPrompt(
const rows = await db
.select({
title: analyticsInsights.title,
description: analyticsInsights.description,
type: analyticsInsights.type,
severity: analyticsInsights.severity,
rootCause: analyticsInsights.rootCause,
changePercent: analyticsInsights.changePercent,
subjectKey: analyticsInsights.subjectKey,
createdAt: analyticsInsights.createdAt,
runId: analyticsInsights.runId,
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 runId is selected in the query but is never referenced in the formatting loop below. It adds an unnecessary column to the fetch and a field to every returned row without contributing to the prompt output.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

})
.from(analyticsInsights)
.where(
Expand All @@ -113,12 +119,43 @@ export async function fetchRecentInsightsForPrompt(
return "";
}

const lines = rows.map(
(row) =>
`- [${row.type}] ${row.title} (${dayjs(row.createdAt).format("YYYY-MM-DD")})`
);
const subjectCounts = new Map<string, number>();
for (const row of rows) {
subjectCounts.set(
row.subjectKey,
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot May 28, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Deduping history by raw subjectKey collapses all empty keys into one recurring issue, which can hide distinct findings and misstate recurrence.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At apps/insights/src/prompts.ts, line 125:

<comment>Deduping history by raw `subjectKey` collapses all empty keys into one recurring issue, which can hide distinct findings and misstate recurrence.</comment>

<file context>
@@ -113,12 +119,43 @@ export async function fetchRecentInsightsForPrompt(
+	const subjectCounts = new Map<string, number>();
+	for (const row of rows) {
+		subjectCounts.set(
+			row.subjectKey,
+			(subjectCounts.get(row.subjectKey) ?? 0) + 1
+		);
</file context>
Fix with Cubic

(subjectCounts.get(row.subjectKey) ?? 0) + 1
);
}

const seen = new Set<string>();
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot May 28, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Deduplication by subjectKey is applied in-memory after the database LIMIT caps the raw row count. If one subject recurs across many runs, it can consume most of the limited slots, resulting in far fewer unique subjects than intended (e.g., 12 raw rows → 3 unique entries). Consider either increasing the fetch limit to account for expected duplication, or deduplicating at the query level (e.g., DISTINCT ON (subject_key)) so the limit governs unique subjects.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At apps/insights/src/prompts.ts, line 130:

<comment>Deduplication by `subjectKey` is applied in-memory after the database LIMIT caps the raw row count. If one subject recurs across many runs, it can consume most of the limited slots, resulting in far fewer unique subjects than intended (e.g., 12 raw rows → 3 unique entries). Consider either increasing the fetch limit to account for expected duplication, or deduplicating at the query level (e.g., `DISTINCT ON (subject_key)`) so the limit governs unique subjects.</comment>

<file context>
@@ -113,12 +119,43 @@ export async function fetchRecentInsightsForPrompt(
+		);
+	}
+
+	const seen = new Set<string>();
+	const lines: string[] = [];
+	for (const row of rows) {
</file context>
Fix with Cubic

const lines: string[] = [];
for (const row of rows) {
if (seen.has(row.subjectKey)) {
continue;
}
seen.add(row.subjectKey);

const date = dayjs(row.createdAt).format("YYYY-MM-DD");
const recurrence = subjectCounts.get(row.subjectKey) ?? 1;
const recurring = recurrence > 1 ? ` (reported ${recurrence}x)` : "";
const change =
row.changePercent === null
? ""
: ` ${row.changePercent > 0 ? "+" : ""}${Math.round(row.changePercent)}%`;

lines.push(
`- [${row.severity}] ${row.title}${change}${recurring} (${date})`
);
if (row.description) {
lines.push(` ${row.description.slice(0, 150)}`);
}
if (row.rootCause) {
lines.push(` Cause: ${row.rootCause.slice(0, 100)}`);
}
}

Comment on lines 95 to 156
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The LIMIT(12) is applied to the raw rows before deduplication by subjectKey. In a site where a single subject fires on most daily runs (e.g. a persistent checkout-conversion drop), that one subject could consume 10 of the 12 fetched rows, leaving only 2 slots for all other subjects. The deduplication then outputs just 3 unique entries instead of up to 12. The agent's history context for resolving/persisting issues is silently compressed without any indication. Consider increasing RECENT_INSIGHTS_PROMPT_LIMIT or deduplicating at the query level (e.g. DISTINCT ON (subject_key)) so the limit governs unique subjects rather than raw rows.

return `\n\nRecently reported (avoid repeating unless materially changed):\n${lines.join("\n")}`;
return `\n\nPrevious findings for this site (compare against current data — note what resolved, worsened, or persists):
${lines.join("\n")}`;
}

export interface OrgWebsiteRow {
Expand Down Expand Up @@ -252,7 +289,7 @@ export function buildInvestigationPrompt(
githubRepo?: { owner: string; repo: string };
orgContext: string;
period: WeekOverWeekPeriod;
recentInsightsBlock: string;
historyBlock: string;
siteContext: string;
timezone: string;
}
Expand Down Expand Up @@ -285,5 +322,5 @@ ${githubInstruction}
8. Emit findings via emit_insight as you go.

summary_metrics is the canonical source for headline numbers.
${params.orgContext}${params.annotationContext}${params.recentInsightsBlock}${params.dismissedBlock}`;
${params.orgContext}${params.annotationContext}${params.historyBlock}${params.dismissedBlock}`;
}
22 changes: 11 additions & 11 deletions packages/evals/ui/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -794,16 +794,16 @@ <h2 class="panel-title">Latest model board</h2>
const id = escapeHtml(c.id);
const cost = (c.metrics?.costUsd || 0) + (c.metrics?.judgeCostUsd || 0);
return `<tr>
<td><div class="case-id" title="${id}">${id}</div></td>
<td><span class="pill cat">${escapeHtml(c.category || "case")}</span></td>
<td><span class="pill ${c.passed ? "pass" : "fail"}">${c.passed ? "Pass" : "Fail"}</span></td>
<td class="num ${scoreTone(c.scores?.tool_routing ?? 0)}">${c.scores?.tool_routing ?? "--"}</td>
<td class="num ${scoreTone(c.scores?.quality ?? 0)}">${c.scores?.quality ?? "--"}</td>
<td class="num">${((c.metrics?.latencyMs || 0) / 1000).toFixed(1)}s</td>
<td class="num">${c.metrics?.steps ?? "--"}</td>
<td class="num">${money(cost)}</td>
<td><button class="row-action" data-open-case="${id}" type="button">Inspect</button></td>
</tr><tr><td colspan="9"><div class="detail ${openId === c.id ? "open" : ""}" id="detail-${id}">${detail(c)}</div></td></tr>`;
<td><div class="case-id" title="${id}">${id}</div></td>
<td><span class="pill cat">${escapeHtml(c.category || "case")}</span></td>
<td><span class="pill ${c.passed ? "pass" : "fail"}">${c.passed ? "Pass" : "Fail"}</span></td>
<td class="num ${scoreTone(c.scores?.tool_routing ?? 0)}">${c.scores?.tool_routing ?? "--"}</td>
<td class="num ${scoreTone(c.scores?.quality ?? 0)}">${c.scores?.quality ?? "--"}</td>
<td class="num">${((c.metrics?.latencyMs || 0) / 1000).toFixed(1)}s</td>
<td class="num">${c.metrics?.steps ?? "--"}</td>
<td class="num">${money(cost)}</td>
<td><button class="row-action" data-open-case="${id}" type="button">Inspect</button></td>
</tr><tr><td colspan="9"><div class="detail ${openId === c.id ? "open" : ""}" id="detail-${id}">${detail(c)}</div></td></tr>`;
}

function detail(c) {
Expand All @@ -815,7 +815,7 @@ <h2 class="panel-title">Latest model board</h2>
.map((t) => `<span class="pill cat">${escapeHtml(t)}</span>`)
.join("") || '<span class="sub">No tools called</span>';
return `<div class="box"><h3>Response</h3><pre>${escapeHtml(c.response || "No response captured.")}</pre></div>
<div class="box"><h3>Failures</h3><ul class="fail-list">${failures}</ul><h3 style="margin-top:14px">Tools</h3><div class="tools">${tools}</div></div>`;
<div class="box"><h3>Failures</h3><ul class="fail-list">${failures}</ul><h3 style="margin-top:14px">Tools</h3><div class="tools">${tools}</div></div>`;
}

function toggle(id) {
Expand Down
Loading