diff --git a/README.md b/README.md index 78a27bc..21f7f04 100644 --- a/README.md +++ b/README.md @@ -117,11 +117,13 @@ Telemetry is disabled unless you explicitly enable it. madar telemetry status madar telemetry enable madar telemetry disable +madar telemetry clear +madar telemetry report MADAR_ENABLE_TELEMETRY=1 madar generate . ``` -The current telemetry model is local-first and source-safe. It records coarse success events for `install`, `generate`, `pack`, and `compare`, plus version, OS, optional install target, and optional repo-size bucket. It does **not** record prompt text, answer text, source paths, or source content. Full controls: [`docs/telemetry.md`](https://github.com/mohanagy/madar/blob/main/docs/telemetry.md). +The current telemetry model is local-first and source-safe. It records coarse funnel events for `install`, `generate`, `pack`, `prompt`, MCP `context_pack`, `doctor`, `status`, and `compare`, plus command stage, version, OS, Node major version, and optional coarse buckets such as agent target, repo size, graph size, SPI enabled, failure bucket, and status bucket. It does **not** record prompt text, answer text, source paths, source content, or repository names. Full controls: [`docs/telemetry.md`](https://github.com/mohanagy/madar/blob/main/docs/telemetry.md). --- diff --git a/docs/reference/cli-and-mcp.md b/docs/reference/cli-and-mcp.md index 4f9c6e2..cead8b7 100644 --- a/docs/reference/cli-and-mcp.md +++ b/docs/reference/cli-and-mcp.md @@ -96,6 +96,8 @@ madar compare "How does auth work?" --exec '...' --yes madar compare "How does auth work?" --baseline-mode pack_only --exec '...' --yes madar telemetry enable madar telemetry status +madar telemetry clear +madar telemetry report madar time-travel main HEAD --view risk madar federate frontend/graph.json backend/graph.json madar --help diff --git a/docs/telemetry.md b/docs/telemetry.md index 23a1ba0..4636a41 100644 --- a/docs/telemetry.md +++ b/docs/telemetry.md @@ -10,8 +10,13 @@ Telemetry is **disabled by default**. No event is recorded unless you explicitly madar telemetry status madar telemetry enable madar telemetry disable +madar telemetry clear +madar telemetry report [spool.json ...] ``` +- `madar telemetry clear` deletes the local bounded event spool but keeps your persisted opt-in preference unchanged. +- `madar telemetry report [spool.json ...]` prints a local anonymized funnel summary from the current spool plus any extra spool files you pass in. + Environment overrides: - `MADAR_ENABLE_TELEMETRY=1` — enable telemetry for the current command without changing the persisted preference. @@ -21,21 +26,23 @@ Environment overrides: ## What is collected -The current implementation records only these coarse success events: - -- `install_success` -- `generate_success` -- `pack_success` -- `compare_success` - -Each stored event can include: +Each stored event includes these core fields: -- `event` +- `command` +- `stage` - `recorded_at` - `version` - `os` -- `install_platform` (only for install flows) -- `repo_size_bucket` (only for repo-scoped flows such as `generate`, `pack`, and `compare`) +- `node_major` + +Optional coarse fields are added only when they help explain adoption drop-off without revealing source-sensitive data: + +- `agent_target` — install target such as `claude`, `cursor`, `copilot`, `gemini`, `aider`, `codex`, or `opencode` +- `repo_size_bucket` — coarse file-count bucket +- `graph_size_bucket` — coarse node-count bucket +- `spi_enabled` — whether `madar generate` ran with `--use-spi` +- `failure_bucket` — coarse actionable category such as `usage_error`, `invalid_params`, `missing_graph`, `stale_graph`, `stale_context`, `tool_profile`, `unsupported_corpus`, `install_error`, or `unknown` +- `status_bucket` — coarse doctor/status outcome (`healthy` or `attention_needed`) `repo_size_bucket` is intentionally coarse: @@ -45,6 +52,26 @@ Each stored event can include: - `500-999` - `1000+` +`graph_size_bucket` is intentionally coarse: + +- `1-99` +- `100-499` +- `500-999` +- `1000-4999` +- `5000+` + +## Current tracked command surfaces + +When telemetry is enabled, Madar records source-safe funnel stages for: + +- install flows (`madar install --platform ...`, `madar install`) +- `madar generate` (`started`, `succeeded`, `failed`) +- `madar pack` (`succeeded`, `failed`) +- `madar prompt` (`succeeded`, `failed`) +- MCP `context_pack` (`succeeded`, `failed`) +- `madar doctor` and `madar status` (`succeeded`, `failed`, plus `status_bucket`) +- `madar compare` (`succeeded`, `failed`) + ## What is excluded Madar does **not** record: @@ -53,6 +80,7 @@ Madar does **not** record: - answer text - source paths - source content +- repository name - raw snippets - full file counts - graph contents @@ -62,16 +90,6 @@ Madar does **not** record: This release stores telemetry locally only: - persisted opt-in preference under the platform config directory -- bounded event spool under the platform cache directory +- bounded event spool under the platform cache directory (`schema_version: 2`) There is **no cloud upload by default** in this implementation. The goal of this issue is to define the opt-in event model, controls, and source-safe field contract without introducing mandatory network traffic. - -## Current tracked command surfaces - -When telemetry is enabled, Madar records a coarse success event after these CLI flows complete successfully: - -- `madar install --platform ` -- `madar install` -- `madar generate` -- `madar pack` -- `madar compare` diff --git a/src/cli/main.ts b/src/cli/main.ts index d275333..fcaf20f 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -8,7 +8,7 @@ import { BenchmarkReadinessError, NativeAgentInstallRequiredError, runCompareCom import { runContextPackCommand } from '../infrastructure/context-pack-command.js' import { runHandoffCommand } from '../infrastructure/handoff-command.js' import { runContextPromptCommand } from '../infrastructure/context-prompt-command.js' -import { runDoctorCommand, runStatusCommand } from '../infrastructure/doctor.js' +import { buildDoctorReport, runDoctorCommand, runStatusCommand } from '../infrastructure/doctor.js' import { runProofReportCommand, type ProofReportResult } from '../infrastructure/proof-report.js' import { runReviewCompareCommand } from '../infrastructure/review-compare.js' import { runTryCommand } from '../infrastructure/try-command.js' @@ -47,10 +47,15 @@ import { disableTelemetry, enableTelemetry, formatTelemetryStatus, + graphSizeBucketFromNodeCount, getTelemetryStatus, + readTelemetryReport, + clearTelemetry, recordTelemetryEvent as persistTelemetryEvent, repoSizeBucketFromFileCount, + type TelemetryFailureBucket, type TelemetryEventInput, + type TelemetryStatusBucket, } from '../shared/telemetry.js' import { getUpdateNotification } from '../shared/update-notifier.js' import { @@ -197,6 +202,10 @@ export interface CliDependencies { enableTelemetry?: () => string disableTelemetry?: () => string readTelemetryStatus?: () => string + clearTelemetry?: () => string + readTelemetryReport?: (spoolPaths?: string[]) => string + readDoctorTelemetryBucket?: (graphPath: string) => TelemetryStatusBucket + readStatusTelemetryBucket?: (graphPath: string) => TelemetryStatusBucket recordTelemetryEvent?: (event: TelemetryEventInput) => void } @@ -335,6 +344,10 @@ const DEFAULT_DEPENDENCIES: CliDependencies = { enableTelemetry: () => enableTelemetry(), disableTelemetry: () => disableTelemetry(), readTelemetryStatus: () => formatTelemetryStatus(getTelemetryStatus()), + clearTelemetry: () => clearTelemetry(), + readTelemetryReport: (spoolPaths) => readTelemetryReport({}, spoolPaths), + readDoctorTelemetryBucket: (graphPath) => buildDoctorReport({ graphPath }).healthy ? 'healthy' : 'attention_needed', + readStatusTelemetryBucket: (graphPath) => buildDoctorReport({ graphPath }).healthy ? 'healthy' : 'attention_needed', recordTelemetryEvent: (event) => { persistTelemetryEvent(event) }, @@ -357,6 +370,11 @@ function readInstalledVersionForTelemetry(dependencies: CliDependencies): string return readInstalledVersion() } +function readNodeMajorForTelemetry(): number { + const major = Number.parseInt(process.versions.node.split('.', 1)[0] ?? '', 10) + return Number.isInteger(major) && major > 0 ? major : 0 +} + function emitTelemetry(io: CliIO, dependencies: CliDependencies, buildEvent: () => TelemetryEventInput): void { if (!dependencies.recordTelemetryEvent) { return @@ -372,6 +390,41 @@ function repoSizeBucketForGraph(dependencies: CliDependencies, graphPath: string return repoSizeBucketFromFileCount(buildGraphSummary(dependencies.loadGraph(graphPath)).file_count) } +function graphSizeBucketForGraph(dependencies: CliDependencies, graphPath: string) { + return graphSizeBucketFromNodeCount(buildGraphSummary(dependencies.loadGraph(graphPath)).node_count) +} + +function telemetryBase(dependencies: CliDependencies) { + return { + version: readInstalledVersionForTelemetry(dependencies), + os: process.platform, + nodeMajor: readNodeMajorForTelemetry(), + } as const +} + +function classifyTelemetryFailure(error: unknown): TelemetryFailureBucket { + const message = messageFromError(error).toLowerCase() + if (message.includes('context_pack requires') || message.includes('invalid params')) { + return 'invalid_params' + } + if (message.includes('graph file not found') || message.includes('out/graph.json not found') || message.includes('graph.json not found')) { + return 'missing_graph' + } + if (message.includes('require_fresh_graph') || message.includes('non-fresh graph')) { + return 'stale_graph' + } + if (message.includes('require_fresh_context') || message.includes('stale selected context')) { + return 'stale_context' + } + if (message.includes('unsupported corpus')) { + return 'unsupported_corpus' + } + if (message.includes('install')) { + return 'install_error' + } + return error instanceof UsageError ? 'usage_error' : 'unknown' +} + async function confirmPaidCommand( commandName: string, warningMessage: string, @@ -544,7 +597,7 @@ export function formatHelp(binaryName = 'madar'): string { ' install install post-commit and post-checkout hooks', ' uninstall remove madar hook sections', ' status show whether madar hooks are installed', - ' telemetry manage opt-in source-safe local telemetry', + ' telemetry manage opt-in source-safe local telemetry', ' aider manage local AGENTS.md rules', ' claude [--profile core|full|strict] manage local CLAUDE.md madar rules', ' cursor [--profile core|full|strict] manage local Cursor madar rules', @@ -676,14 +729,31 @@ function formatExplainSummary(graph: ReturnType, label: string function handleAgentCommand(command: AgentPlatform, args: string[], io: CliIO, dependencies: CliDependencies): number { const options = parsePlatformActionArgs(command, args) if (options.action === 'install') { - io.log(dependencies.agentsInstall('.', command)) emitTelemetry(io, dependencies, () => ({ - event: 'install_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, - installPlatform: command, + command: 'install', + stage: 'started', + ...telemetryBase(dependencies), + agentTarget: command, })) - return 0 + try { + io.log(dependencies.agentsInstall('.', command)) + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'succeeded', + ...telemetryBase(dependencies), + agentTarget: command, + })) + return 0 + } catch (error) { + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'failed', + ...telemetryBase(dependencies), + agentTarget: command, + failureBucket: classifyTelemetryFailure(error), + })) + throw error + } } io.log(dependencies.agentsUninstall('.', command)) @@ -698,6 +768,7 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci return 0 } + let failureTelemetry: ((bucket: TelemetryFailureBucket) => TelemetryEventInput) | null = null try { if (command === '-v' || command === '--version') { const readInstalledVersion = dependencies.readInstalledVersion ?? (() => readPackageVersion(findPackageRoot())) @@ -718,6 +789,12 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'compare') { const options = parseCompareArgs(args) + failureTelemetry = (failureBucket) => ({ + command: 'compare', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + }) const confirm = async (message: string) => await dependencies.confirm(message) const warningMessage = compareWarningMessage(options) if (!options.yes) { @@ -736,9 +813,9 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci io.log(output) } emitTelemetry(io, dependencies, () => ({ - event: 'compare_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, + command: 'compare', + stage: 'succeeded', + ...telemetryBase(dependencies), repoSizeBucket: repoSizeBucketForGraph(dependencies, options.graphPath), })) return 0 @@ -779,13 +856,37 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'doctor') { const options = parseDoctorArgs(args, 'doctor') + failureTelemetry = (failureBucket) => ({ + command: 'doctor', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + }) io.log(dependencies.runDoctor(options.graphPath)) + emitTelemetry(io, dependencies, () => ({ + command: 'doctor', + stage: 'succeeded', + ...telemetryBase(dependencies), + statusBucket: (dependencies.readDoctorTelemetryBucket ?? ((graphPath: string) => buildDoctorReport({ graphPath }).healthy ? 'healthy' : 'attention_needed'))(options.graphPath), + })) return 0 } if (command === 'status') { const options = parseDoctorArgs(args, 'status') + failureTelemetry = (failureBucket) => ({ + command: 'status', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + }) io.log(dependencies.runStatus(options.graphPath)) + emitTelemetry(io, dependencies, () => ({ + command: 'status', + stage: 'succeeded', + ...telemetryBase(dependencies), + statusBucket: (dependencies.readStatusTelemetryBucket ?? ((graphPath: string) => buildDoctorReport({ graphPath }).healthy ? 'healthy' : 'attention_needed'))(options.graphPath), + })) return 0 } @@ -814,14 +915,20 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'pack') { const options = parsePackArgs(args) + failureTelemetry = (failureBucket) => ({ + command: 'pack', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + }) const output = await dependencies.runContextPack({ options, io }) if (output !== undefined) { io.log(output) } emitTelemetry(io, dependencies, () => ({ - event: 'pack_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, + command: 'pack', + stage: 'succeeded', + ...telemetryBase(dependencies), repoSizeBucket: repoSizeBucketForGraph(dependencies, options.graphPath), })) return 0 @@ -838,10 +945,22 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'prompt') { const options = parsePromptArgs(args) + failureTelemetry = (failureBucket) => ({ + command: 'prompt', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + }) const output = await dependencies.runContextPrompt({ options, io }) if (output !== undefined) { io.log(output) } + emitTelemetry(io, dependencies, () => ({ + command: 'prompt', + stage: 'succeeded', + ...telemetryBase(dependencies), + repoSizeBucket: repoSizeBucketForGraph(dependencies, options.graphPath), + })) return 0 } @@ -855,6 +974,14 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci io.log((dependencies.disableTelemetry ?? (() => disableTelemetry()))()) return 0 } + if (options.action === 'clear') { + io.log((dependencies.clearTelemetry ?? (() => clearTelemetry()))()) + return 0 + } + if (options.action === 'report') { + io.log((dependencies.readTelemetryReport ?? ((spoolPaths?: string[]) => readTelemetryReport({}, spoolPaths)))(options.spoolPaths)) + return 0 + } io.log((dependencies.readTelemetryStatus ?? (() => formatTelemetryStatus(getTelemetryStatus())))()) return 0 } @@ -862,6 +989,19 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'generate' || (command !== undefined && !isAgentPlatform(command) && isImplicitGenerateCommand(command))) { const generateArgs = command === 'generate' ? args : [command, ...args] const options = parseGenerateArgs(generateArgs) + failureTelemetry = (failureBucket) => ({ + command: 'generate', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + spiEnabled: options.useSpi, + }) + emitTelemetry(io, dependencies, () => ({ + command: 'generate', + stage: 'started', + ...telemetryBase(dependencies), + spiEnabled: options.useSpi, + })) const result = dependencies.generateGraph(options.path, { update: options.update, clusterOnly: options.clusterOnly, @@ -894,10 +1034,12 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci } emitTelemetry(io, dependencies, () => ({ - event: 'generate_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, + command: 'generate', + stage: 'succeeded', + ...telemetryBase(dependencies), repoSizeBucket: repoSizeBucketFromFileCount(result.totalFiles), + graphSizeBucket: graphSizeBucketFromNodeCount(result.nodeCount), + spiEnabled: options.useSpi, })) if (options.watch) { await dependencies.watchGraph(options.path, options.debounceSeconds, { @@ -1068,6 +1210,19 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'install') { const options = parseInstallArgs(args, defaultInstallPlatform()) + failureTelemetry = (failureBucket) => ({ + command: 'install', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + agentTarget: options.platform, + }) + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'started', + ...telemetryBase(dependencies), + agentTarget: options.platform, + })) if (options.platform === 'gemini') { io.log(dependencies.geminiInstall('.')) } else if (options.platform === 'cursor') { @@ -1076,10 +1231,10 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci io.log(dependencies.installSkill(options.platform)) } emitTelemetry(io, dependencies, () => ({ - event: 'install_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, - installPlatform: options.platform, + command: 'install', + stage: 'succeeded', + ...telemetryBase(dependencies), + agentTarget: options.platform, })) return 0 } @@ -1103,15 +1258,30 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (options.action === 'install' && !existsSync('out/graph.json')) { io.log("Warning: out/graph.json not found. Run 'madar generate .' first, then re-run this command.") } + if (options.action === 'install') { + failureTelemetry = (failureBucket) => ({ + command: 'install', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + agentTarget: 'claude', + }) + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'started', + ...telemetryBase(dependencies), + agentTarget: 'claude', + })) + } io.log(options.action === 'install' ? dependencies.claudeInstall('.', options.profile ? { profile: options.profile } : {}) : dependencies.claudeUninstall('.')) if (options.action === 'install') { emitTelemetry(io, dependencies, () => ({ - event: 'install_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, - installPlatform: 'claude', + command: 'install', + stage: 'succeeded', + ...telemetryBase(dependencies), + agentTarget: 'claude', })) } return 0 @@ -1122,15 +1292,30 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (options.action === 'install' && !existsSync('out/graph.json')) { io.log("Warning: out/graph.json not found. Run 'madar generate .' first, then re-run this command.") } + if (options.action === 'install') { + failureTelemetry = (failureBucket) => ({ + command: 'install', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + agentTarget: 'cursor', + }) + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'started', + ...telemetryBase(dependencies), + agentTarget: 'cursor', + })) + } io.log(options.action === 'install' ? dependencies.cursorInstall('.', options.profile ? { profile: options.profile } : {}) : dependencies.cursorUninstall('.')) if (options.action === 'install') { emitTelemetry(io, dependencies, () => ({ - event: 'install_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, - installPlatform: 'cursor', + command: 'install', + stage: 'succeeded', + ...telemetryBase(dependencies), + agentTarget: 'cursor', })) } return 0 @@ -1141,13 +1326,28 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (options.action === 'install' && !existsSync('out/graph.json')) { io.log("Warning: out/graph.json not found. Run 'madar generate .' first, then re-run this command.") } + if (options.action === 'install') { + failureTelemetry = (failureBucket) => ({ + command: 'install', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + agentTarget: 'gemini', + }) + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'started', + ...telemetryBase(dependencies), + agentTarget: 'gemini', + })) + } io.log(options.action === 'install' ? dependencies.geminiInstall('.', options.profile ? { profile: options.profile } : {}) : dependencies.geminiUninstall('.')) if (options.action === 'install') { emitTelemetry(io, dependencies, () => ({ - event: 'install_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, - installPlatform: 'gemini', + command: 'install', + stage: 'succeeded', + ...telemetryBase(dependencies), + agentTarget: 'gemini', })) } return 0 @@ -1156,16 +1356,29 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci if (command === 'copilot') { const options = parsePlatformActionArgs(command, args) if (options.action === 'install') { + failureTelemetry = (failureBucket) => ({ + command: 'install', + stage: 'failed', + ...telemetryBase(dependencies), + failureBucket, + agentTarget: 'copilot', + }) + emitTelemetry(io, dependencies, () => ({ + command: 'install', + stage: 'started', + ...telemetryBase(dependencies), + agentTarget: 'copilot', + })) if (!existsSync('out/graph.json')) { io.log("Warning: out/graph.json not found. Run 'madar generate .' first, then re-run this command.") } io.log(dependencies.installSkill('copilot')) io.log(dependencies.installCopilotMcp('.', options.profile ? { profile: options.profile } : {})) emitTelemetry(io, dependencies, () => ({ - event: 'install_success', - version: readInstalledVersionForTelemetry(dependencies), - os: process.platform, - installPlatform: 'copilot', + command: 'install', + stage: 'succeeded', + ...telemetryBase(dependencies), + agentTarget: 'copilot', })) } else { io.log(dependencies.uninstallCopilotMcp('.')) @@ -1182,6 +1395,9 @@ export async function executeCli(argv: string[], io: CliIO = console, dependenci io.error(`Run 'madar --help' for usage.`) return 1 } catch (error) { + if (failureTelemetry) { + emitTelemetry(io, dependencies, () => failureTelemetry!(classifyTelemetryFailure(error))) + } if (error instanceof UsageError) { io.error(error.message) return 2 diff --git a/src/cli/parser.ts b/src/cli/parser.ts index 8dc9b88..e321f01 100644 --- a/src/cli/parser.ts +++ b/src/cli/parser.ts @@ -221,9 +221,9 @@ export interface InstallCliOptions { platform: InstallPlatform } -export interface TelemetryCliOptions { - action: 'enable' | 'disable' | 'status' -} +export type TelemetryCliOptions = + | { action: 'enable' | 'disable' | 'status' | 'clear' } + | { action: 'report'; spoolPaths: string[] } const COMPARE_USAGE = 'Usage: madar compare [question] --exec TEMPLATE [--graph path] [--questions PATH] [--output-dir DIR] [--task TASK] [--baseline-mode MODE] [--per-arm-timeout S] [--validation-timeout S] [--heartbeat-interval-ms N] [--strict-madar-first] [--strict] [--allow-no-install] [--yes] [--limit N] [--why]' @@ -2176,14 +2176,17 @@ export function parseHookArgs(args: string[]): HookCliOptions { export function parseTelemetryArgs(args: string[]): TelemetryCliOptions { const action = args[0] - if (action === 'enable' || action === 'disable' || action === 'status') { + if (action === 'enable' || action === 'disable' || action === 'status' || action === 'clear') { if (args.length > 1) { - throw new UsageError('Usage: madar telemetry ') + throw new UsageError('Usage: madar telemetry ') } return { action } } + if (action === 'report') { + return { action, spoolPaths: args.slice(1) } + } - throw new UsageError('Usage: madar telemetry ') + throw new UsageError('Usage: madar telemetry ') } export function parseInstallArgs(args: string[], defaultPlatform: InstallPlatform): InstallCliOptions { diff --git a/src/infrastructure/doctor.ts b/src/infrastructure/doctor.ts index e91843a..fc28e98 100644 --- a/src/infrastructure/doctor.ts +++ b/src/infrastructure/doctor.ts @@ -42,7 +42,7 @@ interface GraphCheck { recommendation: string } -interface DoctorReport { +export interface DoctorReport { packageVersion: string graph: GraphCheck agents: AgentCheck[] @@ -416,7 +416,7 @@ function computeNextCommands(report: Omit 0 ? major : 0 +} + +function classifyToolTelemetryFailure(message: string, code: number): TelemetryFailureBucket { + const normalizedMessage = message.toLowerCase() + if (code === JSONRPC_INVALID_PARAMS) { + return 'invalid_params' + } + if (normalizedMessage.includes('require_fresh_graph') || normalizedMessage.includes('non-fresh graph')) { + return 'stale_graph' + } + if (normalizedMessage.includes('require_fresh_context') || normalizedMessage.includes('stale selected context')) { + return 'stale_context' + } + if (normalizedMessage.includes('tool') && normalizedMessage.includes('profile')) { + return 'tool_profile' + } + return 'unknown' +} + function stringParam(params: unknown, key: string): string | null { if (!params || typeof params !== 'object' || !(key in params)) { return null @@ -606,7 +640,7 @@ export function handleStdioRequest( `Tool '${toolName}' is not enabled in the active madar MCP tool profile. Default profile: core. Set MADAR_TOOL_PROFILE=full in your MCP server config (e.g. .mcp.json for Claude, .cursor/mcp.json for Cursor, .vscode/mcp.json for VS Code Copilot) to enable advanced tools.`, ) } - return handleToolCallRequest(id, graphPath, params, { + const response = handleToolCallRequest(id, graphPath, params, { ok, failure, textToolResult, @@ -684,6 +718,27 @@ export function handleStdioRequest( maxStdioHops: MAX_STDIO_HOPS, maxStdioTokenBudget: MAX_STDIO_TOKEN_BUDGET, }) + const recordContextPackTelemetry = (toolResponse: StdioResponse): StdioResponse => { + if (toolName === 'context_pack') { + try { + const summary = buildGraphSummary(loadGraphCached(graphPath)) + recordTelemetryEvent({ + command: 'context_pack', + stage: toolResponse.error ? 'failed' : 'succeeded', + version: readInstalledVersionForTelemetry(), + os: process.platform, + nodeMajor: readNodeMajorForTelemetry(), + repoSizeBucket: repoSizeBucketFromFileCount(summary.file_count), + graphSizeBucket: graphSizeBucketFromNodeCount(summary.node_count), + ...(toolResponse.error ? { failureBucket: classifyToolTelemetryFailure(toolResponse.error.message, toolResponse.error.code) } : {}), + }) + } catch { + // Telemetry is best-effort and must never break the MCP response path. + } + } + return toolResponse + } + return response instanceof Promise ? response.then(recordContextPackTelemetry) : recordContextPackTelemetry(response) } case 'ping': return ok(id, { ok: true }) diff --git a/src/shared/telemetry.ts b/src/shared/telemetry.ts index ef5910a..ecd1fc4 100644 --- a/src/shared/telemetry.ts +++ b/src/shared/telemetry.ts @@ -1,16 +1,57 @@ import { closeSync, existsSync, mkdirSync, openSync, readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs' import { homedir, platform } from 'node:os' -import { dirname, join } from 'node:path' - -export type TelemetryEventName = 'install_success' | 'generate_success' | 'pack_success' | 'compare_success' +import { dirname, join, resolve } from 'node:path' + +export type TelemetryCommand = + | 'install' + | 'generate' + | 'pack' + | 'prompt' + | 'context_pack' + | 'doctor' + | 'status' + | 'compare' + +export type TelemetryStage = 'started' | 'succeeded' | 'failed' export type TelemetryRepoSizeBucket = '1-24' | '25-99' | '100-499' | '500-999' | '1000+' +export type TelemetryGraphSizeBucket = '1-99' | '100-499' | '500-999' | '1000-4999' | '5000+' +export type TelemetryFailureBucket = + | 'usage_error' + | 'invalid_params' + | 'missing_graph' + | 'stale_graph' + | 'stale_context' + | 'tool_profile' + | 'unsupported_corpus' + | 'install_error' + | 'unknown' +export type TelemetryStatusBucket = 'healthy' | 'attention_needed' +export type TelemetryAgentTarget = + | 'claude' + | 'cursor' + | 'codex' + | 'copilot' + | 'gemini' + | 'aider' + | 'opencode' + | 'windows' + | 'claw' + | 'droid' + | 'trae' + | 'trae-cn' export interface TelemetryEventInput { - event: TelemetryEventName + command: TelemetryCommand + stage: TelemetryStage version: string os: NodeJS.Platform + nodeMajor: number repoSizeBucket?: TelemetryRepoSizeBucket - installPlatform?: string + graphSizeBucket?: TelemetryGraphSizeBucket + agentTarget?: TelemetryAgentTarget + spiEnabled?: boolean + failureBucket?: TelemetryFailureBucket + statusBucket?: TelemetryStatusBucket } export interface TelemetryOptions { @@ -35,8 +76,8 @@ interface TelemetryConfig { updated_at: number } -interface PersistedTelemetryEvent { - event: TelemetryEventName +interface LegacyPersistedTelemetryEvent { + event: 'install_success' | 'generate_success' | 'pack_success' | 'compare_success' recorded_at: string version: string os: NodeJS.Platform @@ -44,8 +85,28 @@ interface PersistedTelemetryEvent { install_platform?: string } -interface TelemetrySpool { +interface PersistedTelemetryEvent { + command: TelemetryCommand + stage: TelemetryStage + recorded_at: string + version: string + os: NodeJS.Platform + node_major?: number + repo_size_bucket?: TelemetryRepoSizeBucket + graph_size_bucket?: TelemetryGraphSizeBucket + agent_target?: TelemetryAgentTarget + spi_enabled?: boolean + failure_bucket?: TelemetryFailureBucket + status_bucket?: TelemetryStatusBucket +} + +interface LegacyTelemetrySpool { schema_version: 1 + events: LegacyPersistedTelemetryEvent[] +} + +interface TelemetrySpool { + schema_version: 2 events: PersistedTelemetryEvent[] } @@ -53,6 +114,46 @@ const DEFAULT_MAX_EVENTS = 200 const LOCK_RETRY_DELAY_MS = 10 const LOCK_TIMEOUT_MS = 1_000 +const TELEMETRY_COMMANDS: readonly TelemetryCommand[] = [ + 'install', + 'generate', + 'pack', + 'prompt', + 'context_pack', + 'doctor', + 'status', + 'compare', +] +const TELEMETRY_STAGES: readonly TelemetryStage[] = ['started', 'succeeded', 'failed'] +const TELEMETRY_REPO_SIZE_BUCKETS: readonly TelemetryRepoSizeBucket[] = ['1-24', '25-99', '100-499', '500-999', '1000+'] +const TELEMETRY_GRAPH_SIZE_BUCKETS: readonly TelemetryGraphSizeBucket[] = ['1-99', '100-499', '500-999', '1000-4999', '5000+'] +const TELEMETRY_FAILURE_BUCKETS: readonly TelemetryFailureBucket[] = [ + 'usage_error', + 'invalid_params', + 'missing_graph', + 'stale_graph', + 'stale_context', + 'tool_profile', + 'unsupported_corpus', + 'install_error', + 'unknown', +] +const TELEMETRY_STATUS_BUCKETS: readonly TelemetryStatusBucket[] = ['healthy', 'attention_needed'] +const TELEMETRY_AGENT_TARGETS: readonly TelemetryAgentTarget[] = [ + 'claude', + 'cursor', + 'codex', + 'copilot', + 'gemini', + 'aider', + 'opencode', + 'windows', + 'claw', + 'droid', + 'trae', + 'trae-cn', +] + function defaultConfigRoot(env: NodeJS.ProcessEnv): string { if (typeof env.XDG_CONFIG_HOME === 'string' && env.XDG_CONFIG_HOME.trim().length > 0) { return env.XDG_CONFIG_HOME @@ -122,28 +223,151 @@ function loadConfig(configFile: string): TelemetryConfig | null { return parseConfig(readFileSync(configFile, 'utf8')) } +function isTelemetryCommand(value: unknown): value is TelemetryCommand { + return typeof value === 'string' && TELEMETRY_COMMANDS.includes(value as TelemetryCommand) +} + +function isTelemetryStage(value: unknown): value is TelemetryStage { + return typeof value === 'string' && TELEMETRY_STAGES.includes(value as TelemetryStage) +} + +function isTelemetryRepoSizeBucket(value: unknown): value is TelemetryRepoSizeBucket { + return typeof value === 'string' && TELEMETRY_REPO_SIZE_BUCKETS.includes(value as TelemetryRepoSizeBucket) +} + +function isTelemetryGraphSizeBucket(value: unknown): value is TelemetryGraphSizeBucket { + return typeof value === 'string' && TELEMETRY_GRAPH_SIZE_BUCKETS.includes(value as TelemetryGraphSizeBucket) +} + +function isTelemetryFailureBucket(value: unknown): value is TelemetryFailureBucket { + return typeof value === 'string' && TELEMETRY_FAILURE_BUCKETS.includes(value as TelemetryFailureBucket) +} + +function isTelemetryStatusBucket(value: unknown): value is TelemetryStatusBucket { + return typeof value === 'string' && TELEMETRY_STATUS_BUCKETS.includes(value as TelemetryStatusBucket) +} + +function isTelemetryAgentTarget(value: unknown): value is TelemetryAgentTarget { + return typeof value === 'string' && TELEMETRY_AGENT_TARGETS.includes(value as TelemetryAgentTarget) +} + +function normalizeTelemetryEvent(record: Partial): PersistedTelemetryEvent | null { + if (!isTelemetryCommand(record.command) || !isTelemetryStage(record.stage)) { + return null + } + if (typeof record.recorded_at !== 'string' || record.recorded_at.length === 0) { + return null + } + if (typeof record.version !== 'string' || record.version.length === 0) { + return null + } + if (typeof record.os !== 'string') { + return null + } + + const normalized: PersistedTelemetryEvent = { + command: record.command, + stage: record.stage, + recorded_at: record.recorded_at, + version: record.version, + os: record.os, + } + + if (typeof record.node_major === 'number' && Number.isInteger(record.node_major) && record.node_major > 0) { + normalized.node_major = record.node_major + } + if (isTelemetryRepoSizeBucket(record.repo_size_bucket)) { + normalized.repo_size_bucket = record.repo_size_bucket + } + if (isTelemetryGraphSizeBucket(record.graph_size_bucket)) { + normalized.graph_size_bucket = record.graph_size_bucket + } + if (isTelemetryAgentTarget(record.agent_target)) { + normalized.agent_target = record.agent_target + } + if (typeof record.spi_enabled === 'boolean') { + normalized.spi_enabled = record.spi_enabled + } + if (isTelemetryFailureBucket(record.failure_bucket)) { + normalized.failure_bucket = record.failure_bucket + } + if (isTelemetryStatusBucket(record.status_bucket)) { + normalized.status_bucket = record.status_bucket + } + + return normalized +} + +function migrateLegacyEvent(record: Partial): PersistedTelemetryEvent | null { + if (typeof record.recorded_at !== 'string' || record.recorded_at.length === 0) { + return null + } + if (typeof record.version !== 'string' || record.version.length === 0) { + return null + } + if (typeof record.os !== 'string') { + return null + } + + const base: PersistedTelemetryEvent = { + command: 'generate', + stage: 'succeeded', + recorded_at: record.recorded_at, + version: record.version, + os: record.os, + } + + switch (record.event) { + case 'install_success': + base.command = 'install' + if (isTelemetryAgentTarget(record.install_platform)) { + base.agent_target = record.install_platform + } + return base + case 'generate_success': + base.command = 'generate' + if (isTelemetryRepoSizeBucket(record.repo_size_bucket)) { + base.repo_size_bucket = record.repo_size_bucket + } + return base + case 'pack_success': + base.command = 'pack' + if (isTelemetryRepoSizeBucket(record.repo_size_bucket)) { + base.repo_size_bucket = record.repo_size_bucket + } + return base + case 'compare_success': + base.command = 'compare' + if (isTelemetryRepoSizeBucket(record.repo_size_bucket)) { + base.repo_size_bucket = record.repo_size_bucket + } + return base + default: + return null + } +} + function parseSpool(text: string): TelemetrySpool | null { try { - const parsed = JSON.parse(text) as Partial - if (parsed.schema_version !== 1 || !Array.isArray(parsed.events)) { + const parsed = JSON.parse(text) as { schema_version?: unknown; events?: unknown } + if (parsed.schema_version === 1 && Array.isArray(parsed.events)) { + return { + schema_version: 2, + events: parsed.events + .filter((event: unknown): event is LegacyPersistedTelemetryEvent => typeof event === 'object' && event !== null) + .map((event) => migrateLegacyEvent(event)) + .filter((event): event is PersistedTelemetryEvent => event !== null), + } + } + if (parsed.schema_version !== 2 || !Array.isArray(parsed.events)) { return null } return { - schema_version: 1, + schema_version: 2, events: parsed.events - .filter((event): event is PersistedTelemetryEvent => typeof event === 'object' && event !== null) - .map((event) => { - const record = event as Partial - return { - event: record.event as TelemetryEventName, - recorded_at: String(record.recorded_at ?? ''), - version: String(record.version ?? ''), - os: (record.os as NodeJS.Platform | undefined) ?? platform(), - ...(record.repo_size_bucket ? { repo_size_bucket: record.repo_size_bucket } : {}), - ...(record.install_platform ? { install_platform: String(record.install_platform) } : {}), - } - }) - .filter((event) => event.recorded_at.length > 0 && event.version.length > 0), + .filter((event: unknown): event is PersistedTelemetryEvent => typeof event === 'object' && event !== null) + .map((event) => normalizeTelemetryEvent(event)) + .filter((event): event is PersistedTelemetryEvent => event !== null), } } catch { return null @@ -152,9 +376,9 @@ function parseSpool(text: string): TelemetrySpool | null { function loadSpool(spoolFile: string): TelemetrySpool { if (!existsSync(spoolFile)) { - return { schema_version: 1, events: [] } + return { schema_version: 2, events: [] } } - return parseSpool(readFileSync(spoolFile, 'utf8')) ?? { schema_version: 1, events: [] } + return parseSpool(readFileSync(spoolFile, 'utf8')) ?? { schema_version: 2, events: [] } } function uniqueTempPath(targetPath: string): string { @@ -203,6 +427,16 @@ function writeJsonAtomic(targetPath: string, value: unknown): void { } } +function summarizeCounts(values: string[]): string[] { + const counts = new Map() + for (const value of values) { + counts.set(value, (counts.get(value) ?? 0) + 1) + } + return [...counts.entries()] + .sort(([leftLabel, leftCount], [rightLabel, rightCount]) => rightCount - leftCount || leftLabel.localeCompare(rightLabel)) + .map(([label, count]) => `- ${label} ${count}`) +} + export function repoSizeBucketFromFileCount(fileCount: number): TelemetryRepoSizeBucket { if (fileCount <= 24) { return '1-24' @@ -219,6 +453,22 @@ export function repoSizeBucketFromFileCount(fileCount: number): TelemetryRepoSiz return '1000+' } +export function graphSizeBucketFromNodeCount(nodeCount: number): TelemetryGraphSizeBucket { + if (nodeCount <= 99) { + return '1-99' + } + if (nodeCount <= 499) { + return '100-499' + } + if (nodeCount <= 999) { + return '500-999' + } + if (nodeCount <= 4_999) { + return '1000-4999' + } + return '5000+' +} + export function getTelemetryStatus(options: TelemetryOptions = {}): TelemetryStatus { const env = options.env ?? process.env const configRoot = options.configRoot ?? defaultConfigRoot(env) @@ -292,8 +542,10 @@ export function formatTelemetryStatus(status: TelemetryStatus): string { `Telemetry: ${status.enabled ? 'enabled' : 'disabled'}`, `Reason: ${status.reason}`, `Event cache: ${status.eventCount} event(s) at ${status.spoolFile}`, - 'Tracked fields: event, version, os, optional install target, optional repo-size bucket', - 'Excluded fields: prompts, answers, source paths, source content', + 'Tracked fields: command, stage, recorded_at, version, os, node_major, optional agent_target, optional repo_size_bucket, optional graph_size_bucket, optional spi_enabled, optional failure_bucket, optional status_bucket', + 'Tracked surfaces: install, generate, pack, prompt, context_pack, doctor, status, compare', + 'Local controls: madar telemetry clear, madar telemetry report [spool.json ...]', + 'Excluded fields: prompts, answers, source paths, source content, repository names', ].join('\n') } @@ -303,8 +555,8 @@ function formatTelemetryPreferenceUpdate(preferenceEnabled: boolean, runtimeStat `Telemetry preference: ${preferenceEnabled ? 'enabled' : 'disabled'}`, `Config file: ${runtimeStatus.configFile}`, `Event cache: ${runtimeStatus.eventCount} event(s) at ${runtimeStatus.spoolFile}`, - 'Tracked fields: event, version, os, optional install target, optional repo-size bucket', - 'Excluded fields: prompts, answers, source paths, source content', + 'Tracked fields: command, stage, recorded_at, version, os, node_major, optional agent_target, optional repo_size_bucket, optional graph_size_bucket, optional spi_enabled, optional failure_bucket, optional status_bucket', + 'Excluded fields: prompts, answers, source paths, source content, repository names', ] if (runtimeStatus.enabled !== preferenceEnabled || runtimeStatus.reason !== persistedReason) { @@ -346,6 +598,18 @@ export function disableTelemetry(options: TelemetryOptions = {}): string { return formatTelemetryPreferenceUpdate(false, getTelemetryStatus({ ...options, configRoot, cacheRoot, env })) } +export function clearTelemetry(options: TelemetryOptions = {}): string { + const env = options.env ?? process.env + const cacheRoot = options.cacheRoot ?? defaultCacheRoot(env) + const spoolFile = telemetrySpoolFilePath(cacheRoot) + let clearedEvents = 0 + withExclusiveLock(spoolFile, () => { + clearedEvents = loadSpool(spoolFile).events.length + writeJsonAtomic(spoolFile, { schema_version: 2, events: [] } satisfies TelemetrySpool) + }) + return `Telemetry cache cleared: removed ${clearedEvents} event(s) at ${spoolFile}` +} + export function recordTelemetryEvent(input: TelemetryEventInput, options: TelemetryOptions = {}): boolean { const env = options.env ?? process.env const cacheRoot = options.cacheRoot ?? defaultCacheRoot(env) @@ -363,12 +627,18 @@ export function recordTelemetryEvent(input: TelemetryEventInput, options: Teleme withExclusiveLock(spoolFile, () => { const spool = loadSpool(spoolFile) spool.events.push({ - event: input.event, + command: input.command, + stage: input.stage, recorded_at: new Date(now()).toISOString(), version: input.version, os: input.os, + node_major: input.nodeMajor, ...(input.repoSizeBucket ? { repo_size_bucket: input.repoSizeBucket } : {}), - ...(input.installPlatform ? { install_platform: input.installPlatform } : {}), + ...(input.graphSizeBucket ? { graph_size_bucket: input.graphSizeBucket } : {}), + ...(input.agentTarget ? { agent_target: input.agentTarget } : {}), + ...(typeof input.spiEnabled === 'boolean' ? { spi_enabled: input.spiEnabled } : {}), + ...(input.failureBucket ? { failure_bucket: input.failureBucket } : {}), + ...(input.statusBucket ? { status_bucket: input.statusBucket } : {}), }) if (spool.events.length > maxEvents) { spool.events = spool.events.slice(-maxEvents) @@ -377,3 +647,58 @@ export function recordTelemetryEvent(input: TelemetryEventInput, options: Teleme }) return true } + +function safeLoadSpool(spoolFile: string): TelemetrySpool { + try { + return loadSpool(spoolFile) + } catch { + return { schema_version: 2, events: [] } + } +} + +export function readTelemetryReport(options: TelemetryOptions = {}, spoolPaths: string[] = []): string { + const env = options.env ?? process.env + const cacheRoot = options.cacheRoot ?? defaultCacheRoot(env) + const resolvedPaths = [...new Set([ + telemetrySpoolFilePath(cacheRoot), + ...spoolPaths.map((spoolPath) => resolve(spoolPath)), + ])] + const events = resolvedPaths.flatMap((spoolPath) => safeLoadSpool(spoolPath).events) + + const lines = [ + 'Telemetry funnel summary', + `Spools: ${resolvedPaths.length}`, + `Events: ${events.length}`, + ] + + if (events.length === 0) { + lines.push('No telemetry events found.') + return lines.join('\n') + } + + lines.push('Commands:') + lines.push(...summarizeCounts(events.map((event) => event.command))) + + lines.push('Stages:') + lines.push(...summarizeCounts(events.map((event) => event.stage))) + + const agentTargets = events.flatMap((event) => event.agent_target ? [event.agent_target] : []) + if (agentTargets.length > 0) { + lines.push('Agent targets:') + lines.push(...summarizeCounts(agentTargets)) + } + + const failureBuckets = events.flatMap((event) => event.failure_bucket ? [event.failure_bucket] : []) + if (failureBuckets.length > 0) { + lines.push('Failure buckets:') + lines.push(...summarizeCounts(failureBuckets)) + } + + const statusBuckets = events.flatMap((event) => event.status_bucket ? [event.status_bucket] : []) + if (statusBuckets.length > 0) { + lines.push('Status buckets:') + lines.push(...summarizeCounts(statusBuckets)) + } + + return lines.join('\n') +} diff --git a/tests/unit/cli.test.ts b/tests/unit/cli.test.ts index cafb989..e008f49 100644 --- a/tests/unit/cli.test.ts +++ b/tests/unit/cli.test.ts @@ -27,6 +27,7 @@ import { parseTimeTravelArgs, parseTryArgs, parseWatchArgs, + UsageError, } from '../../src/cli/parser.js' import { KnowledgeGraph } from '../../src/contracts/graph.js' @@ -1088,7 +1089,13 @@ describe('cli parser', () => { expect(parseTelemetryArgs(['enable'])).toEqual({ action: 'enable' }) expect(parseTelemetryArgs(['disable'])).toEqual({ action: 'disable' }) expect(parseTelemetryArgs(['status'])).toEqual({ action: 'status' }) - expect(() => parseTelemetryArgs([])).toThrow('Usage: madar telemetry ') + expect(parseTelemetryArgs(['clear'])).toEqual({ action: 'clear' }) + expect(parseTelemetryArgs(['report'])).toEqual({ action: 'report', spoolPaths: [] }) + expect(parseTelemetryArgs(['report', 'one.json', 'two.json'])).toEqual({ + action: 'report', + spoolPaths: ['one.json', 'two.json'], + }) + expect(() => parseTelemetryArgs([])).toThrow('Usage: madar telemetry ') }) it('parses install args and platform actions', () => { @@ -1261,7 +1268,7 @@ describe('cli main', () => { expect(help).toContain(' --pack PATH optional saved context-pack JSON for pack-quality evidence') expect(help).toContain('question coverage') expect(help).toContain('hook ') - expect(help).toContain('telemetry ') + expect(help).toContain('telemetry ') expect(help).toContain('install [--platform P]') expect(help).toContain('If you update madar, re-run your platform install command to refresh local agent rules:') expect(help).toContain('madar install --platform ') @@ -1362,18 +1369,88 @@ describe('cli main', () => { expect(confirmCalls).toBe(1) expect(telemetryEvents).toEqual([ { - event: 'compare_success', + command: 'compare', + stage: 'succeeded', version: '0.27.4', os: process.platform, + nodeMajor: expect.any(Number), repoSizeBucket: '1-24', }, ]) }) + it('records telemetry after compare failures with an actionable bucket', async () => { + const { io, logs, errors } = createIo() + const dependencies = createDependencies() as CliDependencies & { + recordTelemetryEvent: (event: unknown) => void + readInstalledVersion: () => string + } + const telemetryEvents: unknown[] = [] + + dependencies.runCompare = async () => { + throw new Error('Graph file not found at out/graph.json') + } + dependencies.recordTelemetryEvent = (event) => { + telemetryEvents.push(event) + } + dependencies.readInstalledVersion = () => '0.27.4' + + const exitCode = await executeCli(['compare', 'How does auth work?', '--exec', 'claude --print "$(cat {prompt_file})"', '--yes'], io, dependencies) + + expect(exitCode).toBe(1) + expect(logs).toEqual([]) + expect(errors).toContain('error: Graph file not found at out/graph.json') + expect(telemetryEvents).toEqual([ + { + command: 'compare', + stage: 'failed', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + failureBucket: 'missing_graph', + }, + ]) + }) + + it('keeps specific failure buckets when compare throws a wrapped UsageError', async () => { + const { io, logs, errors } = createIo() + const dependencies = createDependencies() as CliDependencies & { + recordTelemetryEvent: (event: unknown) => void + readInstalledVersion: () => string + } + const telemetryEvents: unknown[] = [] + + dependencies.runCompare = async () => { + throw new UsageError('error: install required before compare') + } + dependencies.recordTelemetryEvent = (event) => { + telemetryEvents.push(event) + } + dependencies.readInstalledVersion = () => '0.27.4' + + const exitCode = await executeCli(['compare', 'How does auth work?', '--exec', 'claude --print "$(cat {prompt_file})"', '--yes'], io, dependencies) + + expect(exitCode).toBe(2) + expect(logs).toEqual([]) + expect(errors).toContain('error: install required before compare') + expect(telemetryEvents).toEqual([ + { + command: 'compare', + stage: 'failed', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + failureBucket: 'install_error', + }, + ]) + }) + it('routes telemetry commands through the injected dependencies', async () => { const enable = createIo() const disable = createIo() const status = createIo() + const clear = createIo() + const report = createIo() const enabledDependencies = createDependencies() as CliDependencies & { enableTelemetry: () => string } @@ -1383,18 +1460,30 @@ describe('cli main', () => { const statusDependencies = createDependencies() as CliDependencies & { readTelemetryStatus: () => string } + const clearDependencies = createDependencies() as CliDependencies & { + clearTelemetry: () => string + } + const reportDependencies = createDependencies() as CliDependencies & { + readTelemetryReport: (spoolPaths?: string[]) => string + } enabledDependencies.enableTelemetry = () => 'Telemetry enabled.' disabledDependencies.disableTelemetry = () => 'Telemetry disabled.' statusDependencies.readTelemetryStatus = () => 'Telemetry: disabled' + clearDependencies.clearTelemetry = () => 'Telemetry cache cleared.' + reportDependencies.readTelemetryReport = (spoolPaths) => `Telemetry funnel summary for ${spoolPaths?.join(',') ?? 'default'}` await expect(executeCli(['telemetry', 'enable'], enable.io, enabledDependencies)).resolves.toBe(0) await expect(executeCli(['telemetry', 'disable'], disable.io, disabledDependencies)).resolves.toBe(0) await expect(executeCli(['telemetry', 'status'], status.io, statusDependencies)).resolves.toBe(0) + await expect(executeCli(['telemetry', 'clear'], clear.io, clearDependencies)).resolves.toBe(0) + await expect(executeCli(['telemetry', 'report', 'one.json', 'two.json'], report.io, reportDependencies)).resolves.toBe(0) expect(enable.logs).toContain('Telemetry enabled.') expect(disable.logs).toContain('Telemetry disabled.') expect(status.logs).toContain('Telemetry: disabled') + expect(clear.logs).toContain('Telemetry cache cleared.') + expect(report.logs).toContain('Telemetry funnel summary for one.json,two.json') }) it('prefers the telemetry command over implicit generate when a telemetry path exists', async () => { @@ -1442,9 +1531,11 @@ describe('cli main', () => { expect(logs).toEqual(['pack result']) expect(telemetryEvents).toEqual([ { - event: 'pack_success', + command: 'pack', + stage: 'succeeded', version: '0.27.4', os: process.platform, + nodeMajor: expect.any(Number), repoSizeBucket: '1-24', }, ]) @@ -1470,10 +1561,22 @@ describe('cli main', () => { expect(logs[0]).toContain('[madar generate]') expect(telemetryEvents).toEqual([ { - event: 'generate_success', + command: 'generate', + stage: 'started', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + spiEnabled: false, + }, + { + command: 'generate', + stage: 'succeeded', version: '0.27.4', os: process.platform, + nodeMajor: expect.any(Number), repoSizeBucket: '1-24', + graphSizeBucket: '1-99', + spiEnabled: false, }, ]) }) @@ -1499,16 +1602,161 @@ describe('cli main', () => { expect(agent.errors).toEqual([]) expect(telemetryEvents).toEqual([ { - event: 'install_success', + command: 'install', + stage: 'started', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + agentTarget: 'aider', + }, + { + command: 'install', + stage: 'succeeded', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + agentTarget: 'aider', + }, + { + command: 'install', + stage: 'started', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + agentTarget: 'aider', + }, + { + command: 'install', + stage: 'succeeded', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + agentTarget: 'aider', + }, + ]) + }) + + it('records telemetry after direct agent install failures', async () => { + const { io, logs, errors } = createIo() + const dependencies = createDependencies() as CliDependencies & { + recordTelemetryEvent: (event: unknown) => void + readInstalledVersion: () => string + agentsInstall: (projectDir: string, platform: string) => string + } + const telemetryEvents: unknown[] = [] + + dependencies.recordTelemetryEvent = (event) => { + telemetryEvents.push(event) + } + dependencies.readInstalledVersion = () => '0.27.4' + dependencies.agentsInstall = () => { + throw new Error('install hook write failed') + } + + const exitCode = await executeCli(['aider', 'install'], io, dependencies) + + expect(exitCode).toBe(1) + expect(logs).toEqual([]) + expect(errors).toContain('error: install hook write failed') + expect(telemetryEvents).toEqual([ + { + command: 'install', + stage: 'started', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + agentTarget: 'aider', + }, + { + command: 'install', + stage: 'failed', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + agentTarget: 'aider', + failureBucket: 'install_error', + }, + ]) + }) + + it('records telemetry for prompt failures with an actionable failure bucket', async () => { + const { io, logs, errors } = createIo() + const dependencies = createDependencies() as CliDependencies & { + recordTelemetryEvent: (event: unknown) => void + readInstalledVersion: () => string + } + const telemetryEvents: unknown[] = [] + + dependencies.runContextPrompt = async () => { + throw new Error('Graph file not found at out/graph.json') + } + dependencies.recordTelemetryEvent = (event) => { + telemetryEvents.push(event) + } + dependencies.readInstalledVersion = () => '0.27.4' + + const exitCode = await executeCli(['prompt', 'how does auth work?', '--provider', 'claude'], io, dependencies) + + expect(exitCode).toBe(1) + expect(logs).toEqual([]) + expect(errors).toContain('error: Graph file not found at out/graph.json') + expect(telemetryEvents).toEqual([ + { + command: 'prompt', + stage: 'failed', + version: '0.27.4', + os: process.platform, + nodeMajor: expect.any(Number), + failureBucket: 'missing_graph', + }, + ]) + }) + + it('records telemetry buckets after doctor and status succeed', async () => { + const doctor = createIo() + const status = createIo() + const doctorDependencies = createDependencies() as CliDependencies & { + recordTelemetryEvent: (event: unknown) => void + readInstalledVersion: () => string + readDoctorTelemetryBucket: (graphPath: string) => string + } + const statusDependencies = createDependencies() as CliDependencies & { + recordTelemetryEvent: (event: unknown) => void + readInstalledVersion: () => string + readStatusTelemetryBucket: (graphPath: string) => string + } + const telemetryEvents: unknown[] = [] + + doctorDependencies.recordTelemetryEvent = (event) => { + telemetryEvents.push(event) + } + statusDependencies.recordTelemetryEvent = (event) => { + telemetryEvents.push(event) + } + doctorDependencies.readInstalledVersion = () => '0.27.4' + statusDependencies.readInstalledVersion = () => '0.27.4' + doctorDependencies.readDoctorTelemetryBucket = () => 'attention_needed' + statusDependencies.readStatusTelemetryBucket = () => 'healthy' + + await expect(executeCli(['doctor'], doctor.io, doctorDependencies)).resolves.toBe(0) + await expect(executeCli(['status'], status.io, statusDependencies)).resolves.toBe(0) + + expect(telemetryEvents).toEqual([ + { + command: 'doctor', + stage: 'succeeded', version: '0.27.4', os: process.platform, - installPlatform: 'aider', + nodeMajor: expect.any(Number), + statusBucket: 'attention_needed', }, { - event: 'install_success', + command: 'status', + stage: 'succeeded', version: '0.27.4', os: process.platform, - installPlatform: 'aider', + nodeMajor: expect.any(Number), + statusBucket: 'healthy', }, ]) }) diff --git a/tests/unit/stdio-server.test.ts b/tests/unit/stdio-server.test.ts index 57ec4ba..640ad07 100644 --- a/tests/unit/stdio-server.test.ts +++ b/tests/unit/stdio-server.test.ts @@ -993,6 +993,113 @@ describe('stdio runtime', () => { } }) + it('records opt-in context_pack telemetry for MCP success and failure without source-sensitive payloads', async () => { + const root = createGraphFixtureRoot() + const previousEnv = { + XDG_CONFIG_HOME: process.env.XDG_CONFIG_HOME, + XDG_CACHE_HOME: process.env.XDG_CACHE_HOME, + MADAR_ENABLE_TELEMETRY: process.env.MADAR_ENABLE_TELEMETRY, + MADAR_DISABLE_TELEMETRY: process.env.MADAR_DISABLE_TELEMETRY, + DO_NOT_TRACK: process.env.DO_NOT_TRACK, + CI: process.env.CI, + } + + try { + const graphPath = join(root, 'graph.json') + const configRoot = join(root, 'xdg-config') + const cacheRoot = join(root, 'xdg-cache') + mkdirSync(configRoot, { recursive: true }) + mkdirSync(cacheRoot, { recursive: true }) + process.env.XDG_CONFIG_HOME = configRoot + process.env.XDG_CACHE_HOME = cacheRoot + process.env.MADAR_ENABLE_TELEMETRY = '1' + delete process.env.MADAR_DISABLE_TELEMETRY + delete process.env.DO_NOT_TRACK + delete process.env.CI + + const success = await Promise.resolve(handleStdioRequest(graphPath, { + id: 301, + method: 'tools/call', + params: { + name: 'context_pack', + arguments: { + prompt: 'How does auth reach transport?', + budget: 1200, + }, + }, + })) + const failure = await Promise.resolve(handleStdioRequest(graphPath, { + id: 302, + method: 'tools/call', + params: { + name: 'context_pack', + arguments: { + budget: 1200, + }, + }, + })) + + expect(success?.error).toBeUndefined() + expect(failure?.error?.message).toContain('context_pack requires a string prompt parameter') + + const spool = JSON.parse(readFileSync(join(cacheRoot, 'madar', 'telemetry-events.json'), 'utf8')) as { + schema_version: number + events: Array> + } + expect(spool).toEqual({ + schema_version: 2, + events: [ + expect.objectContaining({ + command: 'context_pack', + stage: 'succeeded', + repo_size_bucket: '1-24', + graph_size_bucket: '1-99', + }), + expect.objectContaining({ + command: 'context_pack', + stage: 'failed', + failure_bucket: 'invalid_params', + }), + ], + }) + const serialized = JSON.stringify(spool) + expect(serialized).not.toContain('How does auth reach transport?') + expect(serialized).not.toContain('auth.ts') + } finally { + if (previousEnv.XDG_CONFIG_HOME === undefined) { + delete process.env.XDG_CONFIG_HOME + } else { + process.env.XDG_CONFIG_HOME = previousEnv.XDG_CONFIG_HOME + } + if (previousEnv.XDG_CACHE_HOME === undefined) { + delete process.env.XDG_CACHE_HOME + } else { + process.env.XDG_CACHE_HOME = previousEnv.XDG_CACHE_HOME + } + if (previousEnv.MADAR_ENABLE_TELEMETRY === undefined) { + delete process.env.MADAR_ENABLE_TELEMETRY + } else { + process.env.MADAR_ENABLE_TELEMETRY = previousEnv.MADAR_ENABLE_TELEMETRY + } + if (previousEnv.MADAR_DISABLE_TELEMETRY === undefined) { + delete process.env.MADAR_DISABLE_TELEMETRY + } else { + process.env.MADAR_DISABLE_TELEMETRY = previousEnv.MADAR_DISABLE_TELEMETRY + } + if (previousEnv.DO_NOT_TRACK === undefined) { + delete process.env.DO_NOT_TRACK + } else { + process.env.DO_NOT_TRACK = previousEnv.DO_NOT_TRACK + } + if (previousEnv.CI === undefined) { + delete process.env.CI + } else { + process.env.CI = previousEnv.CI + } + rmSync(root, { recursive: true, force: true }) + } + }) + it('exposes context-pack and context-prompt MCP flows with reusable prompt sessions', async () => { const root = createGraphFixtureRoot() try { diff --git a/tests/unit/telemetry-doc.test.ts b/tests/unit/telemetry-doc.test.ts index d7da132..20d46eb 100644 --- a/tests/unit/telemetry-doc.test.ts +++ b/tests/unit/telemetry-doc.test.ts @@ -9,6 +9,8 @@ describe('telemetry documentation', () => { expect(readme).toContain('madar telemetry enable') expect(readme).toContain('madar telemetry disable') + expect(readme).toContain('madar telemetry clear') + expect(readme).toContain('madar telemetry report') expect(readme).toContain('MADAR_ENABLE_TELEMETRY=1') expect(readme).toContain('docs/telemetry.md') expect(readme).toContain('Telemetry is disabled unless you explicitly enable it') @@ -17,13 +19,19 @@ describe('telemetry documentation', () => { it('documents collected and excluded telemetry fields explicitly', () => { const doc = readFileSync(resolve('docs/telemetry.md'), 'utf8') - expect(doc).toContain('install_success') - expect(doc).toContain('generate_success') - expect(doc).toContain('pack_success') - expect(doc).toContain('compare_success') + expect(doc).toContain('command') + expect(doc).toContain('stage') expect(doc).toContain('version') expect(doc).toContain('os') + expect(doc).toContain('node_major') + expect(doc).toContain('graph_size_bucket') expect(doc).toContain('repo_size_bucket') + expect(doc).toContain('failure_bucket') + expect(doc).toContain('status_bucket') + expect(doc).toContain('madar telemetry clear') + expect(doc).toContain('madar telemetry report') + expect(doc).toContain('`madar compare` (`succeeded`, `failed`)') + expect(doc).toContain('`madar doctor` and `madar status` (`succeeded`, `failed`, plus `status_bucket`)') expect(doc).toContain('prompt text') expect(doc).toContain('answer text') expect(doc).toContain('source paths') diff --git a/tests/unit/telemetry.test.ts b/tests/unit/telemetry.test.ts index 8d92257..7910851 100644 --- a/tests/unit/telemetry.test.ts +++ b/tests/unit/telemetry.test.ts @@ -1,14 +1,17 @@ -import { mkdtempSync, readFileSync, rmSync } from 'node:fs' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' import { join } from 'node:path' import { tmpdir } from 'node:os' import { describe, expect, it } from 'vitest' import { + clearTelemetry, disableTelemetry, enableTelemetry, formatTelemetryStatus, getTelemetryStatus, + graphSizeBucketFromNodeCount, + readTelemetryReport, recordTelemetryEvent, repoSizeBucketFromFileCount, } from '../../src/shared/telemetry.js' @@ -47,10 +50,12 @@ describe('telemetry', () => { }) expect(recordTelemetryEvent({ - event: 'generate_success', + command: 'generate', + stage: 'started', version: '0.27.4', os: 'darwin', - repoSizeBucket: repoSizeBucketFromFileCount(240), + nodeMajor: 20, + spiEnabled: true, }, { configRoot, cacheRoot, @@ -60,10 +65,14 @@ describe('telemetry', () => { })).toBe(true) expect(recordTelemetryEvent({ - event: 'pack_success', + command: 'generate', + stage: 'succeeded', version: '0.27.4', os: 'darwin', - repoSizeBucket: repoSizeBucketFromFileCount(12), + nodeMajor: 20, + repoSizeBucket: repoSizeBucketFromFileCount(240), + graphSizeBucket: graphSizeBucketFromNodeCount(1_200), + spiEnabled: true, }, { configRoot, cacheRoot, @@ -73,10 +82,14 @@ describe('telemetry', () => { })).toBe(true) expect(recordTelemetryEvent({ - event: 'compare_success', + command: 'context_pack', + stage: 'failed', version: '0.27.4', os: 'darwin', - repoSizeBucket: repoSizeBucketFromFileCount(2_100), + nodeMajor: 20, + repoSizeBucket: repoSizeBucketFromFileCount(12), + graphSizeBucket: graphSizeBucketFromNodeCount(5), + failureBucket: 'invalid_params', }, { configRoot, cacheRoot, @@ -87,19 +100,27 @@ describe('telemetry', () => { const spoolFile = join(cacheRoot, 'madar', 'telemetry-events.json') expect(JSON.parse(readFileSync(spoolFile, 'utf8'))).toEqual({ - schema_version: 1, + schema_version: 2, events: [ expect.objectContaining({ - event: 'pack_success', + command: 'generate', + stage: 'succeeded', version: '0.27.4', os: 'darwin', - repo_size_bucket: '1-24', + node_major: 20, + repo_size_bucket: '100-499', + graph_size_bucket: '1000-4999', + spi_enabled: true, }), expect.objectContaining({ - event: 'compare_success', + command: 'context_pack', + stage: 'failed', version: '0.27.4', os: 'darwin', - repo_size_bucket: '1000+', + node_major: 20, + repo_size_bucket: '1-24', + graph_size_bucket: '1-99', + failure_bucket: 'invalid_params', }), ], }) @@ -129,9 +150,11 @@ describe('telemetry', () => { expect(status.enabled).toBe(false) expect(formatTelemetryStatus(status)).toContain('MADAR_DISABLE_TELEMETRY=1') expect(recordTelemetryEvent({ - event: 'generate_success', + command: 'generate', + stage: 'succeeded', version: '0.27.4', os: 'darwin', + nodeMajor: 20, repoSizeBucket: '100-499', }, { configRoot, @@ -190,9 +213,11 @@ describe('telemetry', () => { }) expect(recordTelemetryEvent({ - event: 'generate_success', + command: 'generate', + stage: 'succeeded', version: '0.27.4', os: 'darwin', + nodeMajor: 20, repoSizeBucket: '25-99', }, { configRoot, @@ -203,9 +228,11 @@ describe('telemetry', () => { })).toBe(true) expect(recordTelemetryEvent({ - event: 'pack_success', + command: 'pack', + stage: 'succeeded', version: '0.27.4', os: 'darwin', + nodeMajor: 20, repoSizeBucket: '25-99', }, { configRoot, @@ -222,4 +249,152 @@ describe('telemetry', () => { rmSync(cacheRoot, { recursive: true, force: true }) } }) + + it('clears the local telemetry spool without deleting the persisted preference', () => { + const configRoot = mkdtempSync(join(tmpdir(), 'madar-telemetry-config-')) + const cacheRoot = mkdtempSync(join(tmpdir(), 'madar-telemetry-cache-')) + + try { + enableTelemetry({ + configRoot, + cacheRoot, + env: {}, + }) + + expect(recordTelemetryEvent({ + command: 'install', + stage: 'started', + version: '0.27.4', + os: 'darwin', + nodeMajor: 20, + agentTarget: 'copilot', + }, { + configRoot, + cacheRoot, + env: {}, + })).toBe(true) + + const message = clearTelemetry({ + configRoot, + cacheRoot, + env: {}, + }) + + expect(message).toContain('Telemetry cache cleared') + expect(getTelemetryStatus({ + configRoot, + cacheRoot, + env: {}, + }).enabled).toBe(true) + + const spoolFile = join(cacheRoot, 'madar', 'telemetry-events.json') + expect(JSON.parse(readFileSync(spoolFile, 'utf8'))).toEqual({ + schema_version: 2, + events: [], + }) + } finally { + rmSync(configRoot, { recursive: true, force: true }) + rmSync(cacheRoot, { recursive: true, force: true }) + } + }) + + it('summarizes anonymized funnel counts from both v2 and legacy v1 spools', () => { + const configRoot = mkdtempSync(join(tmpdir(), 'madar-telemetry-config-')) + const cacheRoot = mkdtempSync(join(tmpdir(), 'madar-telemetry-cache-')) + + try { + const legacySpoolPath = join(cacheRoot, 'legacy-telemetry-events.json') + writeFileSync(legacySpoolPath, JSON.stringify({ + schema_version: 1, + events: [ + { + event: 'install_success', + recorded_at: '2026-06-02T00:00:00.000Z', + version: '0.27.4', + os: 'darwin', + install_platform: 'cursor', + }, + { + event: 'generate_success', + recorded_at: '2026-06-02T00:01:00.000Z', + version: '0.27.4', + os: 'darwin', + repo_size_bucket: '25-99', + }, + ], + }, null, 2)) + + enableTelemetry({ + configRoot, + cacheRoot, + env: {}, + }) + expect(recordTelemetryEvent({ + command: 'status', + stage: 'succeeded', + version: '0.27.8', + os: 'darwin', + nodeMajor: 20, + statusBucket: 'healthy', + }, { + configRoot, + cacheRoot, + env: {}, + })).toBe(true) + + const report = readTelemetryReport({ + configRoot, + cacheRoot, + env: {}, + }, [legacySpoolPath]) + + expect(report).toContain('Telemetry funnel summary') + expect(report).toContain('install 1') + expect(report).toContain('generate 1') + expect(report).toContain('status 1') + expect(report).toContain('cursor 1') + expect(report).toContain('healthy 1') + } finally { + rmSync(configRoot, { recursive: true, force: true }) + rmSync(cacheRoot, { recursive: true, force: true }) + } + }) + + it('skips unreadable report inputs instead of crashing the summary', () => { + const configRoot = mkdtempSync(join(tmpdir(), 'madar-telemetry-config-')) + const cacheRoot = mkdtempSync(join(tmpdir(), 'madar-telemetry-cache-')) + + try { + enableTelemetry({ + configRoot, + cacheRoot, + env: {}, + }) + expect(recordTelemetryEvent({ + command: 'status', + stage: 'succeeded', + version: '0.27.8', + os: 'darwin', + nodeMajor: 20, + statusBucket: 'healthy', + }, { + configRoot, + cacheRoot, + env: {}, + })).toBe(true) + + const report = readTelemetryReport({ + configRoot, + cacheRoot, + env: {}, + }, [cacheRoot]) + + expect(report).toContain('Telemetry funnel summary') + expect(report).toContain('status 1') + expect(report).toContain('healthy 1') + } finally { + rmSync(configRoot, { recursive: true, force: true }) + rmSync(cacheRoot, { recursive: true, force: true }) + } + }) })