diff --git a/.claude/skills/frigg/SKILL.md b/.claude/skills/frigg/SKILL.md index 7b418c025..ba6739866 100644 --- a/.claude/skills/frigg/SKILL.md +++ b/.claude/skills/frigg/SKILL.md @@ -1,6 +1,6 @@ --- name: frigg -description: "Core reference and entry point for the Frigg integration framework: what Frigg is, hexagonal architecture and the golden rule, the integration definition pattern, the frigg CLI (install, start, build, deploy, doctor, repair, ui, generate-iam), AWS infrastructure (domain builders, scheduler, VPC, osls), field-level encryption, the Admin Script Runner (admin scripts, sync/async execution, chaining, scheduling), telemetry & usage tracking (OpenTelemetry, this.telemetry, Definition.usage, frigg.usage.*), the monorepo layout, and anti-patterns. Use when working in a Frigg project or repo (friggframework packages, IntegrationBase, infrastructure.js), understanding Frigg's architecture, configuring infrastructure/VPC/encryption/telemetry, adding observability or usage counters, or running frigg CLI commands. Links to the focused companion skills: frigg-api-modules, frigg-management-api, frigg-user-actions, and frigg-development-best-practices." +description: "Core reference and entry point for the Frigg integration framework: what Frigg is, hexagonal architecture and the golden rule, the integration definition pattern, the frigg CLI (install, start, build, deploy, doctor, repair, ui, generate-iam), AWS infrastructure (domain builders, scheduler, VPC, osls), field-level encryption, the Admin Script Runner (admin scripts, sync/async execution, chaining, scheduling), reporting as an admin operation (ReportBase, run modes live/recorded/snapshot, artifacts, scheduling), telemetry & usage tracking (OpenTelemetry, this.telemetry, Definition.usage, frigg.usage.*), the monorepo layout, and anti-patterns. Use when working in a Frigg project or repo (friggframework packages, IntegrationBase, infrastructure.js), understanding Frigg's architecture, configuring infrastructure/VPC/encryption/telemetry, adding observability or usage counters, or running frigg CLI commands. Links to the focused companion skills: frigg-api-modules, frigg-management-api, frigg-user-actions, and frigg-development-best-practices." --- # Frigg Integration Framework Expert @@ -98,6 +98,65 @@ A script extends `AdminScriptBase` with a static `Definition` (name, version, `i - **When to use it:** work that won't fit one execution — beat the 15-min executor cap by paging/resuming; fan out one child per item/batch; isolate per-item failures; stage pipelines (A queues B with its output). For small bounded work, or when you need the result in the response, just use one sync/async execution. - **Caveats:** fire-and-forget (you don't get the child's result back — correlate via `parentExecutionId`); at-least-once delivery, so **make child scripts idempotent**; and there is **no depth guard**, so keep continuation targets terminal or a self-queuing script fans out unbounded. +## Reports (ADR-010) + +A **report is an admin operation whose output is its payload** — a sibling of admin scripts on the same runner, admin API key (`ADMIN_API_KEY`), async/SQS execution, and EventBridge scheduling. **Core ships built-in reports; adopters register their own.** Register with `reports: [MyReport]` in the app definition (+ `admin: { includeBuiltinReports: true }` for the core built-ins, e.g. `integrations`). A report extends `ReportBase` (from `@friggframework/core`) with a static `Definition` and `async execute(frigg, params)`, where `frigg` is the same command bundle scripts get as `context.commands` — **reads go through it, never a repository**. + +- **Run modes** (`Definition.runModes`, first is default): **`live`** computes inline and persists nothing (cheap, always-fresh); **`recorded`** persists an execution record (input/results/logs) you poll async; **`snapshot`** is a recorded run tagged into a named series (trends). All three write to the isolated `AdminScriptExecution` store as `type: 'REPORT'` (no user/integration FK), so a user-scoped query can never return a report record. +- **Endpoints** (auth: `x-frigg-admin-api-key`): `GET /api/v2/reports` (list), `GET /api/v2/reports/{name}` (definition), `POST /api/v2/reports/{name}/run` with `{ mode, params }` (**`live` → 200 inline**; **`recorded`/`snapshot` → 202 `{ executionId }`**, queued to the dedicated `ReportQueue`), `GET .../{name}/snapshots?from=&to=` (the series), `GET .../executions/{id}`, and `GET|PUT|DELETE .../{name}/schedule`. +- **Output**: `output.format: 'json'` returns inline; `'csv'|'pdf'|'zip'` is written to artifact storage (S3, private + SSE, retrieved via signed URL) with a `{ summary, artifact }` on the record — non-JSON therefore runs `recorded`/`snapshot`, not `live`. Set `REPORT_ARTIFACT_BUCKET` (the infra provisions it when a non-JSON report is registered). +- **Scheduling** reuses the admin scheduler; a scheduled run targets the report executor with `{ reportName, mode: schedule.mode || 'snapshot', trigger: 'SCHEDULED' }`. Report and script names share one namespace (bootstrap rejects collisions). **A `schedule` block in the Definition only supplies the default scheduled *mode*; the recurring trigger is activated via `PUT /:name/schedule` — the DB schedule is the single source of truth (a declared `enabled`/`cron` does not fire on its own).** + +**Example 1 — adopter report, snapshot + daily schedule, reads via `frigg`:** + +```javascript +const { ReportBase } = require('@friggframework/core'); + +class ConnectedAccountsActivity extends ReportBase { + static Definition = { + name: 'connected-accounts-activity', + version: '1.0.0', + runModes: ['snapshot', 'recorded', 'live'], // first = default + inputSchema: { type: 'object', properties: { + windowDays: { type: 'integer', enum: [30, 60, 90], default: 30 } } }, + output: { format: 'json' }, + schedule: { enabled: true, cron: 'cron(0 6 * * ? *)', mode: 'snapshot' }, // default mode; activate via PUT .../schedule + }; + + async execute(frigg, params) { // frigg === context.commands + const since = new Date(Date.now() - (params.windowDays ?? 30) * 864e5); + const byType = await frigg.credentials.countActiveByType({ since }); + return { windowDays: params.windowDays ?? 30, byType }; + } +} +// POST /api/v2/reports/connected-accounts-activity/run {"mode":"snapshot"} → 202 {executionId} +// GET /api/v2/reports/connected-accounts-activity/snapshots?from=&to= → the daily trend +``` + +**Example 2 — CSV export to artifact storage (recorded), non-JSON output:** + +```javascript +class ContactExportReport extends ReportBase { + static Definition = { + name: 'contact-export', + version: '1.0.0', + runModes: ['recorded'], // non-JSON can't run live + inputSchema: { type: 'object', properties: { type: { type: 'string' } } }, + output: { format: 'csv' }, // → S3 + signed URL + }; + + async execute(frigg, params) { + const rows = await frigg.integrations.listForReport({ type: params.type }); + const csv = ['id,type,status', ...rows.map((r) => `${r.id},${r.type},${r.status}`)].join('\n'); + // Non-JSON reports return { file, summary, fileType }; the runner stores the + // file and records { summary, artifact } — retrieve via GET .../executions/{id}. + return { file: csv, summary: { rows: rows.length }, fileType: 'csv' }; + } +} +``` + +The built-in `integrations` report (PR #607) is itself a `ReportBase` in core; use it as the reference implementation. Full guide: `packages/core/reporting/README.md`. + ## Telemetry & Usage (ADR-011) Vendor-neutral OpenTelemetry (traces + metrics) plus durable per-integration diff --git a/packages/admin-scripts/index.js b/packages/admin-scripts/index.js index be13ff7b8..e8428d710 100644 --- a/packages/admin-scripts/index.js +++ b/packages/admin-scripts/index.js @@ -16,6 +16,10 @@ const { ScriptRunner, createScriptRunner, } = require('./src/application/script-runner'); +const { + ReportRunner, + createReportRunner, +} = require('./src/application/report-runner'); // Infrastructure const { @@ -26,9 +30,15 @@ const { app, handler: routerHandler, } = require('./src/infrastructure/admin-script-router'); +const { + handler: reportRouterHandler, +} = require('./src/infrastructure/report-router'); const { handler: executorHandler, } = require('./src/infrastructure/script-executor-handler'); +const { + handler: reportExecutorHandler, +} = require('./src/infrastructure/report-executor-handler'); // Adapters const { SchedulerAdapter } = require('./src/adapters/scheduler-adapter'); @@ -48,13 +58,17 @@ module.exports = { createAdminScriptContext, ScriptRunner, createScriptRunner, + ReportRunner, + createReportRunner, // Infrastructure layer validateAdminApiKey, router, app, routerHandler, + reportRouterHandler, executorHandler, + reportExecutorHandler, // Adapters SchedulerAdapter, diff --git a/packages/admin-scripts/src/adapters/__tests__/aws-scheduler-adapter.test.js b/packages/admin-scripts/src/adapters/__tests__/aws-scheduler-adapter.test.js index e1c251fad..4792d4b6e 100644 --- a/packages/admin-scripts/src/adapters/__tests__/aws-scheduler-adapter.test.js +++ b/packages/admin-scripts/src/adapters/__tests__/aws-scheduler-adapter.test.js @@ -422,6 +422,60 @@ describe('AWSSchedulerAdapter', () => { }); }); + describe('report parameterization (namePrefix + buildInput)', () => { + const reportParams = { + targetLambdaArn: + 'arn:aws:lambda:us-east-1:123456789012:function:report-executor', + scheduleGroupName: 'frigg-admin-scripts', + roleArn: 'arn:aws:iam::123456789012:role/test-role', + namePrefix: 'frigg-report-', + buildInput: ({ input }) => ({ + reportName: 'integrations', + mode: 'snapshot', + trigger: 'SCHEDULED', + params: input || {}, + }), + }; + + it('targets the report executor and emits a report-shaped message', async () => { + const reportAdapter = new AWSSchedulerAdapter({ ...reportParams }); + mockSend.mockResolvedValue({ + ScheduleArn: + 'arn:aws:scheduler:us-east-1:123456789012:schedule/frigg-admin-scripts/frigg-report-integrations', + }); + + const result = await reportAdapter.createSchedule({ + scriptName: 'integrations', + cronExpression: 'cron(0 6 * * ? *)', + }); + + expect(result.scheduleName).toBe('frigg-report-integrations'); + + const command = mockSend.mock.calls[0][0]; + expect(command.params.Name).toBe('frigg-report-integrations'); + expect(command.params.Target.Arn).toBe( + 'arn:aws:lambda:us-east-1:123456789012:function:report-executor' + ); + expect(JSON.parse(command.params.Target.Input)).toEqual({ + reportName: 'integrations', + mode: 'snapshot', + trigger: 'SCHEDULED', + params: {}, + }); + }); + + it('deletes the report schedule by its prefixed name', async () => { + const reportAdapter = new AWSSchedulerAdapter({ ...reportParams }); + mockSend.mockResolvedValue({}); + + await reportAdapter.deleteSchedule('integrations'); + + const command = mockSend.mock.calls[0][0]; + expect(command._type).toBe('DeleteScheduleCommand'); + expect(command.params.Name).toBe('frigg-report-integrations'); + }); + }); + describe('Lazy SDK loading', () => { it('should load AWS SDK on first client access', () => { const newAdapter = new AWSSchedulerAdapter({ ...defaultParams }); diff --git a/packages/admin-scripts/src/adapters/__tests__/scheduler-adapter-factory.test.js b/packages/admin-scripts/src/adapters/__tests__/scheduler-adapter-factory.test.js index b369d3f7d..22024b15e 100644 --- a/packages/admin-scripts/src/adapters/__tests__/scheduler-adapter-factory.test.js +++ b/packages/admin-scripts/src/adapters/__tests__/scheduler-adapter-factory.test.js @@ -1,6 +1,7 @@ const { createSchedulerAdapter, createSchedulerAdapterFromEnv, + createReportSchedulerAdapterFromEnv, } = require('../scheduler-adapter-factory'); const { AWSSchedulerAdapter } = require('../aws-scheduler-adapter'); const { LocalSchedulerAdapter } = require('../local-scheduler-adapter'); @@ -185,4 +186,52 @@ describe('Scheduler Adapter Factory', () => { expect(adapter.roleArn).toBe(awsAdapterParams.roleArn); }); }); + + describe('createReportSchedulerAdapterFromEnv()', () => { + it('builds an AWS adapter targeting the report executor with a report-shaped message', () => { + process.env.SCHEDULER_PROVIDER = 'aws'; + process.env.REPORT_EXECUTOR_LAMBDA_ARN = + 'arn:aws:lambda:us-east-1:123456789012:function:report-executor'; + process.env.REPORT_SCHEDULE_GROUP = 'frigg-admin-scripts'; + process.env.SCHEDULER_ROLE_ARN = awsAdapterParams.roleArn; + + const adapter = createReportSchedulerAdapterFromEnv({ + reportName: 'integrations', + mode: 'snapshot', + }); + + expect(adapter).toBeInstanceOf(AWSSchedulerAdapter); + expect(adapter.targetLambdaArn).toBe( + 'arn:aws:lambda:us-east-1:123456789012:function:report-executor' + ); + expect(adapter.scheduleNameFor('integrations')).toBe( + 'frigg-report-integrations' + ); + expect(adapter.buildInput({ input: { a: 1 } })).toEqual({ + reportName: 'integrations', + mode: 'snapshot', + trigger: 'SCHEDULED', + params: { a: 1 }, + }); + }); + + it('throws 503 when SCHEDULER_PROVIDER is unset in a deployed Lambda', () => { + delete process.env.SCHEDULER_PROVIDER; + process.env.AWS_LAMBDA_FUNCTION_NAME = 'report-router'; + + let error; + try { + createReportSchedulerAdapterFromEnv({ + reportName: 'x', + mode: 'snapshot', + }); + } catch (e) { + error = e; + } + + expect(error).toBeDefined(); + expect(error.isBoom).toBe(true); + expect(error.output.statusCode).toBe(503); + }); + }); }); diff --git a/packages/admin-scripts/src/adapters/aws-scheduler-adapter.js b/packages/admin-scripts/src/adapters/aws-scheduler-adapter.js index bb51d2ace..37e3ee3ab 100644 --- a/packages/admin-scripts/src/adapters/aws-scheduler-adapter.js +++ b/packages/admin-scripts/src/adapters/aws-scheduler-adapter.js @@ -28,12 +28,22 @@ function loadSchedulerSDK() { * Implements scheduling using AWS EventBridge Scheduler. * Supports cron expressions, timezone configuration, and Lambda invocation. */ +// Prefix and input builder are parameterized so one adapter can target either the script or report executor. +const DEFAULT_NAME_PREFIX = 'frigg-script-'; +const defaultBuildInput = ({ scriptName, input }) => ({ + scriptName, + trigger: 'SCHEDULED', + params: input || {}, +}); + class AWSSchedulerAdapter extends SchedulerAdapter { constructor({ credentials, targetLambdaArn, scheduleGroupName, roleArn, + namePrefix, + buildInput, } = {}) { super(); if (!targetLambdaArn) @@ -52,9 +62,15 @@ class AWSSchedulerAdapter extends SchedulerAdapter { this.targetLambdaArn = targetLambdaArn; this.scheduleGroupName = scheduleGroupName; this.roleArn = roleArn; + this.namePrefix = namePrefix || DEFAULT_NAME_PREFIX; + this.buildInput = buildInput || defaultBuildInput; this.scheduler = null; } + scheduleNameFor(scriptName) { + return `${this.namePrefix}${scriptName}`; + } + getSchedulerClient() { if (!this.scheduler) { loadSchedulerSDK(); @@ -72,7 +88,7 @@ class AWSSchedulerAdapter extends SchedulerAdapter { async createSchedule({ scriptName, cronExpression, timezone, input }) { const client = this.getSchedulerClient(); - const scheduleName = `frigg-script-${scriptName}`; + const scheduleName = this.scheduleNameFor(scriptName); const scheduleParams = { Name: scheduleName, @@ -83,11 +99,7 @@ class AWSSchedulerAdapter extends SchedulerAdapter { Target: { Arn: this.targetLambdaArn, RoleArn: this.roleArn, - Input: JSON.stringify({ - scriptName, - trigger: 'SCHEDULED', - params: input || {}, - }), + Input: JSON.stringify(this.buildInput({ scriptName, input })), }, State: 'ENABLED', }; @@ -116,7 +128,7 @@ class AWSSchedulerAdapter extends SchedulerAdapter { async deleteSchedule(scriptName) { const client = this.getSchedulerClient(); - const scheduleName = `frigg-script-${scriptName}`; + const scheduleName = this.scheduleNameFor(scriptName); await client.send( new DeleteScheduleCommand({ @@ -128,7 +140,7 @@ class AWSSchedulerAdapter extends SchedulerAdapter { async setScheduleEnabled(scriptName, enabled) { const client = this.getSchedulerClient(); - const scheduleName = `frigg-script-${scriptName}`; + const scheduleName = this.scheduleNameFor(scriptName); // Get the current schedule first to preserve all settings const getCommand = new GetScheduleCommand({ @@ -167,7 +179,7 @@ class AWSSchedulerAdapter extends SchedulerAdapter { async getSchedule(scriptName) { const client = this.getSchedulerClient(); - const scheduleName = `frigg-script-${scriptName}`; + const scheduleName = this.scheduleNameFor(scriptName); const response = await client.send( new GetScheduleCommand({ diff --git a/packages/admin-scripts/src/adapters/scheduler-adapter-factory.js b/packages/admin-scripts/src/adapters/scheduler-adapter-factory.js index 3bcb977c7..3f5b2a888 100644 --- a/packages/admin-scripts/src/adapters/scheduler-adapter-factory.js +++ b/packages/admin-scripts/src/adapters/scheduler-adapter-factory.js @@ -39,6 +39,8 @@ function createSchedulerAdapter(options = {}) { targetLambdaArn: options.targetLambdaArn, scheduleGroupName: options.scheduleGroupName, roleArn: options.roleArn, + namePrefix: options.namePrefix, + buildInput: options.buildInput, }); case 'local': @@ -79,7 +81,45 @@ function createSchedulerAdapterFromEnv() { }); } +/** + * Resolve and build a scheduler adapter that targets the report executor Lambda. + * + * A distinct name prefix keeps report schedules from colliding with script + * schedules in the shared EventBridge group. + * + * @param {Object} params + * @param {string} params.reportName - Registered report name (also the ScriptSchedule key). + * @param {string} params.mode - Run mode for the scheduled invocation (e.g. 'snapshot'). + * @returns {SchedulerAdapter} + * @throws {Boom.Boom} 503 when SCHEDULER_PROVIDER is unset in a deployed Lambda. + */ +function createReportSchedulerAdapterFromEnv({ reportName, mode }) { + const type = + process.env.SCHEDULER_PROVIDER || + (process.env.AWS_LAMBDA_FUNCTION_NAME ? null : 'local'); + if (!type) { + throw Boom.serverUnavailable( + 'SCHEDULER_PROVIDER is not configured. Set it (e.g. "aws") via appDefinition.admin.enableScheduling.' + ); + } + + return createSchedulerAdapter({ + type, + targetLambdaArn: process.env.REPORT_EXECUTOR_LAMBDA_ARN, + scheduleGroupName: process.env.REPORT_SCHEDULE_GROUP, + roleArn: process.env.SCHEDULER_ROLE_ARN, + namePrefix: 'frigg-report-', + buildInput: ({ input }) => ({ + reportName, + mode, + trigger: 'SCHEDULED', + params: input || {}, + }), + }); +} + module.exports = { createSchedulerAdapter, createSchedulerAdapterFromEnv, + createReportSchedulerAdapterFromEnv, }; diff --git a/packages/admin-scripts/src/application/__tests__/report-runner.test.js b/packages/admin-scripts/src/application/__tests__/report-runner.test.js new file mode 100644 index 000000000..3ed061b0a --- /dev/null +++ b/packages/admin-scripts/src/application/__tests__/report-runner.test.js @@ -0,0 +1,428 @@ +const { createReportRunner } = require('../report-runner'); +const { ScriptFactory } = require('../script-factory'); + +class FakeReport { + static Definition = { + name: 'fake', + version: '1.0.0', + runModes: ['live', 'recorded', 'snapshot'], + output: { format: 'json' }, + inputSchema: { + type: 'object', + properties: { n: { type: 'integer' } }, + }, + }; + + constructor(params = {}) { + this.context = params.context || null; + } + + async execute(frigg, params) { + return { ok: true, n: params.n ?? 0, friggReceived: frigg }; + } +} + +class CsvReport { + static Definition = { + name: 'csv', + version: '1.0.0', + runModes: ['recorded'], + output: { format: 'csv' }, + }; + async execute() { + return { file: 'a,b\n1,2\n', summary: { rows: 1 }, fileType: 'csv' }; + } +} + +const FRIGG = { integrations: {}, integrationMappings: {}, usage: {} }; + +function makeRunner() { + const reportCommands = { + createExecution: jest.fn(), + completeExecution: jest.fn(), + updateExecutionState: jest.fn(), + }; + const runner = createReportRunner({ + reportFactory: new ScriptFactory([FakeReport, CsvReport]), + reportCommands, + friggCommands: FRIGG, + }); + return { runner, reportCommands }; +} + +describe('ReportRunner', () => { + it('requires a reportFactory', () => { + expect(() => createReportRunner({})).toThrow(/reportFactory/); + }); + + describe('live mode', () => { + it('computes inline, returns COMPLETED, and persists NOTHING', async () => { + const { runner, reportCommands } = makeRunner(); + + const result = await runner.execute('fake', { n: 5 }, { mode: 'live' }); + + expect(result.status).toBe('COMPLETED'); + expect(result.mode).toBe('live'); + expect(result.output).toMatchObject({ ok: true, n: 5 }); + // The report reads via the injected frigg command bundle. + expect(result.output.friggReceived).toBe(FRIGG); + // The core invariant: live creates no execution record. + expect(reportCommands.createExecution).not.toHaveBeenCalled(); + expect(reportCommands.completeExecution).not.toHaveBeenCalled(); + expect(reportCommands.updateExecutionState).not.toHaveBeenCalled(); + }); + + it('defaults to the first runMode when none is given (live)', async () => { + const { runner, reportCommands } = makeRunner(); + + const result = await runner.execute('fake', {}); + + expect(result.mode).toBe('live'); + expect(reportCommands.createExecution).not.toHaveBeenCalled(); + }); + }); + + it('rejects a mode the report does not allow', async () => { + const { runner } = makeRunner(); + await expect( + runner.execute('fake', {}, { mode: 'bogus' }) + ).rejects.toMatchObject({ code: 'INVALID_MODE' }); + }); + + it('rejects input that violates the inputSchema', async () => { + const { runner } = makeRunner(); + await expect( + runner.execute('fake', { n: 'not-a-number' }, { mode: 'live' }) + ).rejects.toMatchObject({ code: 'INVALID_INPUT' }); + }); + + it('rejects a non-JSON output format in live mode (JSON-only inline)', async () => { + class LiveCsv { + static Definition = { + name: 'live-csv', + version: '1.0.0', + runModes: ['live'], + output: { format: 'csv' }, + }; + async execute() { + return {}; + } + } + const runner = createReportRunner({ + reportFactory: new ScriptFactory([LiveCsv]), + friggCommands: FRIGG, + }); + await expect( + runner.execute('live-csv', {}, { mode: 'live' }) + ).rejects.toMatchObject({ code: 'ARTIFACT_STORAGE_UNAVAILABLE' }); + }); + + describe('non-JSON artifact output (recorded / snapshot)', () => { + it('stores the file and completes with summary + artifact ref', async () => { + const reportCommands = { + createExecution: jest.fn().mockResolvedValue({ id: 'exec-a' }), + updateExecutionState: jest.fn().mockResolvedValue({}), + completeExecution: jest.fn().mockResolvedValue({ success: true }), + }; + const artifactRepository = { + put: jest.fn().mockResolvedValue({ + bucket: 'bkt', + key: 'reports/exec-a/csv.csv', + }), + }; + const runner = createReportRunner({ + reportFactory: new ScriptFactory([CsvReport]), + reportCommands, + friggCommands: FRIGG, + artifactRepository, + }); + + const result = await runner.execute( + 'csv', + {}, + { mode: 'recorded', trigger: 'MANUAL' } + ); + + expect(result.status).toBe('COMPLETED'); + expect(result.mode).toBe('recorded'); + expect(result.artifact).toEqual({ + bucket: 'bkt', + key: 'reports/exec-a/csv.csv', + }); + expect(result.summary).toEqual({ rows: 1 }); + // No inline output for non-json. + expect(result.output).toBeUndefined(); + + expect(artifactRepository.put).toHaveBeenCalledTimes(1); + const [key, body, contentType] = + artifactRepository.put.mock.calls[0]; + expect(key).toMatch(/^reports\/exec-a\/csv-.+\.csv$/); + expect(body).toBe('a,b\n1,2\n'); + expect(contentType).toBe('text/csv'); + + expect(reportCommands.completeExecution).toHaveBeenCalledWith( + 'exec-a', + expect.objectContaining({ + state: 'COMPLETED', + summary: { rows: 1 }, + artifact: { bucket: 'bkt', key: 'reports/exec-a/csv.csv' }, + }) + ); + }); + + it('records FAILED when artifact storage throws', async () => { + const reportCommands = { + createExecution: jest.fn().mockResolvedValue({ id: 'exec-b' }), + updateExecutionState: jest.fn().mockResolvedValue({}), + completeExecution: jest.fn().mockResolvedValue({ success: true }), + }; + const artifactRepository = { + put: jest.fn().mockRejectedValue(new Error('S3 down')), + }; + const runner = createReportRunner({ + reportFactory: new ScriptFactory([CsvReport]), + reportCommands, + friggCommands: FRIGG, + artifactRepository, + }); + + const result = await runner.execute( + 'csv', + {}, + { mode: 'recorded', trigger: 'MANUAL' } + ); + + expect(result.status).toBe('FAILED'); + expect(result.error.message).toBe('S3 down'); + expect(reportCommands.completeExecution).toHaveBeenCalledWith( + 'exec-b', + expect.objectContaining({ state: 'FAILED' }) + ); + }); + }); + + describe('recorded / snapshot modes', () => { + it('creates a record, marks RUNNING, then completes COMPLETED', async () => { + const { runner, reportCommands } = makeRunner(); + reportCommands.createExecution.mockResolvedValue({ id: 'exec-1' }); + reportCommands.updateExecutionState.mockResolvedValue({}); + reportCommands.completeExecution.mockResolvedValue({ success: true }); + + const result = await runner.execute( + 'fake', + { n: 2 }, + { mode: 'recorded', trigger: 'MANUAL' } + ); + + expect(result.status).toBe('COMPLETED'); + expect(result.mode).toBe('recorded'); + expect(result.executionId).toBe('exec-1'); + expect(result.output).toMatchObject({ ok: true, n: 2 }); + + expect(reportCommands.createExecution).toHaveBeenCalledTimes(1); + expect(reportCommands.updateExecutionState).toHaveBeenCalledWith( + 'exec-1', + 'RUNNING' + ); + expect(reportCommands.completeExecution).toHaveBeenCalledWith( + 'exec-1', + expect.objectContaining({ state: 'COMPLETED', output: result.output }) + ); + }); + + it('passes seriesName for snapshot mode (defaulting to report name)', async () => { + const { runner, reportCommands } = makeRunner(); + reportCommands.createExecution.mockResolvedValue({ id: 'exec-2' }); + reportCommands.updateExecutionState.mockResolvedValue({}); + reportCommands.completeExecution.mockResolvedValue({ success: true }); + + await runner.execute('fake', {}, { mode: 'snapshot', trigger: 'SCHEDULED' }); + + expect(reportCommands.createExecution).toHaveBeenCalledWith( + expect.objectContaining({ mode: 'snapshot', seriesName: 'fake' }) + ); + }); + + it('does NOT set seriesName for recorded mode', async () => { + const { runner, reportCommands } = makeRunner(); + reportCommands.createExecution.mockResolvedValue({ id: 'exec-3' }); + reportCommands.updateExecutionState.mockResolvedValue({}); + reportCommands.completeExecution.mockResolvedValue({ success: true }); + + await runner.execute('fake', {}, { mode: 'recorded', trigger: 'MANUAL' }); + + const arg = reportCommands.createExecution.mock.calls[0][0]; + expect(arg.seriesName).toBeUndefined(); + }); + + it('resumes an existing record without creating a new one', async () => { + const { runner, reportCommands } = makeRunner(); + reportCommands.updateExecutionState.mockResolvedValue({}); + reportCommands.completeExecution.mockResolvedValue({ success: true }); + + const result = await runner.execute( + 'fake', + {}, + { mode: 'recorded', trigger: 'QUEUE', executionId: 'given-1' } + ); + + expect(reportCommands.createExecution).not.toHaveBeenCalled(); + expect(result.executionId).toBe('given-1'); + expect(reportCommands.updateExecutionState).toHaveBeenCalledWith( + 'given-1', + 'RUNNING' + ); + }); + + it('records FAILED when the report throws', async () => { + class BoomReport { + static Definition = { + name: 'boom', + version: '1.0.0', + runModes: ['recorded'], + output: { format: 'json' }, + }; + async execute() { + throw new Error('kaboom'); + } + } + const reportCommands = { + createExecution: jest.fn().mockResolvedValue({ id: 'exec-9' }), + updateExecutionState: jest.fn().mockResolvedValue({}), + completeExecution: jest.fn().mockResolvedValue({ success: true }), + }; + const runner = createReportRunner({ + reportFactory: new ScriptFactory([BoomReport]), + reportCommands, + friggCommands: FRIGG, + }); + + const result = await runner.execute( + 'boom', + {}, + { mode: 'recorded', trigger: 'MANUAL' } + ); + + expect(result.status).toBe('FAILED'); + expect(result.error.message).toBe('kaboom'); + expect(reportCommands.completeExecution).toHaveBeenCalledWith( + 'exec-9', + expect.objectContaining({ state: 'FAILED' }) + ); + }); + + it('throws when the execution record cannot be created', async () => { + const { runner, reportCommands } = makeRunner(); + reportCommands.createExecution.mockResolvedValue({ + error: 500, + reason: 'DB down', + }); + + await expect( + runner.execute('fake', {}, { mode: 'recorded', trigger: 'MANUAL' }) + ).rejects.toThrow('DB down'); + }); + }); + + describe('self-requeue continuation', () => { + // Opts into chunking: yields a continuation marker while the Lambda is + // low on time and no resume state has arrived yet; otherwise finishes. + class ChunkedReport { + static Definition = { + name: 'chunked', + version: '1.0.0', + runModes: ['recorded'], + output: { format: 'json' }, + }; + async execute(frigg, params, context) { + const remaining = context.getRemainingTimeInMillis(); + if (remaining < 60000 && !params.__resume) { + return { __continuation: { cursor: 42 } }; + } + return { done: true }; + } + } + + function makeChunkedRunner() { + const reportCommands = { + createExecution: jest + .fn() + .mockResolvedValue({ id: 'exec-c' }), + updateExecutionState: jest.fn().mockResolvedValue({}), + completeExecution: jest.fn().mockResolvedValue({ success: true }), + appendExecutionLog: jest.fn().mockResolvedValue({}), + }; + const runner = createReportRunner({ + reportFactory: new ScriptFactory([ChunkedReport]), + reportCommands, + friggCommands: FRIGG, + }); + return { runner, reportCommands }; + } + + it('returns CONTINUE without completing when the report yields on low time', async () => { + const { runner, reportCommands } = makeChunkedRunner(); + const lambdaContext = { + getRemainingTimeInMillis: jest.fn().mockReturnValue(5000), + }; + + const result = await runner.execute( + 'chunked', + {}, + { mode: 'recorded', trigger: 'QUEUE', lambdaContext } + ); + + expect(result.status).toBe('CONTINUE'); + expect(result.executionId).toBe('exec-c'); + expect(result.continuation).toEqual({ cursor: 42 }); + // Stays RUNNING — not completed — between hops. + expect(reportCommands.completeExecution).not.toHaveBeenCalled(); + expect(reportCommands.appendExecutionLog).toHaveBeenCalledWith( + 'exec-c', + expect.objectContaining({ + message: expect.stringMatching(/continuation/i), + }) + ); + }); + + it('completes normally when time is ample (no continuation)', async () => { + const { runner, reportCommands } = makeChunkedRunner(); + const lambdaContext = { + getRemainingTimeInMillis: jest.fn().mockReturnValue(900000), + }; + + const result = await runner.execute( + 'chunked', + {}, + { mode: 'recorded', trigger: 'QUEUE', lambdaContext } + ); + + expect(result.status).toBe('COMPLETED'); + expect(reportCommands.completeExecution).toHaveBeenCalledWith( + 'exec-c', + expect.objectContaining({ state: 'COMPLETED' }) + ); + }); + + it('finishes when resumed with the prior continuation state', async () => { + const { runner, reportCommands } = makeChunkedRunner(); + const lambdaContext = { + getRemainingTimeInMillis: jest.fn().mockReturnValue(5000), + }; + + const result = await runner.execute( + 'chunked', + { __resume: { cursor: 42 } }, + { + mode: 'recorded', + trigger: 'QUEUE', + executionId: 'exec-c', + lambdaContext, + } + ); + + expect(result.status).toBe('COMPLETED'); + expect(reportCommands.createExecution).not.toHaveBeenCalled(); + }); + }); +}); diff --git a/packages/admin-scripts/src/application/admin-script-context.js b/packages/admin-scripts/src/application/admin-script-context.js index 93678e77f..e7b3dd23e 100644 --- a/packages/admin-scripts/src/application/admin-script-context.js +++ b/packages/admin-scripts/src/application/admin-script-context.js @@ -35,6 +35,18 @@ class AdminScriptContext { // Injected by bootstrap.js — the context never reaches for repositories // or command factories itself. this.commands = params.commands || null; + + // Set by the executor; reports read getRemainingTimeInMillis() to yield + // before the Lambda timeout and self-requeue. + this.lambdaContext = params.lambdaContext || null; + } + + // Infinity when there is no Lambda context (live runs, local dev, tests). + getRemainingTimeInMillis() { + return this.lambdaContext && + typeof this.lambdaContext.getRemainingTimeInMillis === 'function' + ? this.lambdaContext.getRemainingTimeInMillis() + : Infinity; } // ==================== INTEGRATION INSTANTIATION ==================== diff --git a/packages/admin-scripts/src/application/report-runner.js b/packages/admin-scripts/src/application/report-runner.js new file mode 100644 index 000000000..21d968b5a --- /dev/null +++ b/packages/admin-scripts/src/application/report-runner.js @@ -0,0 +1,273 @@ +const { createAdminScriptContext } = require('./admin-script-context'); +const { validateParams } = require('./validate-script-input'); + +const ARTIFACT_CONTENT_TYPES = { + csv: 'text/csv', + json: 'application/json', + pdf: 'application/pdf', + xlsx: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + html: 'text/html', + txt: 'text/plain', + xml: 'application/xml', + zip: 'application/zip', +}; + +// The run mode decides persistence: live computes inline and records nothing, +// recorded persists an execution record, snapshot additionally tags the run +// into a named series. Reports read only through the injected frigg command +// bundle (exposed as context.commands). +class ReportRunner { + constructor(params = {}) { + if (!params.reportFactory) { + throw new Error('ReportRunner requires a reportFactory'); + } + this.reportFactory = params.reportFactory; + this.reportCommands = params.reportCommands || null; + this.friggCommands = params.friggCommands || null; + this.integrationFactory = params.integrationFactory || null; + this.artifactRepository = params.artifactRepository || null; + } + + _getArtifactRepository() { + if (!this.artifactRepository) { + const { + createArtifactRepository, + } = require('@friggframework/core/artifacts/repositories/artifact-repository-factory'); + this.artifactRepository = createArtifactRepository(); + } + return this.artifactRepository; + } + + async execute(reportName, params = {}, options = {}) { + const ReportClass = this.reportFactory.get(reportName); + const definition = ReportClass.Definition; + + const runModes = + Array.isArray(definition.runModes) && definition.runModes.length + ? definition.runModes + : ['live']; + const mode = options.mode || runModes[0]; + if (!runModes.includes(mode)) { + const error = new Error( + `Report "${reportName}" does not support mode "${mode}". Allowed: ${runModes.join( + ', ' + )}` + ); + error.code = 'INVALID_MODE'; + throw error; + } + + const validation = validateParams(definition, params); + if (!validation.valid) { + const error = new Error( + `Invalid input: ${validation.errors.join(', ')}` + ); + error.code = 'INVALID_INPUT'; + throw error; + } + + const format = definition.output?.format || 'json'; + + if (mode === 'live') { + // Non-JSON can't be returned inline as a response body; it needs an + // artifact, so it must run recorded/snapshot. + if (format !== 'json') { + const error = new Error( + `Report "${reportName}" output format "${format}" cannot be returned inline; run it in recorded or snapshot mode` + ); + error.code = 'ARTIFACT_STORAGE_UNAVAILABLE'; + throw error; + } + return this._runLive(reportName, params); + } + + return this._runRecorded(reportName, definition, params, mode, options); + } + + async _runLive(reportName, params) { + const startTime = new Date(); + // executionId null: live persists nothing, so record-dependent logging + // and chaining are intentionally inert. + const context = createAdminScriptContext({ + executionId: null, + integrationFactory: this.integrationFactory, + commands: this.friggCommands, + }); + + const report = this.reportFactory.createInstance(reportName, { + context, + executionId: null, + integrationFactory: this.integrationFactory, + }); + + const output = await report.execute(context.commands, params, context); + + return { + status: 'COMPLETED', + reportName, + mode: 'live', + output, + metrics: { durationMs: new Date() - startTime }, + }; + } + + // Completion is written OUTSIDE the try on success so a persistence failure + // is never misreported as a report failure. + async _runRecorded(reportName, definition, params, mode, options = {}) { + let executionId = options.executionId; + + if (!executionId) { + const execution = await this.reportCommands.createExecution({ + reportName, + reportVersion: definition.version, + trigger: options.trigger || 'MANUAL', + mode, + input: params, + audit: options.audit, + seriesName: + mode === 'snapshot' + ? options.seriesName || reportName + : undefined, + parentExecutionId: options.parentExecutionId, + }); + if (execution.error) { + throw new Error( + execution.reason || + 'Failed to create report execution record' + ); + } + executionId = execution.id; + } + + const startTime = new Date(); + const context = createAdminScriptContext({ + executionId, + integrationFactory: this.integrationFactory, + commands: this.friggCommands, + lambdaContext: options.lambdaContext, + }); + + const format = definition.output?.format || 'json'; + let output; + let artifact = null; + let summary; + try { + await this.reportCommands.updateExecutionState( + executionId, + 'RUNNING' + ); + + const report = this.reportFactory.createInstance(reportName, { + context, + executionId, + integrationFactory: this.integrationFactory, + }); + + output = await report.execute(context.commands, params, context); + + // Store non-JSON output as an artifact inside the try so a put + // failure marks the run FAILED (a continuation yield has none yet). + if (format !== 'json' && !(output && output.__continuation)) { + const stamp = new Date() + .toISOString() + .replace(/[:.]/g, '-'); + const fileType = output.fileType || format; + const key = `reports/${executionId}/${reportName}-${stamp}.${fileType}`; + const contentType = + ARTIFACT_CONTENT_TYPES[fileType] || + 'application/octet-stream'; + artifact = await this._getArtifactRepository().put( + key, + output.file, + contentType + ); + summary = output.summary; + } + } catch (error) { + const durationMs = new Date() - startTime; + await this.reportCommands.completeExecution(executionId, { + state: 'FAILED', + error: { + name: error.name, + message: error.message, + stack: error.stack, + }, + metrics: { + startTime: startTime.toISOString(), + endTime: new Date().toISOString(), + durationMs, + }, + logs: context.getLogs(), + }); + + return { + executionId, + status: 'FAILED', + reportName, + mode, + error: { name: error.name, message: error.message }, + metrics: { durationMs }, + }; + } + + // Opt-in self-requeue: a report yields by returning a truthy + // `__continuation` (its resume state) instead of a final result. The + // execution stays RUNNING and the executor re-enqueues the SAME id, so + // the report resumes from the marker on the next invocation. + if (output && output.__continuation) { + const resumeState = output.__continuation; + if (typeof this.reportCommands.appendExecutionLog === 'function') { + await this.reportCommands.appendExecutionLog(executionId, { + level: 'info', + message: 'report yielded a continuation; re-queueing', + data: { resumeAt: new Date().toISOString() }, + timestamp: new Date().toISOString(), + }); + } + return { + executionId, + status: 'CONTINUE', + reportName, + mode, + continuation: resumeState, + metrics: { durationMs: new Date() - startTime }, + }; + } + + const durationMs = new Date() - startTime; + const isArtifact = format !== 'json'; + const result = { + executionId, + status: 'COMPLETED', + reportName, + mode, + ...(isArtifact ? { summary, artifact } : { output }), + metrics: { durationMs }, + }; + + const completion = await this.reportCommands.completeExecution( + executionId, + { + state: 'COMPLETED', + ...(isArtifact ? { summary, artifact } : { output }), + metrics: { + startTime: startTime.toISOString(), + endTime: new Date().toISOString(), + durationMs, + }, + logs: context.getLogs(), + } + ); + if (completion?.error) { + result.stateUpdateFailed = true; + } + + return result; + } +} + +function createReportRunner(params = {}) { + return new ReportRunner(params); +} + +module.exports = { ReportRunner, createReportRunner }; diff --git a/packages/admin-scripts/src/infrastructure/__tests__/report-executor-handler.test.js b/packages/admin-scripts/src/infrastructure/__tests__/report-executor-handler.test.js new file mode 100644 index 000000000..4046d5e34 --- /dev/null +++ b/packages/admin-scripts/src/infrastructure/__tests__/report-executor-handler.test.js @@ -0,0 +1,314 @@ +jest.mock('../bootstrap'); +jest.mock('../../application/report-runner'); +jest.mock('@friggframework/core/application/commands/report-commands'); +jest.mock('@friggframework/core/queues', () => ({ + QueuerUtil: { send: jest.fn().mockResolvedValue({}) }, +})); + +const { bootstrapAdminScripts } = require('../bootstrap'); +const { createReportRunner } = require('../../application/report-runner'); +const { + createReportCommands, +} = require('@friggframework/core/application/commands/report-commands'); +const { QueuerUtil } = require('@friggframework/core/queues'); +const { handler } = require('../report-executor-handler'); + +describe('Report Executor Handler', () => { + let mockReportFactory; + let mockIntegrationFactory; + let mockReportFriggCommands; + let mockRunner; + let mockCommands; + + beforeEach(() => { + mockReportFactory = { id: 'report-factory' }; + mockIntegrationFactory = { id: 'integration-factory' }; + mockReportFriggCommands = { id: 'report-frigg-commands' }; + mockRunner = { execute: jest.fn() }; + mockCommands = { + completeExecution: jest.fn().mockResolvedValue({}), + }; + + bootstrapAdminScripts.mockReturnValue({ + reportFactory: mockReportFactory, + reportCommands: { id: 'report-commands' }, + reportFriggCommands: mockReportFriggCommands, + integrationFactory: mockIntegrationFactory, + }); + createReportRunner.mockReturnValue(mockRunner); + createReportCommands.mockReturnValue(mockCommands); + + jest.spyOn(console, 'log').mockImplementation(() => {}); + jest.spyOn(console, 'error').mockImplementation(() => {}); + }); + + afterEach(() => { + console.log.mockRestore(); + console.error.mockRestore(); + jest.clearAllMocks(); + }); + + describe('EventBridge Scheduler direct invoke (no Records)', () => { + it('injects the bootstrapped factory into the runner and reports the result', async () => { + mockRunner.execute.mockResolvedValue({ + status: 'COMPLETED', + executionId: 'exec-1', + }); + + const response = await handler({ + reportName: 'my-report', + trigger: 'SCHEDULED', + mode: 'snapshot', + seriesName: 'daily', + params: { foo: 'bar' }, + }); + + expect(createReportRunner).toHaveBeenCalledWith({ + reportFactory: mockReportFactory, + reportCommands: { id: 'report-commands' }, + friggCommands: mockReportFriggCommands, + integrationFactory: mockIntegrationFactory, + }); + expect(mockRunner.execute).toHaveBeenCalledWith( + 'my-report', + { foo: 'bar' }, + expect.objectContaining({ + trigger: 'SCHEDULED', + mode: 'snapshot', + seriesName: 'daily', + }) + ); + + expect(response.statusCode).toBe(200); + const body = JSON.parse(response.body); + expect(body.processed).toBe(1); + expect(body.results[0]).toEqual({ + reportName: 'my-report', + status: 'COMPLETED', + executionId: 'exec-1', + }); + }); + + it('defaults mode to recorded and trigger to QUEUE', async () => { + mockRunner.execute.mockResolvedValue({ + status: 'COMPLETED', + executionId: 'exec-d', + }); + + await handler({ reportName: 'my-report', params: {} }); + + expect(mockRunner.execute).toHaveBeenCalledWith( + 'my-report', + {}, + expect.objectContaining({ mode: 'recorded', trigger: 'QUEUE' }) + ); + }); + + it('marks the execution FAILED when the runner throws', async () => { + mockRunner.execute.mockRejectedValue(new Error('boom')); + + const response = await handler({ + reportName: 'my-report', + executionId: 'exec-2', + trigger: 'SCHEDULED', + }); + + expect(response.statusCode).toBe(200); + const body = JSON.parse(response.body); + expect(body.results[0].status).toBe('FAILED'); + expect(mockCommands.completeExecution).toHaveBeenCalledWith( + 'exec-2', + expect.objectContaining({ state: 'FAILED' }) + ); + }); + }); + + describe('SQS batch (Records[])', () => { + it('injects the bootstrapped factory and processes each record', async () => { + mockRunner.execute.mockResolvedValue({ + status: 'COMPLETED', + executionId: 'exec-3', + }); + + const response = await handler({ + Records: [ + { + body: JSON.stringify({ + reportName: 'my-report', + executionId: 'exec-3', + mode: 'recorded', + params: {}, + }), + }, + ], + }); + + expect(createReportRunner).toHaveBeenCalledWith({ + reportFactory: mockReportFactory, + reportCommands: { id: 'report-commands' }, + friggCommands: mockReportFriggCommands, + integrationFactory: mockIntegrationFactory, + }); + expect(mockRunner.execute).toHaveBeenCalledWith( + 'my-report', + {}, + expect.objectContaining({ + trigger: 'QUEUE', + mode: 'recorded', + executionId: 'exec-3', + }) + ); + + expect(response.statusCode).toBe(200); + const body = JSON.parse(response.body); + expect(body.processed).toBe(1); + expect(body.results[0].status).toBe('COMPLETED'); + }); + + it('isolates a bad record without dropping the rest of the batch', async () => { + mockRunner.execute.mockResolvedValue({ + status: 'COMPLETED', + executionId: 'exec-4', + }); + + const response = await handler({ + Records: [ + // Missing reportName -> runMessage throws before the runner + { body: JSON.stringify({ executionId: 'exec-bad' }) }, + { + body: JSON.stringify({ + reportName: 'my-report', + executionId: 'exec-4', + params: {}, + }), + }, + ], + }); + + const body = JSON.parse(response.body); + expect(body.processed).toBe(2); + expect(body.results[0].status).toBe('FAILED'); + expect(body.results[1].status).toBe('COMPLETED'); + expect(mockCommands.completeExecution).toHaveBeenCalledWith( + 'exec-bad', + expect.objectContaining({ state: 'FAILED' }) + ); + }); + }); + + describe('self-requeue on CONTINUE', () => { + afterEach(() => { + delete process.env.REPORT_QUEUE_URL; + }); + + it('passes the Lambda context to the runner', async () => { + mockRunner.execute.mockResolvedValue({ + status: 'COMPLETED', + executionId: 'exec-ctx', + }); + const lambdaContext = { + getRemainingTimeInMillis: () => 12345, + }; + + await handler( + { reportName: 'my-report', executionId: 'exec-ctx', params: {} }, + lambdaContext + ); + + expect(mockRunner.execute).toHaveBeenCalledWith( + 'my-report', + {}, + expect.objectContaining({ lambdaContext }) + ); + }); + + it('re-enqueues the same execution with a resume marker when the runner returns CONTINUE', async () => { + process.env.REPORT_QUEUE_URL = 'https://sqs.local/report-queue'; + mockRunner.execute.mockResolvedValue({ + status: 'CONTINUE', + executionId: 'exec-cont', + continuation: { cursor: 7 }, + }); + + const response = await handler( + { + reportName: 'my-report', + executionId: 'exec-cont', + mode: 'recorded', + params: { since: '2026-01-01' }, + }, + { getRemainingTimeInMillis: () => 5000 } + ); + + expect(QueuerUtil.send).toHaveBeenCalledWith( + expect.objectContaining({ + reportName: 'my-report', + executionId: 'exec-cont', + mode: 'recorded', + trigger: 'QUEUE', + params: { + since: '2026-01-01', + __resume: { cursor: 7 }, + }, + resumeAt: expect.any(String), + }), + 'https://sqs.local/report-queue' + ); + + const body = JSON.parse(response.body); + expect(body.results[0].status).toBe('CONTINUE'); + }); + + it('fails the record when a continuation is yielded but REPORT_QUEUE_URL is unset', async () => { + delete process.env.REPORT_QUEUE_URL; + mockRunner.execute.mockResolvedValue({ + status: 'CONTINUE', + executionId: 'exec-noqueue', + continuation: { cursor: 1 }, + }); + + const response = await handler({ + reportName: 'my-report', + executionId: 'exec-noqueue', + params: {}, + }); + + expect(QueuerUtil.send).not.toHaveBeenCalled(); + const body = JSON.parse(response.body); + expect(body.results[0].status).toBe('FAILED'); + expect(mockCommands.completeExecution).toHaveBeenCalledWith( + 'exec-noqueue', + expect.objectContaining({ state: 'FAILED' }) + ); + }); + + it('compensates with the RUNNER-created id on a scheduled first run that cannot re-queue (no inbound executionId)', async () => { + // A real EventBridge scheduled event carries NO executionId — the + // runner creates the record. If the continuation can't be re-queued, + // the record must still be marked FAILED using the runner-created id, + // not the absent inbound one. + delete process.env.REPORT_QUEUE_URL; + mockRunner.execute.mockResolvedValue({ + status: 'CONTINUE', + executionId: 'runner-created-exec', + continuation: { cursor: 1 }, + }); + + const response = await handler({ + reportName: 'my-report', + trigger: 'SCHEDULED', + mode: 'snapshot', + params: {}, + // note: no executionId, matching the scheduler's buildInput + }); + + expect(QueuerUtil.send).not.toHaveBeenCalled(); + expect(mockCommands.completeExecution).toHaveBeenCalledWith( + 'runner-created-exec', + expect.objectContaining({ state: 'FAILED' }) + ); + const body = JSON.parse(response.body); + expect(body.results[0].status).toBe('FAILED'); + }); + }); +}); diff --git a/packages/admin-scripts/src/infrastructure/__tests__/report-router.test.js b/packages/admin-scripts/src/infrastructure/__tests__/report-router.test.js new file mode 100644 index 000000000..4e144ec2f --- /dev/null +++ b/packages/admin-scripts/src/infrastructure/__tests__/report-router.test.js @@ -0,0 +1,525 @@ +const request = require('supertest'); + +jest.mock('../admin-auth-middleware', () => ({ + validateAdminApiKey: (req, res, next) => next(), +})); +jest.mock('../bootstrap'); +jest.mock('@friggframework/core/queues', () => ({ + QueuerUtil: { send: jest.fn().mockResolvedValue({}) }, +})); +jest.mock('@friggframework/core/application/commands/admin-script-commands'); +jest.mock('../../adapters/scheduler-adapter-factory'); + +const { app } = require('../report-router'); +const { bootstrapAdminScripts } = require('../bootstrap'); +const { QueuerUtil } = require('@friggframework/core/queues'); +const { + createAdminScriptCommands, +} = require('@friggframework/core/application/commands/admin-script-commands'); +const { + createReportSchedulerAdapterFromEnv, +} = require('../../adapters/scheduler-adapter-factory'); +const { ScriptFactory } = require('../../application/script-factory'); + +class DemoReport { + static Definition = { + name: 'demo', + version: '1.0.0', + description: 'demo report', + runModes: ['live', 'recorded', 'snapshot'], + output: { format: 'json' }, + inputSchema: { + type: 'object', + properties: { n: { type: 'integer' } }, + }, + display: { category: 'reporting' }, + }; + async execute(frigg, params) { + return { ok: true, n: params.n ?? 0 }; + } +} + +class IntegrationsReport { + static Definition = { + name: 'integrations', + version: '1.0.0', + description: 'integrations report', + runModes: ['live', 'recorded', 'snapshot'], + output: { format: 'json' }, + display: { category: 'reporting' }, + }; + async execute(frigg, params) { + return { schemaVersion: 1, filters: params }; + } +} + +class ScheduledReport { + static Definition = { + name: 'scheduled', + version: '2.0.0', + runModes: ['recorded', 'snapshot'], + output: { format: 'json' }, + // Declares a preferred scheduled run mode. + schedule: { mode: 'recorded' }, + }; + async execute() { + return { ok: true }; + } +} + +describe('Report Router', () => { + let server; + let mockReportCommands; + let mockScheduleCommands; + let mockSchedulerAdapter; + + beforeAll((done) => { + server = app.listen(0, done); + }); + afterAll((done) => { + server.close(done); + }); + + beforeEach(() => { + process.env.REPORT_QUEUE_URL = 'https://sqs.local/report-queue'; + mockReportCommands = { + createExecution: jest + .fn() + .mockResolvedValue({ id: 'report-exec-1' }), + completeExecution: jest.fn().mockResolvedValue({ success: true }), + findExecutionById: jest.fn(), + findSnapshotSeries: jest.fn(), + }; + bootstrapAdminScripts.mockReturnValue({ + reportFactory: new ScriptFactory([ + DemoReport, + IntegrationsReport, + ScheduledReport, + ]), + reportFriggCommands: {}, + reportCommands: mockReportCommands, + integrationFactory: null, + }); + + mockScheduleCommands = { + getScheduleByScriptName: jest.fn().mockResolvedValue(null), + upsertSchedule: jest.fn(), + deleteSchedule: jest.fn(), + updateScheduleExternalInfo: jest.fn().mockResolvedValue({}), + }; + mockSchedulerAdapter = { + createSchedule: jest.fn(), + deleteSchedule: jest.fn(), + }; + createAdminScriptCommands.mockReturnValue(mockScheduleCommands); + createReportSchedulerAdapterFromEnv.mockReturnValue( + mockSchedulerAdapter + ); + + QueuerUtil.send.mockClear(); + }); + + afterEach(() => { + delete process.env.REPORT_QUEUE_URL; + jest.clearAllMocks(); + }); + + it('GET /api/v2/reports lists report definitions', async () => { + const res = await request(server).get('/api/v2/reports'); + expect(res.status).toBe(200); + expect(res.body.service).toBe('frigg-core-api'); + const names = res.body.reports.map((r) => r.name).sort(); + expect(names).toEqual(['demo', 'integrations', 'scheduled']); + }); + + it('GET /api/v2/reports/:name returns the definition detail', async () => { + const res = await request(server).get('/api/v2/reports/demo'); + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ + name: 'demo', + runModes: ['live', 'recorded', 'snapshot'], + output: { format: 'json' }, + }); + }); + + it('GET /api/v2/reports/:name 404s an unknown report', async () => { + const res = await request(server).get('/api/v2/reports/nope'); + expect(res.status).toBe(404); + expect(res.body.code).toBe('REPORT_NOT_FOUND'); + }); + + it('POST /:name/run live returns the runner envelope with inline output', async () => { + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'live', params: { n: 3 } }); + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ + status: 'COMPLETED', + mode: 'live', + output: { ok: true, n: 3 }, + }); + expect(QueuerUtil.send).not.toHaveBeenCalled(); + }); + + it('POST /:name/run 400s invalid input', async () => { + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'live', params: { n: 'not-a-number' } }); + expect(res.status).toBe(400); + expect(res.body.code).toBe('INVALID_INPUT'); + }); + + it('POST /:name/run recorded 400s invalid input WITHOUT creating a record or enqueueing', async () => { + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'recorded', params: { n: 'not-a-number' } }); + + expect(res.status).toBe(400); + expect(res.body.code).toBe('INVALID_INPUT'); + // The bad request must be rejected up front, not persisted + queued. + expect(mockReportCommands.createExecution).not.toHaveBeenCalled(); + expect(QueuerUtil.send).not.toHaveBeenCalled(); + }); + + it('POST /:name/run 400s an unsupported mode WITHOUT creating a record', async () => { + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'bogus', params: {} }); + + expect(res.status).toBe(400); + expect(res.body.code).toBe('INVALID_MODE'); + expect(mockReportCommands.createExecution).not.toHaveBeenCalled(); + expect(QueuerUtil.send).not.toHaveBeenCalled(); + }); + + it('POST /:name/run recorded creates a record and enqueues it (202)', async () => { + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'recorded', params: { n: 5 } }); + + expect(res.status).toBe(202); + expect(res.body).toEqual({ + executionId: 'report-exec-1', + status: 'QUEUED', + reportName: 'demo', + }); + expect(mockReportCommands.createExecution).toHaveBeenCalledWith( + expect.objectContaining({ + reportName: 'demo', + reportVersion: '1.0.0', + trigger: 'MANUAL', + mode: 'recorded', + input: { n: 5 }, + }) + ); + expect(QueuerUtil.send).toHaveBeenCalledWith( + expect.objectContaining({ + reportName: 'demo', + executionId: 'report-exec-1', + mode: 'recorded', + trigger: 'MANUAL', + params: { n: 5 }, + }), + 'https://sqs.local/report-queue' + ); + }); + + it('POST /:name/run snapshot forwards the seriesName', async () => { + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'snapshot', params: {}, seriesName: 'daily' }); + + expect(res.status).toBe(202); + expect(mockReportCommands.createExecution).toHaveBeenCalledWith( + expect.objectContaining({ mode: 'snapshot', seriesName: 'daily' }) + ); + expect(QueuerUtil.send).toHaveBeenCalledWith( + expect.objectContaining({ mode: 'snapshot', seriesName: 'daily' }), + 'https://sqs.local/report-queue' + ); + }); + + it('POST /:name/run 503s when REPORT_QUEUE_URL is not configured', async () => { + delete process.env.REPORT_QUEUE_URL; + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'recorded', params: {} }); + + expect(res.status).toBe(503); + expect(res.body.code).toBe('QUEUE_NOT_CONFIGURED'); + expect(QueuerUtil.send).not.toHaveBeenCalled(); + }); + + it('POST /:name/run surfaces a createExecution error and does not enqueue', async () => { + mockReportCommands.createExecution.mockResolvedValue({ + error: 500, + reason: 'db down', + code: 'DB_ERROR', + }); + + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'recorded', params: {} }); + + expect(res.status).toBe(500); + expect(QueuerUtil.send).not.toHaveBeenCalled(); + }); + + it('POST /:name/run marks the record FAILED if enqueue throws after createExecution', async () => { + QueuerUtil.send.mockRejectedValueOnce(new Error('sqs down')); + + const res = await request(server) + .post('/api/v2/reports/demo/run') + .send({ mode: 'recorded', params: {} }); + + expect(res.status).toBe(500); + // The persisted record must be compensated, not left non-terminal. + expect(mockReportCommands.completeExecution).toHaveBeenCalledWith( + 'report-exec-1', + expect.objectContaining({ state: 'FAILED' }) + ); + }); + + it('GET /api/v2/reports/executions/:id returns the report execution', async () => { + mockReportCommands.findExecutionById.mockResolvedValue({ + id: 'report-exec-1', + type: 'REPORT', + state: 'COMPLETED', + }); + + const res = await request(server).get( + '/api/v2/reports/executions/report-exec-1' + ); + + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ + id: 'report-exec-1', + state: 'COMPLETED', + }); + expect(mockReportCommands.findExecutionById).toHaveBeenCalledWith( + 'report-exec-1' + ); + }); + + it('GET /api/v2/reports/executions/:id 404s a non-report execution', async () => { + mockReportCommands.findExecutionById.mockResolvedValue({ + error: 404, + reason: 'Report execution x not found', + code: 'EXECUTION_NOT_FOUND', + }); + + const res = await request(server).get( + '/api/v2/reports/executions/x' + ); + + expect(res.status).toBe(404); + expect(res.body.code).toBe('EXECUTION_NOT_FOUND'); + }); + + it('GET /api/v2/reports/:name/snapshots returns the series', async () => { + mockReportCommands.findSnapshotSeries.mockResolvedValue([ + { executionId: 'e1', capturedAt: '2026-01-01', summary: null }, + ]); + + const res = await request(server) + .get('/api/v2/reports/demo/snapshots') + .query({ from: '2026-01-01', to: '2026-02-01' }); + + expect(res.status).toBe(200); + expect(res.body).toEqual({ + snapshots: [ + { executionId: 'e1', capturedAt: '2026-01-01', summary: null }, + ], + }); + expect(mockReportCommands.findSnapshotSeries).toHaveBeenCalledWith( + 'demo', + { from: '2026-01-01', to: '2026-02-01' } + ); + }); + + it('GET /api/v2/reports/:name/snapshots 404s an unknown report', async () => { + const res = await request(server).get( + '/api/v2/reports/nope/snapshots' + ); + + expect(res.status).toBe(404); + expect(res.body.code).toBe('REPORT_NOT_FOUND'); + expect(mockReportCommands.findSnapshotSeries).not.toHaveBeenCalled(); + }); + + it('GET /api/v2/reports/integrations (back-compat) returns the payload directly', async () => { + const res = await request(server) + .get('/api/v2/reports/integrations') + .query({ status: 'ENABLED' }); + expect(res.status).toBe(200); + // payload directly, not the runner envelope + expect(res.body).toMatchObject({ + schemaVersion: 1, + filters: { status: 'ENABLED' }, + }); + expect(res.body.status).toBeUndefined(); + }); + + describe('schedule routes', () => { + it('GET /:name/schedule returns the effective schedule (none when no override)', async () => { + mockScheduleCommands.getScheduleByScriptName.mockResolvedValue(null); + + const res = await request(server).get( + '/api/v2/reports/demo/schedule' + ); + + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ + source: 'none', + reportName: 'demo', + enabled: false, + }); + expect( + mockScheduleCommands.getScheduleByScriptName + ).toHaveBeenCalledWith('demo'); + }); + + it('GET /:name/schedule returns the DB override when present', async () => { + mockScheduleCommands.getScheduleByScriptName.mockResolvedValue({ + scriptName: 'demo', + enabled: true, + cronExpression: '0 9 * * *', + timezone: 'UTC', + }); + + const res = await request(server).get( + '/api/v2/reports/demo/schedule' + ); + + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ + source: 'database', + reportName: 'demo', + enabled: true, + cronExpression: '0 9 * * *', + }); + }); + + it('GET /:name/schedule 404s an unknown report', async () => { + const res = await request(server).get( + '/api/v2/reports/nope/schedule' + ); + expect(res.status).toBe(404); + expect(res.body.code).toBe('REPORT_NOT_FOUND'); + }); + + it('PUT /:name/schedule creates the override and provisions the report scheduler', async () => { + mockScheduleCommands.upsertSchedule.mockResolvedValue({ + scriptName: 'demo', + enabled: true, + cronExpression: '0 12 * * *', + timezone: 'UTC', + }); + mockSchedulerAdapter.createSchedule.mockResolvedValue({ + scheduleArn: + 'arn:aws:scheduler:us-east-1:123:schedule/g/frigg-report-demo', + scheduleName: 'frigg-report-demo', + }); + + const res = await request(server) + .put('/api/v2/reports/demo/schedule') + .send({ enabled: true, cronExpression: '0 12 * * *' }); + + expect(res.status).toBe(200); + expect(res.body.success).toBe(true); + expect(res.body.schedule).toMatchObject({ + source: 'database', + enabled: true, + cronExpression: '0 12 * * *', + }); + // Default scheduled mode is snapshot (no schedule.mode on demo). + expect(createReportSchedulerAdapterFromEnv).toHaveBeenCalledWith({ + reportName: 'demo', + mode: 'snapshot', + }); + expect(mockScheduleCommands.upsertSchedule).toHaveBeenCalledWith({ + scriptName: 'demo', + enabled: true, + cronExpression: '0 12 * * *', + timezone: 'UTC', + }); + expect(mockSchedulerAdapter.createSchedule).toHaveBeenCalledWith({ + scriptName: 'demo', + cronExpression: '0 12 * * *', + timezone: 'UTC', + }); + }); + + it('PUT /:name/schedule uses the report Definition.schedule.mode when declared', async () => { + mockScheduleCommands.upsertSchedule.mockResolvedValue({ + scriptName: 'scheduled', + enabled: true, + cronExpression: '0 6 * * *', + timezone: 'UTC', + }); + mockSchedulerAdapter.createSchedule.mockResolvedValue({ + scheduleArn: 'arn:...:frigg-report-scheduled', + scheduleName: 'frigg-report-scheduled', + }); + + const res = await request(server) + .put('/api/v2/reports/scheduled/schedule') + .send({ enabled: true, cronExpression: '0 6 * * *' }); + + expect(res.status).toBe(200); + expect(createReportSchedulerAdapterFromEnv).toHaveBeenCalledWith({ + reportName: 'scheduled', + mode: 'recorded', + }); + }); + + it('PUT /:name/schedule 400s when enabled without a cronExpression', async () => { + const res = await request(server) + .put('/api/v2/reports/demo/schedule') + .send({ enabled: true }); + + expect(res.status).toBe(400); + expect(mockScheduleCommands.upsertSchedule).not.toHaveBeenCalled(); + }); + + it('PUT /:name/schedule 404s an unknown report', async () => { + const res = await request(server) + .put('/api/v2/reports/nope/schedule') + .send({ enabled: false }); + expect(res.status).toBe(404); + expect(res.body.code).toBe('REPORT_NOT_FOUND'); + }); + + it('DELETE /:name/schedule removes the override and tears down the scheduler', async () => { + mockScheduleCommands.deleteSchedule.mockResolvedValue({ + deletedCount: 1, + deleted: { + externalScheduleId: 'arn:...:frigg-report-demo', + }, + }); + mockSchedulerAdapter.deleteSchedule.mockResolvedValue(); + + const res = await request(server).delete( + '/api/v2/reports/demo/schedule' + ); + + expect(res.status).toBe(200); + expect(res.body.success).toBe(true); + expect(res.body.deletedCount).toBe(1); + expect(mockScheduleCommands.deleteSchedule).toHaveBeenCalledWith( + 'demo' + ); + expect(mockSchedulerAdapter.deleteSchedule).toHaveBeenCalledWith( + 'demo' + ); + }); + + it('DELETE /:name/schedule 404s an unknown report', async () => { + const res = await request(server).delete( + '/api/v2/reports/nope/schedule' + ); + expect(res.status).toBe(404); + expect(res.body.code).toBe('REPORT_NOT_FOUND'); + }); + }); +}); diff --git a/packages/admin-scripts/src/infrastructure/bootstrap.js b/packages/admin-scripts/src/infrastructure/bootstrap.js index e85ed972f..8135cf202 100644 --- a/packages/admin-scripts/src/infrastructure/bootstrap.js +++ b/packages/admin-scripts/src/infrastructure/bootstrap.js @@ -1,22 +1,26 @@ const { ScriptFactory } = require('../application/script-factory'); /** - * Admin Script Bootstrap + * Admin Operations Bootstrap * * Composition root for the admin-scripts runtime. Loads the host app's - * definition at Lambda runtime, builds a ScriptFactory, and registers the app's - * admin scripts into it so the router and SQS worker can resolve scripts by - * name. Also constructs the integrationFactory used by scripts that need - * hydrated integration instances. Both are returned for the caller to inject. + * definition at Lambda runtime and builds two name-keyed registries — one for + * admin scripts, one for reports — plus the command bundles they need, so the + * routers and SQS workers can resolve operations by name. The registry class is + * shared (ScriptFactory is generic over `static Definition.name`); reports are a + * second instance rather than a duplicate class. * * Runs once per process (memoized) and never throws — a missing/unloadable app - * definition is logged and leaves the factory empty rather than crashing the + * definition is logged and leaves the registries empty rather than crashing the * Lambda cold start. */ let bootstrapped = false; let scriptFactory = null; let scriptCommands = null; let integrationFactory = null; +let reportFactory = null; +let reportCommands = null; +let reportFriggCommands = null; function registerScripts(factory, scriptClasses) { for (const ScriptClass of scriptClasses || []) { @@ -28,6 +32,24 @@ function registerScripts(factory, scriptClasses) { } } +// A report whose name collides with a registered script is skipped: scripts and +// reports share ScriptSchedule.scriptName (@unique), so names must not clash. +function registerReports(factory, scriptRegistry, reportClasses) { + for (const ReportClass of reportClasses || []) { + const name = ReportClass?.Definition?.name; + if (!name) continue; + if (scriptRegistry.has(name)) { + console.error( + `[admin-scripts] bootstrap: report "${name}" collides with a registered script name; skipping.` + ); + continue; + } + if (!factory.has(name)) { + factory.register(ReportClass); + } + } +} + /** * Build an integration factory backed by the framework's runtime hydration. * Scripts call `context.instantiate(integrationId)` which delegates here. @@ -46,7 +68,7 @@ function createIntegrationFactory() { } /** - * Build the command bundle injected into every AdminScriptContext. Scripts + * Build the command bundle injected into every AdminScriptContext. Operations * interact with the database only through these Frigg commands — never through * repositories directly. The require is lazy so the package keeps no load-time * coupling to core's command/repository modules. @@ -75,27 +97,62 @@ function buildScriptCommands() { }; } +function buildReportFriggCommands() { + const { + createIntegrationMappingCommands, + } = require('@friggframework/core/application/commands/integration-mapping-commands'); + const { + createUsageCommands, + } = require('@friggframework/core/application/commands/usage-commands'); + + return { + ...buildScriptCommands(), + integrationMappings: createIntegrationMappingCommands(), + usage: createUsageCommands(), + }; +} + /** - * @returns {{ scriptFactory: ScriptFactory, scriptCommands: object, integrationFactory: object }} + * @returns {{ scriptFactory: ScriptFactory, scriptCommands: object, integrationFactory: object, reportFactory: ScriptFactory, reportCommands: object, reportFriggCommands: object }} */ function bootstrapAdminScripts() { if (bootstrapped) { - return { scriptFactory, scriptCommands, integrationFactory }; + return { + scriptFactory, + scriptCommands, + integrationFactory, + reportFactory, + reportCommands, + reportFriggCommands, + }; } bootstrapped = true; - // Create the registry up front so consumers always get a (possibly empty) - // factory even when the app definition can't be loaded — mirrors the + // Create the registries up front so consumers always get (possibly empty) + // factories even when the app definition can't be loaded — mirrors the // never-throw contract above. scriptFactory = new ScriptFactory(); + reportFactory = new ScriptFactory(); try { const { loadAppDefinition, } = require('@friggframework/core/handlers/app-definition-loader'); - const { adminScripts = [] } = loadAppDefinition(); + const { + adminScripts = [], + reports = [], + admin = {}, + } = loadAppDefinition(); registerScripts(scriptFactory, adminScripts); + registerReports(reportFactory, scriptFactory, reports); + + if (admin.includeBuiltinReports) { + const { + BUILTIN_REPORTS, + } = require('@friggframework/core/reporting/builtin-reports'); + registerReports(reportFactory, scriptFactory, BUILTIN_REPORTS); + } } catch (error) { console.error( '[admin-scripts] bootstrap: could not load app definition:', @@ -103,8 +160,8 @@ function bootstrapAdminScripts() { ); } - // Built in its own try so a command-layer failure never blocks script - // registration (and vice versa) — both honor the never-throw contract. + // Built in their own try so a command-layer failure never blocks operation + // registration (and vice versa) — all honor the never-throw contract. try { scriptCommands = buildScriptCommands(); } catch (error) { @@ -114,8 +171,28 @@ function bootstrapAdminScripts() { ); } + try { + reportFriggCommands = buildReportFriggCommands(); + const { + createReportCommands, + } = require('@friggframework/core/application/commands/report-commands'); + reportCommands = createReportCommands(); + } catch (error) { + console.error( + '[admin-scripts] bootstrap: could not build report commands:', + error.message + ); + } + integrationFactory = createIntegrationFactory(); - return { scriptFactory, scriptCommands, integrationFactory }; + return { + scriptFactory, + scriptCommands, + integrationFactory, + reportFactory, + reportCommands, + reportFriggCommands, + }; } /** Test-only: reset memoized bootstrap state. */ @@ -124,6 +201,9 @@ function _resetBootstrapForTests() { scriptFactory = null; scriptCommands = null; integrationFactory = null; + reportFactory = null; + reportCommands = null; + reportFriggCommands = null; } module.exports = { diff --git a/packages/admin-scripts/src/infrastructure/report-executor-handler.js b/packages/admin-scripts/src/infrastructure/report-executor-handler.js new file mode 100644 index 000000000..7d83bf16c --- /dev/null +++ b/packages/admin-scripts/src/infrastructure/report-executor-handler.js @@ -0,0 +1,202 @@ +const { createReportRunner } = require('../application/report-runner'); +const { + createReportCommands, +} = require('@friggframework/core/application/commands/report-commands'); +const { QueuerUtil } = require('@friggframework/core/queues'); +const { bootstrapAdminScripts } = require('./bootstrap'); + +// Resume state travels in the message (params.__resume); the execution record +// stays RUNNING across hops so the same execution resumes. +async function requeueContinuation(message, result) { + const queueUrl = process.env.REPORT_QUEUE_URL; + if (!queueUrl) { + throw new Error( + 'Report yielded a continuation but REPORT_QUEUE_URL is not set; cannot resume' + ); + } + + await QueuerUtil.send( + { + reportName: message.reportName, + executionId: result.executionId, + mode: message.mode || 'recorded', + ...(message.seriesName && { seriesName: message.seriesName }), + trigger: 'QUEUE', + params: { ...(message.params || {}), __resume: result.continuation }, + resumeAt: new Date().toISOString(), + }, + queueUrl + ); +} + +async function runMessage( + message, + { reportFactory, reportCommands, reportFriggCommands, integrationFactory }, + lambdaContext +) { + const { + reportName, + executionId, + mode, + seriesName, + trigger, + params, + parentExecutionId, + } = message; + + if (!reportName) { + throw new Error('Invalid message: missing reportName'); + } + + console.log( + `Processing report: ${reportName}${ + executionId ? `, executionId: ${executionId}` : '' + }` + ); + + const runner = createReportRunner({ + reportFactory, + reportCommands, + friggCommands: reportFriggCommands, + integrationFactory, + }); + const result = await runner.execute(reportName, params, { + mode: mode || 'recorded', + trigger: trigger || 'QUEUE', + // Scheduled first runs carry no executionId; ReportRunner creates the record. + ...(executionId && { executionId }), + ...(seriesName && { seriesName }), + ...(parentExecutionId && { parentExecutionId }), + ...(lambdaContext && { lambdaContext }), + }); + + if (result.status === 'CONTINUE') { + // A failed re-enqueue means the execution never resumes; mark it FAILED + // (via the runner-created id) instead of leaving it stuck in RUNNING. + try { + await requeueContinuation(message, result); + } catch (requeueError) { + await markFailed(result.executionId, requeueError); + throw requeueError; + } + console.log( + `Report re-queued for continuation: ${reportName}, executionId: ${result.executionId}` + ); + } + + return { + reportName, + status: result.status, + executionId: result.executionId, + }; +} + +// Mark a report execution FAILED when the worker itself throws, so the record +// doesn't stay stuck in a non-terminal state. +async function markFailed(executionId, error) { + if (!executionId) return; + try { + const commands = createReportCommands(); + await commands.completeExecution(executionId, { + state: 'FAILED', + error: { + name: error.name, + message: error.message, + stack: error.stack, + }, + }); + } catch (updateError) { + console.error( + `Failed to update report execution ${executionId} state:`, + updateError + ); + } +} + +// EventBridge Scheduler direct invoke: the event itself is the message (no `Records` wrapper). +async function handleScheduledInvoke(event, deps, lambdaContext) { + try { + const result = await runMessage(event, deps, lambdaContext); + console.log( + `Report completed: ${result.reportName}, status: ${result.status}` + ); + return { + statusCode: 200, + body: JSON.stringify({ processed: 1, results: [result] }), + }; + } catch (error) { + console.error('Unexpected error processing scheduled invoke:', error); + await markFailed(event.executionId, error); + return { + statusCode: 200, + body: JSON.stringify({ + processed: 1, + results: [ + { + reportName: event.reportName || 'unknown', + status: 'FAILED', + error: error.message, + }, + ], + }), + }; + } +} + +// SQS batch: each record body is a JSON message. Failures are isolated per +// record so one bad message doesn't drop the rest of the batch. +async function handleSqsBatch(event, deps, lambdaContext) { + const results = []; + for (const record of event.Records) { + let message = {}; + try { + message = JSON.parse(record.body); + const result = await runMessage(message, deps, lambdaContext); + console.log( + `Report completed: ${result.reportName}, status: ${result.status}` + ); + results.push(result); + } catch (error) { + // Report execution errors are handled by ReportRunner; only + // unexpected failures (parse, runner construction) reach here. + console.error('Unexpected error processing record:', error); + await markFailed(message.executionId, error); + results.push({ + reportName: message.reportName || 'unknown', + status: 'FAILED', + error: error.message, + }); + } + } + + return { + statusCode: 200, + body: JSON.stringify({ processed: results.length, results }), + }; +} + +/** + * Report Executor Lambda handler. Two invocation shapes: + * - SQS: `event.Records[]`, each body a JSON execution message. + * - EventBridge Scheduler direct invoke: the event itself is the message, no `Records`. + */ +async function handler(event, context) { + const { + reportFactory, + reportCommands, + reportFriggCommands, + integrationFactory, + } = bootstrapAdminScripts(); + const deps = { + reportFactory, + reportCommands, + reportFriggCommands, + integrationFactory, + }; + + return event.Records + ? handleSqsBatch(event, deps, context) + : handleScheduledInvoke(event, deps, context); +} + +module.exports = { handler }; diff --git a/packages/admin-scripts/src/infrastructure/report-router.js b/packages/admin-scripts/src/infrastructure/report-router.js new file mode 100644 index 000000000..204d519d4 --- /dev/null +++ b/packages/admin-scripts/src/infrastructure/report-router.js @@ -0,0 +1,421 @@ +const express = require('express'); +const serverless = require('serverless-http'); +const { validateAdminApiKey } = require('./admin-auth-middleware'); +const { createReportRunner } = require('../application/report-runner'); +const { validateParams } = require('../application/validate-script-input'); +const { QueuerUtil } = require('@friggframework/core/queues'); +const { + createAdminScriptCommands, +} = require('@friggframework/core/application/commands/admin-script-commands'); +const { + createReportSchedulerAdapterFromEnv, +} = require('../adapters/scheduler-adapter-factory'); +const { + GetEffectiveScheduleUseCase, + UpsertScheduleUseCase, + DeleteScheduleUseCase, +} = require('../application/use-cases'); +const { bootstrapAdminScripts } = require('./bootstrap'); + +const router = express.Router(); + +// Reports use the admin API key; the dedicated REPORTING_API_KEY is retired (ADR-010). +router.use(validateAdminApiKey); + +const CODE_STATUS = { + INVALID_INPUT: 400, + INVALID_MODE: 400, + ARTIFACT_STORAGE_UNAVAILABLE: 501, +}; + +// Boom errors carry their own status; runner errors carry a `.code`; anything else is a 500. +function sendReportError(res, error, fallbackMessage) { + if (error.isBoom) { + return res + .status(error.output.statusCode) + .json({ error: error.message }); + } + if (error.code && CODE_STATUS[error.code]) { + return res + .status(CODE_STATUS[error.code]) + .json({ error: error.message, code: error.code }); + } + console.error(fallbackMessage, error); + return res.status(500).json({ error: fallbackMessage }); +} + +function buildAudit(req) { + const apiKey = req.headers['x-frigg-admin-api-key']; + const forwardedFor = (req.headers['x-forwarded-for'] || '') + .split(',')[0] + .trim(); + return { + ipAddress: forwardedFor || req.ip || null, + apiKeyLast4: apiKey ? String(apiKey).slice(-4) : null, + }; +} + +function toDefinitionSummary(definition) { + return { + name: definition.name, + version: definition.version, + description: definition.description, + runModes: definition.runModes, + category: definition.display?.category || 'reporting', + }; +} + +router.get('/', (_req, res) => { + try { + const { reportFactory } = bootstrapAdminScripts(); + res.json({ + service: 'frigg-core-api', + reports: reportFactory.getAll().map((r) => + toDefinitionSummary(r.definition) + ), + }); + } catch (error) { + console.error('Error listing reports:', error); + res.status(500).json({ error: 'Failed to list reports' }); + } +}); + +// Deprecated #607 back-compat. Registered before '/:name' so it wins the match. +router.get('/integrations', async (req, res) => { + try { + const { + reportFactory, + reportFriggCommands, + reportCommands, + integrationFactory, + } = bootstrapAdminScripts(); + + if (!reportFactory.has('integrations')) { + return res.status(404).json({ + error: 'Report "integrations" not found', + code: 'REPORT_NOT_FOUND', + }); + } + + const { status, type, userId } = req.query; + const runner = createReportRunner({ + reportFactory, + reportCommands, + friggCommands: reportFriggCommands, + integrationFactory, + }); + const result = await runner.execute( + 'integrations', + { status, type, userId }, + { mode: 'live', trigger: 'MANUAL', audit: buildAudit(req) } + ); + // #607 returned the report payload directly (not the runner envelope). + return res.json(result.output); + } catch (error) { + return sendReportError(res, error, 'Failed to run integrations report'); + } +}); + +// Registered before '/:name' so 'executions' isn't captured as a report name. +router.get('/executions/:id', async (req, res) => { + try { + const { reportCommands } = bootstrapAdminScripts(); + const execution = await reportCommands.findExecutionById(req.params.id); + if (execution.error) { + return res.status(execution.error).json({ + error: execution.reason, + code: execution.code, + }); + } + return res.json(execution); + } catch (error) { + return sendReportError(res, error, 'Failed to get report execution'); + } +}); + +router.get('/:name/snapshots', async (req, res) => { + try { + const { name } = req.params; + const { reportFactory, reportCommands } = bootstrapAdminScripts(); + if (!reportFactory.has(name)) { + return res.status(404).json({ + error: `Report "${name}" not found`, + code: 'REPORT_NOT_FOUND', + }); + } + const snapshots = await reportCommands.findSnapshotSeries(name, { + from: req.query.from, + to: req.query.to, + }); + return res.json({ snapshots }); + } catch (error) { + return sendReportError(res, error, 'Failed to get report snapshots'); + } +}); + +// Default to snapshot so a scheduled report captures a time series out of the box. +function scheduledRunMode(definition) { + return definition.schedule?.mode || 'snapshot'; +} + +// ScriptSchedule.scriptName is a shared operation-name namespace, so the report name keys the same row a script would. +router.get('/:name/schedule', async (req, res) => { + try { + const { name } = req.params; + const { reportFactory } = bootstrapAdminScripts(); + if (!reportFactory.has(name)) { + return res.status(404).json({ + error: `Report "${name}" not found`, + code: 'REPORT_NOT_FOUND', + }); + } + + const commands = createAdminScriptCommands(); + const getEffectiveSchedule = new GetEffectiveScheduleUseCase({ + commands, + scriptFactory: reportFactory, + }); + const result = await getEffectiveSchedule.execute(name); + + return res.json({ + source: result.source, + reportName: name, + ...result.schedule, + }); + } catch (error) { + return sendReportError(res, error, 'Failed to get report schedule'); + } +}); + +// The AWS scheduler targets the REPORT executor and enqueues a report-shaped message, not the script executor. +router.put('/:name/schedule', async (req, res) => { + try { + const { name } = req.params; + const { enabled, cronExpression, timezone } = req.body || {}; + const { reportFactory } = bootstrapAdminScripts(); + if (!reportFactory.has(name)) { + return res.status(404).json({ + error: `Report "${name}" not found`, + code: 'REPORT_NOT_FOUND', + }); + } + + const definition = reportFactory.get(name).Definition; + const commands = createAdminScriptCommands(); + const schedulerAdapter = createReportSchedulerAdapterFromEnv({ + reportName: name, + mode: scheduledRunMode(definition), + }); + const upsertSchedule = new UpsertScheduleUseCase({ + commands, + schedulerAdapter, + scriptFactory: reportFactory, + }); + + const result = await upsertSchedule.execute(name, { + enabled, + cronExpression, + timezone, + }); + + return res.json({ + success: result.success, + schedule: { + source: 'database', + ...result.schedule, + }, + ...(result.schedulerWarning && { + schedulerWarning: result.schedulerWarning, + }), + }); + } catch (error) { + return sendReportError(res, error, 'Failed to update report schedule'); + } +}); + +router.delete('/:name/schedule', async (req, res) => { + try { + const { name } = req.params; + const { reportFactory } = bootstrapAdminScripts(); + if (!reportFactory.has(name)) { + return res.status(404).json({ + error: `Report "${name}" not found`, + code: 'REPORT_NOT_FOUND', + }); + } + + const definition = reportFactory.get(name).Definition; + const commands = createAdminScriptCommands(); + const schedulerAdapter = createReportSchedulerAdapterFromEnv({ + reportName: name, + mode: scheduledRunMode(definition), + }); + const deleteSchedule = new DeleteScheduleUseCase({ + commands, + schedulerAdapter, + scriptFactory: reportFactory, + }); + + const result = await deleteSchedule.execute(name); + return res.json(result); + } catch (error) { + return sendReportError(res, error, 'Failed to delete report schedule'); + } +}); + +router.get('/:name', (req, res) => { + try { + const { name } = req.params; + const { reportFactory } = bootstrapAdminScripts(); + + if (!reportFactory.has(name)) { + return res.status(404).json({ + error: `Report "${name}" not found`, + code: 'REPORT_NOT_FOUND', + }); + } + + const definition = reportFactory.get(name).Definition; + res.json({ + name: definition.name, + version: definition.version, + description: definition.description, + runModes: definition.runModes, + inputSchema: definition.inputSchema, + outputSchema: definition.outputSchema, + output: definition.output, + schedule: definition.schedule, + display: definition.display, + }); + } catch (error) { + console.error('Error getting report:', error); + res.status(500).json({ error: 'Failed to get report details' }); + } +}); + +router.post('/:name/run', async (req, res) => { + try { + const { name } = req.params; + const { mode = 'live', params = {} } = req.body || {}; + const { + reportFactory, + reportFriggCommands, + reportCommands, + integrationFactory, + } = bootstrapAdminScripts(); + + if (!reportFactory.has(name)) { + return res.status(404).json({ + error: `Report "${name}" not found`, + code: 'REPORT_NOT_FOUND', + }); + } + + if (mode === 'live') { + const runner = createReportRunner({ + reportFactory, + reportCommands, + friggCommands: reportFriggCommands, + integrationFactory, + }); + const result = await runner.execute(name, params, { + mode: 'live', + trigger: 'MANUAL', + audit: buildAudit(req), + }); + return res.json(result); + } + + // Validate mode + params before persisting/enqueueing so a bad request gets + // a 400 up front instead of a 202 that only fails later in the worker. + const definition = reportFactory.get(name).Definition; + const runModes = + Array.isArray(definition.runModes) && definition.runModes.length + ? definition.runModes + : ['live']; + if (!runModes.includes(mode)) { + return res.status(400).json({ + error: `Report "${name}" does not support mode "${mode}". Allowed: ${runModes.join( + ', ' + )}`, + code: 'INVALID_MODE', + }); + } + const validation = validateParams(definition, params); + if (!validation.valid) { + return res.status(400).json({ + error: `Invalid input: ${validation.errors.join(', ')}`, + code: 'INVALID_INPUT', + }); + } + + const queueUrl = process.env.REPORT_QUEUE_URL; + if (!queueUrl) { + return res.status(503).json({ + error: 'Async report execution is not configured (REPORT_QUEUE_URL not set)', + code: 'QUEUE_NOT_CONFIGURED', + }); + } + + const { seriesName } = req.body || {}; + const execution = await reportCommands.createExecution({ + reportName: name, + reportVersion: definition.version, + trigger: 'MANUAL', + mode, + input: params, + audit: buildAudit(req), + seriesName, + }); + + // Commands return an error object (never throw) — don't queue a broken execution. + if (execution.error) { + return res.status(execution.error).json({ + error: execution.reason || 'Failed to create report execution record', + code: execution.code, + }); + } + + try { + await QueuerUtil.send( + { + reportName: name, + executionId: execution.id, + mode, + seriesName, + trigger: 'MANUAL', + params, + }, + queueUrl + ); + } catch (enqueueError) { + // The record is persisted but will never be picked up — compensate + // so it doesn't linger non-terminal (symmetric with the + // createExecution-error guard above). + await reportCommands.completeExecution(execution.id, { + state: 'FAILED', + error: { + name: enqueueError.name, + message: enqueueError.message, + }, + }); + throw enqueueError; + } + + return res.status(202).json({ + executionId: execution.id, + status: 'QUEUED', + reportName: name, + }); + } catch (error) { + return sendReportError(res, error, 'Failed to run report'); + } +}); + +const app = express(); +app.use(express.json()); +app.use('/api/v2/reports', router); + +const handler = serverless(app); + +module.exports = { router, app, handler }; diff --git a/packages/core/__tests__/documentdb-factory-selection.test.js b/packages/core/__tests__/documentdb-factory-selection.test.js index a01a88a7e..946e31dfc 100644 --- a/packages/core/__tests__/documentdb-factory-selection.test.js +++ b/packages/core/__tests__/documentdb-factory-selection.test.js @@ -31,11 +31,6 @@ const FACTORIES = [ factoryName: 'createProcessRepository', exportName: 'ProcessRepositoryDocumentDB', }, - { - modulePath: '../reporting/repositories/reporting-repository-factory', - factoryName: 'createReportingRepository', - exportName: 'ReportingRepositoryDocumentDB', - }, { modulePath: '../syncs/repositories/sync-repository-factory', factoryName: 'createSyncRepository', diff --git a/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-mongo.test.js b/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-mongo.test.js index c9f6be801..ba8da948f 100644 --- a/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-mongo.test.js +++ b/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-mongo.test.js @@ -186,6 +186,56 @@ describe('AdminScriptExecutionRepositoryMongo', () => { skip: 5, }); }); + + it('should filter by type when provided', async () => { + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([]); + + await repository.findExecutionsByName('nightly', { type: 'REPORT' }); + + expect(mockPrisma.adminScriptExecution.findMany).toHaveBeenCalledWith({ + where: { name: 'nightly', type: 'REPORT' }, + orderBy: { createdAt: 'desc' }, + take: undefined, + skip: undefined, + }); + }); + + it('should apply a createdAt window with both bounds', async () => { + const from = new Date('2025-01-01'); + const to = new Date('2025-02-01'); + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([]); + + await repository.findExecutionsByName('nightly', { + type: 'REPORT', + from, + to, + }); + + expect(mockPrisma.adminScriptExecution.findMany).toHaveBeenCalledWith({ + where: { + name: 'nightly', + type: 'REPORT', + createdAt: { gte: from, lte: to }, + }, + orderBy: { createdAt: 'desc' }, + take: undefined, + skip: undefined, + }); + }); + + it('should include only the createdAt bound provided', async () => { + const from = new Date('2025-01-01'); + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([]); + + await repository.findExecutionsByName('nightly', { from }); + + expect(mockPrisma.adminScriptExecution.findMany).toHaveBeenCalledWith({ + where: { name: 'nightly', createdAt: { gte: from } }, + orderBy: { createdAt: 'desc' }, + take: undefined, + skip: undefined, + }); + }); }); describe('findExecutionsByState()', () => { @@ -443,5 +493,18 @@ describe('AdminScriptExecutionRepositoryMongo', () => { deletedCount: 0, }); }); + + it('should scope the delete by type when provided', async () => { + const date = new Date('2024-01-01'); + mockPrisma.adminScriptExecution.deleteMany.mockResolvedValue({ + count: 3, + }); + + await repository.deleteExecutionsOlderThan(date, { type: 'REPORT' }); + + expect(mockPrisma.adminScriptExecution.deleteMany).toHaveBeenCalledWith({ + where: { createdAt: { lt: date }, type: 'REPORT' }, + }); + }); }); }); diff --git a/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-postgres.test.js b/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-postgres.test.js new file mode 100644 index 000000000..9d7650b9a --- /dev/null +++ b/packages/core/admin-scripts/repositories/__tests__/admin-script-execution-repository-postgres.test.js @@ -0,0 +1,116 @@ +const { + AdminScriptExecutionRepositoryPostgres, +} = require('../admin-script-execution-repository-postgres'); + +describe('AdminScriptExecutionRepositoryPostgres query window', () => { + let repository; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { + adminScriptExecution: { + findMany: jest.fn(), + deleteMany: jest.fn(), + }, + }; + + repository = new AdminScriptExecutionRepositoryPostgres(); + repository.prisma = mockPrisma; + }); + + describe('findExecutionsByName()', () => { + it('should filter by type when provided', async () => { + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([]); + + await repository.findExecutionsByName('nightly', { type: 'REPORT' }); + + expect(mockPrisma.adminScriptExecution.findMany).toHaveBeenCalledWith({ + where: { name: 'nightly', type: 'REPORT' }, + orderBy: { createdAt: 'desc' }, + take: undefined, + skip: undefined, + }); + }); + + it('should apply a createdAt window with both bounds', async () => { + const from = new Date('2025-01-01'); + const to = new Date('2025-02-01'); + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([]); + + await repository.findExecutionsByName('nightly', { + type: 'REPORT', + from, + to, + }); + + expect(mockPrisma.adminScriptExecution.findMany).toHaveBeenCalledWith({ + where: { + name: 'nightly', + type: 'REPORT', + createdAt: { gte: from, lte: to }, + }, + orderBy: { createdAt: 'desc' }, + take: undefined, + skip: undefined, + }); + }); + + it('should include only the createdAt bound provided', async () => { + const to = new Date('2025-02-01'); + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([]); + + await repository.findExecutionsByName('nightly', { to }); + + expect(mockPrisma.adminScriptExecution.findMany).toHaveBeenCalledWith({ + where: { name: 'nightly', createdAt: { lte: to } }, + orderBy: { createdAt: 'desc' }, + take: undefined, + skip: undefined, + }); + }); + + it('should convert returned ids to strings', async () => { + mockPrisma.adminScriptExecution.findMany.mockResolvedValue([ + { id: 1, parentExecutionId: 2, name: 'nightly', type: 'REPORT' }, + ]); + + const result = await repository.findExecutionsByName('nightly', { + type: 'REPORT', + }); + + expect(result[0].id).toBe('1'); + expect(result[0].parentExecutionId).toBe('2'); + }); + }); + + describe('deleteExecutionsOlderThan()', () => { + it('should scope the delete by type when provided', async () => { + const date = new Date('2024-01-01'); + mockPrisma.adminScriptExecution.deleteMany.mockResolvedValue({ + count: 3, + }); + + const result = await repository.deleteExecutionsOlderThan(date, { + type: 'REPORT', + }); + + expect(result).toEqual({ acknowledged: true, deletedCount: 3 }); + expect(mockPrisma.adminScriptExecution.deleteMany).toHaveBeenCalledWith({ + where: { createdAt: { lt: date }, type: 'REPORT' }, + }); + }); + + it('should omit type when not provided', async () => { + const date = new Date('2024-01-01'); + mockPrisma.adminScriptExecution.deleteMany.mockResolvedValue({ + count: 0, + }); + + await repository.deleteExecutionsOlderThan(date); + + expect(mockPrisma.adminScriptExecution.deleteMany).toHaveBeenCalledWith({ + where: { createdAt: { lt: date } }, + }); + }); + }); +}); diff --git a/packages/core/admin-scripts/repositories/admin-script-execution-repository-interface.js b/packages/core/admin-scripts/repositories/admin-script-execution-repository-interface.js index e07cd7d27..92bd96ae8 100644 --- a/packages/core/admin-scripts/repositories/admin-script-execution-repository-interface.js +++ b/packages/core/admin-scripts/repositories/admin-script-execution-repository-interface.js @@ -34,7 +34,7 @@ class AdminScriptExecutionRepositoryInterface { * @param {string} [params.context.audit.apiKeyName] - Name of API key used * @param {string} [params.context.audit.apiKeyLast4] - Last 4 chars of API key * @param {string} [params.context.audit.ipAddress] - IP address of requester - * @param {string|number} [params.parentExecutionId] - ID of the execution that queued this one; persisted to the parentExecutionId column (self-FK) for the parent/child hierarchy + * @param {string|number} [params.parentExecutionId] - ID of the execution that queued this one, for the parent/child hierarchy * @returns {Promise} The created process record * @abstract */ @@ -65,6 +65,9 @@ class AdminScriptExecutionRepositoryInterface { * @param {string} [options.sortBy] - Field to sort by * @param {string} [options.sortOrder] - Sort order ('asc' or 'desc') * @param {string} [options.state] - Optional state filter ('PENDING', 'RUNNING', 'COMPLETED', 'FAILED') + * @param {string} [options.type] - Optional type filter ('ADMIN_SCRIPT', 'REPORT', 'DB_MIGRATION') + * @param {Date} [options.from] - Optional lower bound (inclusive) on createdAt + * @param {Date} [options.to] - Optional upper bound (inclusive) on createdAt * @returns {Promise} Array of process records * @abstract */ @@ -153,10 +156,12 @@ class AdminScriptExecutionRepositoryInterface { * Used for cleanup and retention policies * * @param {Date} date - Delete processes older than this date + * @param {Object} [options] - Deletion options + * @param {string} [options.type] - Optional type filter, so report vs script retention can diverge * @returns {Promise} Deletion result with count * @abstract */ - async deleteExecutionsOlderThan(date) { + async deleteExecutionsOlderThan(date, options = {}) { throw new Error( 'Method deleteExecutionsOlderThan must be implemented by subclass' ); diff --git a/packages/core/admin-scripts/repositories/admin-script-execution-repository-mongo.js b/packages/core/admin-scripts/repositories/admin-script-execution-repository-mongo.js index 96dde549e..acac688b4 100644 --- a/packages/core/admin-scripts/repositories/admin-script-execution-repository-mongo.js +++ b/packages/core/admin-scripts/repositories/admin-script-execution-repository-mongo.js @@ -75,10 +75,19 @@ class AdminScriptExecutionRepositoryMongo extends AdminScriptExecutionRepository sortBy = 'createdAt', sortOrder = 'desc', state, + type, + from, + to, } = options; const where = { name }; if (state) where.state = state; + if (type) where.type = type; + if (from || to) { + where.createdAt = {}; + if (from) where.createdAt.gte = from; + if (to) where.createdAt.lte = to; + } const processes = await this.prisma.adminScriptExecution.findMany({ where, @@ -207,15 +216,16 @@ class AdminScriptExecutionRepositoryMongo extends AdminScriptExecutionRepository * Used for cleanup and retention policies * * @param {Date} date - Delete processes older than this date + * @param {Object} [options] - Deletion options + * @param {string} [options.type] - Optional type filter * @returns {Promise} Deletion result with count */ - async deleteExecutionsOlderThan(date) { + async deleteExecutionsOlderThan(date, { type } = {}) { + const where = { createdAt: { lt: date } }; + if (type) where.type = type; + const result = await this.prisma.adminScriptExecution.deleteMany({ - where: { - createdAt: { - lt: date, - }, - }, + where, }); return { diff --git a/packages/core/admin-scripts/repositories/admin-script-execution-repository-postgres.js b/packages/core/admin-scripts/repositories/admin-script-execution-repository-postgres.js index 0737b4035..52ae07f14 100644 --- a/packages/core/admin-scripts/repositories/admin-script-execution-repository-postgres.js +++ b/packages/core/admin-scripts/repositories/admin-script-execution-repository-postgres.js @@ -110,10 +110,19 @@ class AdminScriptExecutionRepositoryPostgres extends AdminScriptExecutionReposit sortBy = 'createdAt', sortOrder = 'desc', state, + type, + from, + to, } = options; const where = { name }; if (state) where.state = state; + if (type) where.type = type; + if (from || to) { + where.createdAt = {}; + if (from) where.createdAt.gte = from; + if (to) where.createdAt.lte = to; + } const processes = await this.prisma.adminScriptExecution.findMany({ where, @@ -247,15 +256,16 @@ class AdminScriptExecutionRepositoryPostgres extends AdminScriptExecutionReposit * Used for cleanup and retention policies * * @param {Date} date - Delete processes older than this date + * @param {Object} [options] - Deletion options + * @param {string} [options.type] - Optional type filter * @returns {Promise} Deletion result with count */ - async deleteExecutionsOlderThan(date) { + async deleteExecutionsOlderThan(date, { type } = {}) { + const where = { createdAt: { lt: date } }; + if (type) where.type = type; + const result = await this.prisma.adminScriptExecution.deleteMany({ - where: { - createdAt: { - lt: date, - }, - }, + where, }); return { diff --git a/packages/core/application/commands/__tests__/admin-script-commands.test.js b/packages/core/application/commands/__tests__/admin-script-commands.test.js index a0e4db602..6ca0cd565 100644 --- a/packages/core/application/commands/__tests__/admin-script-commands.test.js +++ b/packages/core/application/commands/__tests__/admin-script-commands.test.js @@ -137,6 +137,15 @@ describe('createAdminScriptCommands', () => { expect(result).toHaveProperty('code', 'EXECUTION_NOT_FOUND'); expect(result.reason).toContain('non-existent'); }); + + it('404s a REPORT row so a script lookup never returns a report execution', async () => { + mockAdminScriptExecutionRepo.findExecutionById.mockResolvedValue({ id: 'r1', type: 'REPORT', state: 'COMPLETED' }); + + const result = await commands.findExecutionById('r1'); + + expect(result).toHaveProperty('error', 404); + expect(result).toHaveProperty('code', 'EXECUTION_NOT_FOUND'); + }); }); describe('findExecutionsByName', () => { diff --git a/packages/core/application/commands/__tests__/credential-commands.test.js b/packages/core/application/commands/__tests__/credential-commands.test.js new file mode 100644 index 000000000..83c6f0df4 --- /dev/null +++ b/packages/core/application/commands/__tests__/credential-commands.test.js @@ -0,0 +1,62 @@ +// Mock the repository factory BEFORE importing the commands +jest.mock('../../../credential/repositories/credential-repository-factory', () => ({ + createCredentialRepository: jest.fn(), +})); + +const { + createCredentialRepository, +} = require('../../../credential/repositories/credential-repository-factory'); +const { createCredentialCommands } = require('../credential-commands'); + +describe('credential-commands countActiveByType', () => { + let repository; + + beforeEach(() => { + repository = { + countActiveByType: jest.fn(), + }; + createCredentialRepository.mockReturnValue(repository); + }); + + afterEach(() => { + jest.clearAllMocks(); + }); + + it('delegates to the repository with { since } and returns its projection', async () => { + const since = new Date('2026-06-15T00:00:00Z'); + repository.countActiveByType.mockResolvedValue([ + { integrationType: 'hubspot', count: 3 }, + { integrationType: 'salesforce', count: 1 }, + ]); + + const commands = createCredentialCommands(); + const result = await commands.countActiveByType({ since }); + + expect(repository.countActiveByType).toHaveBeenCalledWith({ since }); + expect(result).toEqual([ + { integrationType: 'hubspot', count: 3 }, + { integrationType: 'salesforce', count: 1 }, + ]); + }); + + it('passes through an undefined since (count-all)', async () => { + repository.countActiveByType.mockResolvedValue([]); + + const commands = createCredentialCommands(); + const result = await commands.countActiveByType(); + + expect(repository.countActiveByType).toHaveBeenCalledWith({ + since: undefined, + }); + expect(result).toEqual([]); + }); + + it('never throws — maps repository errors to an error response', async () => { + repository.countActiveByType.mockRejectedValue(new Error('db down')); + + const commands = createCredentialCommands(); + const result = await commands.countActiveByType({ since: new Date() }); + + expect(result).toEqual({ error: 500, reason: 'db down' }); + }); +}); diff --git a/packages/core/application/commands/admin-script-commands.js b/packages/core/application/commands/admin-script-commands.js index 234b22145..c95540e6d 100644 --- a/packages/core/application/commands/admin-script-commands.js +++ b/packages/core/application/commands/admin-script-commands.js @@ -105,7 +105,8 @@ function createAdminScriptCommands() { const process = await adminScriptExecutionRepository.findExecutionById( processId ); - if (!process) { + // Scripts and reports share one store; exclude REPORT rows so the two never read each other's executions. + if (!process || process.type === 'REPORT') { const error = new Error(`Execution ${processId} not found`); error.code = 'EXECUTION_NOT_FOUND'; return mapErrorToResponse(error); diff --git a/packages/core/application/commands/credential-commands.js b/packages/core/application/commands/credential-commands.js index faed59cac..e9106267b 100644 --- a/packages/core/application/commands/credential-commands.js +++ b/packages/core/application/commands/credential-commands.js @@ -216,6 +216,23 @@ function createCredentialCommands() { } }, + /** + * Count credentials active (updatedAt >= since) grouped by integration + * type, derived from the related Entity.moduleName. Returns a non-secret + * projection only — never reads or decrypts credential secrets. + * + * @param {Object} params + * @param {Date} [params.since] - Lower bound on updatedAt + * @returns {Promise>} + */ + async countActiveByType({ since } = {}) { + try { + return await credRepo.countActiveByType({ since }); + } catch (error) { + return mapErrorToResponse(error); + } + }, + /** * Delete a credential by ID (alias for deleteCredential) * @param {string} credentialId - Credential ID to delete diff --git a/packages/core/application/commands/integration-commands.js b/packages/core/application/commands/integration-commands.js index abe3d77c3..fed6186e2 100644 --- a/packages/core/application/commands/integration-commands.js +++ b/packages/core/application/commands/integration-commands.js @@ -89,9 +89,21 @@ function createIntegrationCommands({ integrationClass } = {}) { } } + /** + * Report-shaped projection (derived counters + timestamps) — the + * cross-integration read that reports (ADR-010) consume via commands. + */ + async function listForReport(filter = {}) { + try { + return await integrationRepository.findAllForReport(filter); + } catch (error) { + return mapErrorToResponse(error); + } + } + // The remaining commands hydrate/modify integrations for a specific class. if (!integrationClass) { - return { findIntegrationById, listIntegrations }; + return { findIntegrationById, listIntegrations, listForReport }; } const moduleRepository = createModuleRepository(); @@ -154,6 +166,7 @@ function createIntegrationCommands({ integrationClass } = {}) { return { findIntegrationById, listIntegrations, + listForReport, /** * Find integration context by external entity ID and type diff --git a/packages/core/application/commands/integration-mapping-commands.js b/packages/core/application/commands/integration-mapping-commands.js new file mode 100644 index 000000000..13b75237b --- /dev/null +++ b/packages/core/application/commands/integration-mapping-commands.js @@ -0,0 +1,25 @@ +const { + createIntegrationMappingRepository, +} = require('../../integrations/repositories/integration-mapping-repository-factory'); + +function mapErrorToResponse(error) { + return { error: 500, reason: error?.message, code: error?.code }; +} + +// Kept separate from integration-commands so the mapping and integration domains stay decoupled. +function createIntegrationMappingCommands() { + const mappingRepository = createIntegrationMappingRepository(); + + return { + // Returns a Map of integrationId → count, or an error object on failure. + async countByIntegrationIds(ids = []) { + try { + return await mappingRepository.countByIntegrationIds(ids); + } catch (error) { + return mapErrorToResponse(error); + } + }, + }; +} + +module.exports = { createIntegrationMappingCommands }; diff --git a/packages/core/application/commands/integration-mapping-commands.test.js b/packages/core/application/commands/integration-mapping-commands.test.js new file mode 100644 index 000000000..437afebd6 --- /dev/null +++ b/packages/core/application/commands/integration-mapping-commands.test.js @@ -0,0 +1,41 @@ +const mockMappingRepo = { countByIntegrationIds: jest.fn() }; + +jest.mock( + '../../integrations/repositories/integration-mapping-repository-factory', + () => ({ + createIntegrationMappingRepository: () => mockMappingRepo, + }) +); + +const { + createIntegrationMappingCommands, +} = require('./integration-mapping-commands'); + +describe('createIntegrationMappingCommands', () => { + let commands; + + beforeEach(() => { + jest.clearAllMocks(); + commands = createIntegrationMappingCommands(); + }); + + it('delegates countByIntegrationIds to the repository', async () => { + const map = new Map([['1', 3]]); + mockMappingRepo.countByIntegrationIds.mockResolvedValue(map); + + const result = await commands.countByIntegrationIds(['1']); + + expect(mockMappingRepo.countByIntegrationIds).toHaveBeenCalledWith(['1']); + expect(result).toBe(map); + }); + + it('maps a repository error to an error response', async () => { + mockMappingRepo.countByIntegrationIds.mockRejectedValue( + new Error('boom') + ); + + const result = await commands.countByIntegrationIds(['1']); + + expect(result).toMatchObject({ error: 500, reason: 'boom' }); + }); +}); diff --git a/packages/core/application/commands/report-commands.js b/packages/core/application/commands/report-commands.js new file mode 100644 index 000000000..c37a1b231 --- /dev/null +++ b/packages/core/application/commands/report-commands.js @@ -0,0 +1,188 @@ +const ERROR_CODE_MAP = { + EXECUTION_NOT_FOUND: 404, +}; + +function mapErrorToResponse(error) { + const status = ERROR_CODE_MAP[error?.code] || 500; + return { error: status, reason: error?.message, code: error?.code }; +} + +/** + * Report commands share the AdminScriptExecution store, discriminated by + * type:'REPORT'. That store has no user/integration FK, and findExecutionById + * rejects non-REPORT rows, so a report lookup can never return another + * operation type's record (ADR-010 Decision 3). + */ +function createReportCommands({ artifactRepository } = {}) { + const { + createAdminScriptExecutionRepository, + } = require('../../admin-scripts/repositories/admin-script-execution-repository-factory'); + + const executionRepository = createAdminScriptExecutionRepository(); + + let artifactRepo = artifactRepository || null; + function getArtifactRepository() { + if (!artifactRepo) { + const { + createArtifactRepository, + } = require('../../artifacts/repositories/artifact-repository-factory'); + artifactRepo = createArtifactRepository(); + } + return artifactRepo; + } + + // Resolve to null rather than throw: a read must not fail because signing did. + async function signArtifact(ref) { + if (!ref) return null; + try { + return await getArtifactRepository().signedUrl(ref); + } catch (_error) { + return null; + } + } + + return { + async createExecution({ + reportName, + reportVersion, + trigger, + mode, + input, + audit, + seriesName, + parentExecutionId, + }) { + try { + return await executionRepository.createExecution({ + name: reportName, + type: 'REPORT', + parentExecutionId, + context: { + reportVersion, + trigger, + mode: mode || 'recorded', + input, + audit, + ...(seriesName ? { seriesName } : {}), + }, + }); + } catch (error) { + return mapErrorToResponse(error); + } + }, + + async findExecutionById(id) { + try { + const record = await executionRepository.findExecutionById(id); + if (!record || record.type !== 'REPORT') { + const error = new Error(`Report execution ${id} not found`); + error.code = 'EXECUTION_NOT_FOUND'; + return mapErrorToResponse(error); + } + // The stored artifact ref is not retrievable on its own; sign it on read. + if (record.results?.artifact) { + record.results = { + ...record.results, + artifactUrl: await signArtifact(record.results.artifact), + }; + } + return record; + } catch (error) { + return mapErrorToResponse(error); + } + }, + + // Never-throws: returns [] on error (non-critical read). + async listExecutionsByName(reportName, { limit, offset, state } = {}) { + try { + return await executionRepository.findExecutionsByName( + reportName, + { type: 'REPORT', limit, offset, state } + ); + } catch (error) { + return []; + } + }, + + // No mode index in the store, so fetch by name/window and filter snapshots + // in JS. Never-throws: returns [] on error. + async findSnapshotSeries(reportName, { from, to, limit } = {}) { + try { + const rows = await executionRepository.findExecutionsByName( + reportName, + { + type: 'REPORT', + from, + to, + sortBy: 'createdAt', + sortOrder: 'asc', + limit, + } + ); + const snapshots = rows.filter( + (row) => row.context?.mode === 'snapshot' + ); + return Promise.all( + snapshots.map(async (row) => ({ + executionId: row.id, + capturedAt: row.createdAt, + summary: + row.results?.output?.summary ?? + row.results?.summary ?? + null, + artifactUrl: await signArtifact(row.results?.artifact), + })) + ); + } catch (error) { + return []; + } + }, + + async updateExecutionState(id, state) { + try { + return await executionRepository.updateExecutionState(id, state); + } catch (error) { + return mapErrorToResponse(error); + } + }, + + async appendExecutionLog(id, logEntry) { + try { + return await executionRepository.appendExecutionLog(id, logEntry); + } catch (error) { + return mapErrorToResponse(error); + } + }, + + // results.output is the inline JSON payload; results.summary + results.artifact + // are a large/binary payload's summary + object-store reference. + async completeExecution( + id, + { state, output, summary, artifact, error, metrics, logs } = {} + ) { + try { + if (state) { + await executionRepository.updateExecutionState(id, state); + } + const resultsUpdate = {}; + if (output !== undefined) resultsUpdate.output = output; + if (summary !== undefined) resultsUpdate.summary = summary; + if (artifact !== undefined) resultsUpdate.artifact = artifact; + if (error) resultsUpdate.error = error; + if (metrics) resultsUpdate.metrics = metrics; + if (logs) resultsUpdate.logs = logs; + if (Object.keys(resultsUpdate).length > 0) { + await executionRepository.updateExecutionResults( + id, + resultsUpdate + ); + } + return { success: true }; + } catch (err) { + return mapErrorToResponse(err); + } + }, + }; +} + +module.exports = { createReportCommands }; diff --git a/packages/core/application/commands/report-commands.test.js b/packages/core/application/commands/report-commands.test.js new file mode 100644 index 000000000..75edd3262 --- /dev/null +++ b/packages/core/application/commands/report-commands.test.js @@ -0,0 +1,324 @@ +const mockExecutionRepo = { + createExecution: jest.fn(), + findExecutionById: jest.fn(), + findExecutionsByName: jest.fn(), + updateExecutionState: jest.fn(), + updateExecutionResults: jest.fn(), + appendExecutionLog: jest.fn(), +}; + +jest.mock( + '../../admin-scripts/repositories/admin-script-execution-repository-factory', + () => ({ + createAdminScriptExecutionRepository: () => mockExecutionRepo, + }) +); + +const { createReportCommands } = require('./report-commands'); + +describe('createReportCommands', () => { + let commands; + let mockArtifactRepo; + + beforeEach(() => { + jest.clearAllMocks(); + mockArtifactRepo = { + signedUrl: jest + .fn() + .mockResolvedValue('https://signed.example/artifact'), + }; + commands = createReportCommands({ + artifactRepository: mockArtifactRepo, + }); + }); + + describe('createExecution', () => { + it('writes type:REPORT and puts mode/seriesName in context', async () => { + mockExecutionRepo.createExecution.mockResolvedValue({ id: 'r1' }); + + await commands.createExecution({ + reportName: 'integrations', + reportVersion: '1.0.0', + trigger: 'MANUAL', + mode: 'snapshot', + input: { windowDays: 30 }, + audit: { apiKeyLast4: '1234' }, + seriesName: 'nightly', + }); + + expect(mockExecutionRepo.createExecution).toHaveBeenCalledWith({ + name: 'integrations', + type: 'REPORT', + parentExecutionId: undefined, + context: { + reportVersion: '1.0.0', + trigger: 'MANUAL', + mode: 'snapshot', + input: { windowDays: 30 }, + audit: { apiKeyLast4: '1234' }, + seriesName: 'nightly', + }, + }); + }); + + it('defaults mode to recorded and omits seriesName when absent', async () => { + mockExecutionRepo.createExecution.mockResolvedValue({ id: 'r1' }); + + await commands.createExecution({ + reportName: 'integrations', + trigger: 'MANUAL', + }); + + const arg = mockExecutionRepo.createExecution.mock.calls[0][0]; + expect(arg.context.mode).toBe('recorded'); + expect('seriesName' in arg.context).toBe(false); + }); + + it('maps repository errors to an error response', async () => { + mockExecutionRepo.createExecution.mockRejectedValue( + new Error('DB down') + ); + + const result = await commands.createExecution({ + reportName: 'integrations', + trigger: 'MANUAL', + }); + + expect(result).toMatchObject({ error: 500, reason: 'DB down' }); + }); + }); + + describe('findExecutionById (isolation guard)', () => { + it('returns the record when it is a REPORT execution', async () => { + const rec = { id: 'r1', type: 'REPORT', state: 'COMPLETED' }; + mockExecutionRepo.findExecutionById.mockResolvedValue(rec); + + const result = await commands.findExecutionById('r1'); + + expect(result).toEqual(rec); + }); + + it('404s a non-REPORT row so a report lookup never returns a script', async () => { + mockExecutionRepo.findExecutionById.mockResolvedValue({ + id: 's1', + type: 'ADMIN_SCRIPT', + }); + + const result = await commands.findExecutionById('s1'); + + expect(result).toMatchObject({ + error: 404, + code: 'EXECUTION_NOT_FOUND', + }); + }); + + it('404s when the record is missing', async () => { + mockExecutionRepo.findExecutionById.mockResolvedValue(null); + + const result = await commands.findExecutionById('nope'); + + expect(result).toMatchObject({ + error: 404, + code: 'EXECUTION_NOT_FOUND', + }); + }); + + it('attaches a signed artifactUrl when the record has a stored artifact', async () => { + mockExecutionRepo.findExecutionById.mockResolvedValue({ + id: 'r1', + type: 'REPORT', + state: 'COMPLETED', + results: { + summary: { rows: 5 }, + artifact: { bucket: 'b', key: 'k' }, + }, + }); + + const result = await commands.findExecutionById('r1'); + + expect(mockArtifactRepo.signedUrl).toHaveBeenCalledWith({ + bucket: 'b', + key: 'k', + }); + expect(result.results.artifactUrl).toBe( + 'https://signed.example/artifact' + ); + // The raw reference is preserved alongside the URL. + expect(result.results.artifact).toEqual({ bucket: 'b', key: 'k' }); + }); + + it('does not sign when the record has no artifact', async () => { + mockExecutionRepo.findExecutionById.mockResolvedValue({ + id: 'r1', + type: 'REPORT', + state: 'COMPLETED', + results: { output: { total: 3 } }, + }); + + const result = await commands.findExecutionById('r1'); + + expect(mockArtifactRepo.signedUrl).not.toHaveBeenCalled(); + expect(result.results.artifactUrl).toBeUndefined(); + }); + }); + + describe('listExecutionsByName', () => { + it('queries by name scoped to type REPORT', async () => { + const rows = [{ id: 'r1', type: 'REPORT' }]; + mockExecutionRepo.findExecutionsByName.mockResolvedValue(rows); + + const result = await commands.listExecutionsByName('integrations', { + limit: 5, + offset: 0, + state: 'COMPLETED', + }); + + expect(result).toEqual(rows); + expect(mockExecutionRepo.findExecutionsByName).toHaveBeenCalledWith( + 'integrations', + { type: 'REPORT', limit: 5, offset: 0, state: 'COMPLETED' } + ); + }); + + it('returns [] on error (never-throw)', async () => { + mockExecutionRepo.findExecutionsByName.mockRejectedValue( + new Error('DB down') + ); + + const result = await commands.listExecutionsByName('integrations'); + + expect(result).toEqual([]); + }); + }); + + describe('findSnapshotSeries', () => { + it('windows by name, keeps only snapshots, and maps points', async () => { + const from = new Date('2025-01-01'); + const to = new Date('2025-02-01'); + const captured = new Date('2025-01-15'); + mockExecutionRepo.findExecutionsByName.mockResolvedValue([ + { + id: 'r1', + createdAt: captured, + context: { mode: 'snapshot' }, + results: { + output: { summary: { total: 7 } }, + artifact: { bucket: 'b', key: 'k' }, + }, + }, + { + id: 'r2', + createdAt: captured, + context: { mode: 'recorded' }, + results: { output: { summary: { total: 9 } } }, + }, + ]); + + const result = await commands.findSnapshotSeries('integrations', { + from, + to, + limit: 100, + }); + + expect(mockExecutionRepo.findExecutionsByName).toHaveBeenCalledWith( + 'integrations', + { + type: 'REPORT', + from, + to, + sortBy: 'createdAt', + sortOrder: 'asc', + limit: 100, + } + ); + expect(mockArtifactRepo.signedUrl).toHaveBeenCalledWith({ + bucket: 'b', + key: 'k', + }); + expect(result).toEqual([ + { + executionId: 'r1', + capturedAt: captured, + summary: { total: 7 }, + // A real signed download URL, not the raw { bucket, key } ref. + artifactUrl: 'https://signed.example/artifact', + }, + ]); + }); + + it('falls back to results.summary and null artifact', async () => { + mockExecutionRepo.findExecutionsByName.mockResolvedValue([ + { + id: 'r1', + createdAt: new Date('2025-01-15'), + context: { mode: 'snapshot' }, + results: { summary: { total: 3 } }, + }, + ]); + + const result = await commands.findSnapshotSeries('integrations'); + + expect(result).toEqual([ + { + executionId: 'r1', + capturedAt: new Date('2025-01-15'), + summary: { total: 3 }, + artifactUrl: null, + }, + ]); + }); + + it('returns [] on error (never-throw)', async () => { + mockExecutionRepo.findExecutionsByName.mockRejectedValue( + new Error('DB down') + ); + + const result = await commands.findSnapshotSeries('integrations'); + + expect(result).toEqual([]); + }); + }); + + describe('completeExecution', () => { + it('sets state and merges output/summary/artifact into results', async () => { + mockExecutionRepo.updateExecutionState.mockResolvedValue({}); + mockExecutionRepo.updateExecutionResults.mockResolvedValue({}); + + await commands.completeExecution('r1', { + state: 'COMPLETED', + output: { total: 3 }, + summary: { total: 3 }, + artifact: { bucket: 'b', key: 'k' }, + metrics: { durationMs: 12 }, + }); + + expect(mockExecutionRepo.updateExecutionState).toHaveBeenCalledWith( + 'r1', + 'COMPLETED' + ); + expect(mockExecutionRepo.updateExecutionResults).toHaveBeenCalledWith( + 'r1', + expect.objectContaining({ + output: { total: 3 }, + summary: { total: 3 }, + artifact: { bucket: 'b', key: 'k' }, + metrics: { durationMs: 12 }, + }) + ); + }); + + it('updates state only when no results fields are provided', async () => { + mockExecutionRepo.updateExecutionState.mockResolvedValue({}); + + await commands.completeExecution('r1', { state: 'FAILED' }); + + expect(mockExecutionRepo.updateExecutionState).toHaveBeenCalledWith( + 'r1', + 'FAILED' + ); + expect( + mockExecutionRepo.updateExecutionResults + ).not.toHaveBeenCalled(); + }); + }); +}); diff --git a/packages/core/artifacts/repositories/artifact-repository-factory.js b/packages/core/artifacts/repositories/artifact-repository-factory.js new file mode 100644 index 000000000..85cf56a26 --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-factory.js @@ -0,0 +1,19 @@ +// A configured bucket always wins, even in dev (unlike the encryption stage-bypass): +// a deployed Lambda writes artifacts and the router reads them in different containers, +// so routing dev to ephemeral /tmp would silently lose them. +const { ArtifactRepositoryS3 } = require('./artifact-repository-s3'); +const { ArtifactRepositoryLocal } = require('./artifact-repository-local'); + +function createArtifactRepository() { + const bucket = process.env.REPORT_ARTIFACT_BUCKET; + if (bucket) { + return new ArtifactRepositoryS3({ bucket }); + } + return new ArtifactRepositoryLocal(); +} + +module.exports = { + createArtifactRepository, + ArtifactRepositoryS3, + ArtifactRepositoryLocal, +}; diff --git a/packages/core/artifacts/repositories/artifact-repository-factory.test.js b/packages/core/artifacts/repositories/artifact-repository-factory.test.js new file mode 100644 index 000000000..186d86ee1 --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-factory.test.js @@ -0,0 +1,63 @@ +/** + * Tests for createArtifactRepository (adapter selection by env/stage). + */ +const { + createArtifactRepository, + ArtifactRepositoryS3, + ArtifactRepositoryLocal, +} = require('./artifact-repository-factory'); + +describe('createArtifactRepository', () => { + const { REPORT_ARTIFACT_BUCKET, STAGE } = process.env; + + afterEach(() => { + if (REPORT_ARTIFACT_BUCKET === undefined) { + delete process.env.REPORT_ARTIFACT_BUCKET; + } else { + process.env.REPORT_ARTIFACT_BUCKET = REPORT_ARTIFACT_BUCKET; + } + if (STAGE === undefined) { + delete process.env.STAGE; + } else { + process.env.STAGE = STAGE; + } + }); + + it('returns the S3 adapter whenever a bucket is configured', () => { + process.env.REPORT_ARTIFACT_BUCKET = 'my-bucket'; + process.env.STAGE = 'production'; + + const repo = createArtifactRepository(); + + expect(repo).toBeInstanceOf(ArtifactRepositoryS3); + expect(repo.bucket).toBe('my-bucket'); + }); + + it('returns the local adapter when no bucket is configured', () => { + delete process.env.REPORT_ARTIFACT_BUCKET; + process.env.STAGE = 'production'; + + expect(createArtifactRepository()).toBeInstanceOf( + ArtifactRepositoryLocal + ); + }); + + it('uses S3 on a deployed dev stage when the bucket is provisioned (no /tmp loss)', () => { + // A deployed dev Lambda provisions the bucket; artifacts must be durable + // there too — the executor writes and the router reads in different + // containers, so ephemeral /tmp would silently lose them. + process.env.REPORT_ARTIFACT_BUCKET = 'my-bucket'; + process.env.STAGE = 'dev'; + + expect(createArtifactRepository()).toBeInstanceOf(ArtifactRepositoryS3); + }); + + it('falls back to local when no bucket, regardless of stage', () => { + delete process.env.REPORT_ARTIFACT_BUCKET; + process.env.STAGE = 'dev'; + + expect(createArtifactRepository()).toBeInstanceOf( + ArtifactRepositoryLocal + ); + }); +}); diff --git a/packages/core/artifacts/repositories/artifact-repository-interface.js b/packages/core/artifacts/repositories/artifact-repository-interface.js new file mode 100644 index 000000000..14964841a --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-interface.js @@ -0,0 +1,27 @@ +/** + * Port for report artifacts (non-JSON output like CSV, PDF). Implementations + * write bytes to a durable store and return a location reference the caller + * persists; signed URLs are minted on read. + */ +class ArtifactRepositoryInterface { + /** + * @param {string} key - Caller-supplied, e.g. `reports/{executionId}/{name}-{stamp}.{ext}`. + * @returns {Promise<{bucket: string, key: string}>} Location reference. + */ + async put(key, body, contentType) { + throw new Error('ArtifactRepositoryInterface.put not implemented'); + } + + /** + * @param {{bucket: string, key: string}|string} ref - Reference from put(), or a bare key. + */ + async signedUrl(ref, { expiresIn } = {}) { + throw new Error('ArtifactRepositoryInterface.signedUrl not implemented'); + } + + async get(key) { + throw new Error('ArtifactRepositoryInterface.get not implemented'); + } +} + +module.exports = { ArtifactRepositoryInterface }; diff --git a/packages/core/artifacts/repositories/artifact-repository-local.js b/packages/core/artifacts/repositories/artifact-repository-local.js new file mode 100644 index 000000000..dcc60544f --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-local.js @@ -0,0 +1,42 @@ +/** + * Dev/test artifact adapter: writes to a local directory and returns file:// URLs. + * Selected by the factory only when no object store is configured (no REPORT_ARTIFACT_BUCKET). + */ +const fs = require('fs'); +const path = require('path'); +const os = require('os'); +const { + ArtifactRepositoryInterface, +} = require('./artifact-repository-interface'); + +class ArtifactRepositoryLocal extends ArtifactRepositoryInterface { + constructor({ baseDir } = {}) { + super(); + this.baseDir = + baseDir || + process.env.REPORT_ARTIFACT_DIR || + path.join(os.tmpdir(), 'frigg-report-artifacts'); + } + + _pathFor(key) { + return path.join(this.baseDir, key); + } + + async put(key, body, _contentType) { + const filePath = this._pathFor(key); + await fs.promises.mkdir(path.dirname(filePath), { recursive: true }); + await fs.promises.writeFile(filePath, body); + return { bucket: this.baseDir, key }; + } + + async signedUrl(ref, _options = {}) { + const key = ref?.key || ref; + return `file://${this._pathFor(key)}`; + } + + async get(key) { + return fs.promises.readFile(this._pathFor(key), 'utf8'); + } +} + +module.exports = { ArtifactRepositoryLocal }; diff --git a/packages/core/artifacts/repositories/artifact-repository-local.test.js b/packages/core/artifacts/repositories/artifact-repository-local.test.js new file mode 100644 index 000000000..3c9672fad --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-local.test.js @@ -0,0 +1,54 @@ +/** + * Tests for ArtifactRepositoryLocal (filesystem adapter — no network). + */ +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { + ArtifactRepositoryLocal, +} = require('./artifact-repository-local'); + +describe('ArtifactRepositoryLocal', () => { + let baseDir; + let repo; + + beforeEach(() => { + baseDir = fs.mkdtempSync( + path.join(os.tmpdir(), 'frigg-artifacts-test-') + ); + repo = new ArtifactRepositoryLocal({ baseDir }); + }); + + afterEach(() => { + fs.rmSync(baseDir, { recursive: true, force: true }); + }); + + it('put() writes the body under baseDir and returns a reference', async () => { + const ref = await repo.put( + 'reports/exec-1/sales.csv', + 'a,b\n1,2\n', + 'text/csv' + ); + + expect(ref).toEqual({ bucket: baseDir, key: 'reports/exec-1/sales.csv' }); + const written = fs.readFileSync( + path.join(baseDir, 'reports/exec-1/sales.csv'), + 'utf8' + ); + expect(written).toBe('a,b\n1,2\n'); + }); + + it('get() reads back what put() wrote', async () => { + await repo.put('reports/exec-1/sales.csv', 'hello', 'text/csv'); + const body = await repo.get('reports/exec-1/sales.csv'); + expect(body).toBe('hello'); + }); + + it('signedUrl() returns a file:// URL for the key', async () => { + const ref = await repo.put('reports/exec-1/sales.csv', 'x', 'text/csv'); + const url = await repo.signedUrl(ref); + expect(url).toBe( + `file://${path.join(baseDir, 'reports/exec-1/sales.csv')}` + ); + }); +}); diff --git a/packages/core/artifacts/repositories/artifact-repository-s3.js b/packages/core/artifacts/repositories/artifact-repository-s3.js new file mode 100644 index 000000000..9fce1c2dc --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-s3.js @@ -0,0 +1,61 @@ +// aws-sdk modules are lazy-required so importing this file (and the factory that +// references it) never forces the SDK to load for contexts that only need the local adapter. +const { + ArtifactRepositoryInterface, +} = require('./artifact-repository-interface'); + +class ArtifactRepositoryS3 extends ArtifactRepositoryInterface { + constructor({ bucket, region, s3Client } = {}) { + super(); + this.bucket = bucket || process.env.REPORT_ARTIFACT_BUCKET; + this.region = region || process.env.AWS_REGION || 'us-east-1'; + this._s3Client = s3Client || null; + } + + _getClient() { + if (!this._s3Client) { + const { S3Client } = require('@aws-sdk/client-s3'); + this._s3Client = new S3Client({ region: this.region }); + } + return this._s3Client; + } + + async put(key, body, contentType) { + if (!this.bucket) { + throw new Error( + 'REPORT_ARTIFACT_BUCKET is not configured; cannot store report artifact' + ); + } + const { PutObjectCommand } = require('@aws-sdk/client-s3'); + // No ACL set: the bucket blocks public access, so objects stay private. + await this._getClient().send( + new PutObjectCommand({ + Bucket: this.bucket, + Key: key, + Body: body, + ContentType: contentType, + ServerSideEncryption: 'AES256', + }) + ); + return { bucket: this.bucket, key }; + } + + async signedUrl(ref, { expiresIn = 3600 } = {}) { + const { GetObjectCommand } = require('@aws-sdk/client-s3'); + const { getSignedUrl } = require('@aws-sdk/s3-request-presigner'); + const bucket = ref?.bucket || this.bucket; + const key = ref?.key || ref; + const command = new GetObjectCommand({ Bucket: bucket, Key: key }); + return getSignedUrl(this._getClient(), command, { expiresIn }); + } + + async get(key) { + const { GetObjectCommand } = require('@aws-sdk/client-s3'); + const response = await this._getClient().send( + new GetObjectCommand({ Bucket: this.bucket, Key: key }) + ); + return response.Body.transformToString(); + } +} + +module.exports = { ArtifactRepositoryS3 }; diff --git a/packages/core/artifacts/repositories/artifact-repository-s3.test.js b/packages/core/artifacts/repositories/artifact-repository-s3.test.js new file mode 100644 index 000000000..c22956772 --- /dev/null +++ b/packages/core/artifacts/repositories/artifact-repository-s3.test.js @@ -0,0 +1,112 @@ +/** + * Tests for ArtifactRepositoryS3 + * + * The aws-sdk clients are never hit over the network: an S3 client with a + * mocked send() is injected, and the presigner module (not installed in this + * workspace) is virtually mocked so signedUrl stays offline. + */ +jest.mock( + '@aws-sdk/s3-request-presigner', + () => ({ getSignedUrl: jest.fn() }), + { virtual: true } +); + +const { getSignedUrl } = require('@aws-sdk/s3-request-presigner'); +const { ArtifactRepositoryS3 } = require('./artifact-repository-s3'); + +describe('ArtifactRepositoryS3', () => { + let s3Client; + + beforeEach(() => { + s3Client = { send: jest.fn().mockResolvedValue({}) }; + getSignedUrl.mockReset(); + }); + + describe('put()', () => { + it('writes a private, SSE-encrypted object and returns {bucket, key}', async () => { + const repo = new ArtifactRepositoryS3({ + bucket: 'my-bucket', + s3Client, + }); + + const ref = await repo.put( + 'reports/exec-1/sales-2026.csv', + 'a,b\n1,2\n', + 'text/csv' + ); + + expect(ref).toEqual({ + bucket: 'my-bucket', + key: 'reports/exec-1/sales-2026.csv', + }); + expect(s3Client.send).toHaveBeenCalledTimes(1); + const command = s3Client.send.mock.calls[0][0]; + expect(command.input).toMatchObject({ + Bucket: 'my-bucket', + Key: 'reports/exec-1/sales-2026.csv', + Body: 'a,b\n1,2\n', + ContentType: 'text/csv', + ServerSideEncryption: 'AES256', + }); + // Private object: never set an ACL. + expect(command.input.ACL).toBeUndefined(); + }); + + it('throws when no bucket is configured', async () => { + const repo = new ArtifactRepositoryS3({ + bucket: undefined, + s3Client, + }); + + await expect( + repo.put('k', 'body', 'text/csv') + ).rejects.toThrow(/REPORT_ARTIFACT_BUCKET/); + expect(s3Client.send).not.toHaveBeenCalled(); + }); + }); + + describe('signedUrl()', () => { + it('presigns a GET for the referenced object', async () => { + getSignedUrl.mockResolvedValue('https://signed.example/object'); + const repo = new ArtifactRepositoryS3({ + bucket: 'my-bucket', + s3Client, + }); + + const url = await repo.signedUrl( + { bucket: 'my-bucket', key: 'reports/exec-1/sales-2026.csv' }, + { expiresIn: 120 } + ); + + expect(url).toBe('https://signed.example/object'); + expect(getSignedUrl).toHaveBeenCalledTimes(1); + const [, command, options] = getSignedUrl.mock.calls[0]; + expect(command.input).toMatchObject({ + Bucket: 'my-bucket', + Key: 'reports/exec-1/sales-2026.csv', + }); + expect(options).toEqual({ expiresIn: 120 }); + }); + }); + + describe('get()', () => { + it('returns the object body as a string', async () => { + s3Client.send.mockResolvedValue({ + Body: { transformToString: async () => 'file-contents' }, + }); + const repo = new ArtifactRepositoryS3({ + bucket: 'my-bucket', + s3Client, + }); + + const body = await repo.get('reports/exec-1/sales-2026.csv'); + + expect(body).toBe('file-contents'); + const command = s3Client.send.mock.calls[0][0]; + expect(command.input).toMatchObject({ + Bucket: 'my-bucket', + Key: 'reports/exec-1/sales-2026.csv', + }); + }); + }); +}); diff --git a/packages/core/credential/repositories/credential-active-type.js b/packages/core/credential/repositories/credential-active-type.js new file mode 100644 index 000000000..3a1a116f7 --- /dev/null +++ b/packages/core/credential/repositories/credential-active-type.js @@ -0,0 +1,32 @@ +const UNKNOWN_TYPE = 'unknown'; + +/** + * Type is derived from the related Entity.moduleName since Credential carries + * no type column. Counted once per distinct linked moduleName; none → `unknown`. + * Reads only the non-encrypted projection — never `data`/secrets. + */ +function tallyActiveCredentialsByType(credentials = []) { + const counts = new Map(); + + for (const credential of credentials) { + const modules = new Set( + (credential?.entities || []) + .map((entity) => entity?.moduleName) + .filter( + (moduleName) => moduleName != null && moduleName !== '' + ) + ); + const types = modules.size > 0 ? [...modules] : [UNKNOWN_TYPE]; + + for (const type of types) { + counts.set(type, (counts.get(type) || 0) + 1); + } + } + + return [...counts].map(([integrationType, count]) => ({ + integrationType, + count, + })); +} + +module.exports = { tallyActiveCredentialsByType, UNKNOWN_TYPE }; diff --git a/packages/core/credential/repositories/credential-active-type.test.js b/packages/core/credential/repositories/credential-active-type.test.js new file mode 100644 index 000000000..4a4f38a71 --- /dev/null +++ b/packages/core/credential/repositories/credential-active-type.test.js @@ -0,0 +1,51 @@ +const { + tallyActiveCredentialsByType, + UNKNOWN_TYPE, +} = require('./credential-active-type'); + +describe('tallyActiveCredentialsByType', () => { + it('counts credentials grouped by their related Entity.moduleName', () => { + const result = tallyActiveCredentialsByType([ + { entities: [{ moduleName: 'hubspot' }] }, + { entities: [{ moduleName: 'hubspot' }] }, + { entities: [{ moduleName: 'salesforce' }] }, + ]); + + expect(result).toEqual([ + { integrationType: 'hubspot', count: 2 }, + { integrationType: 'salesforce', count: 1 }, + ]); + }); + + it('counts a credential once per DISTINCT module (no double-count within a module)', () => { + const result = tallyActiveCredentialsByType([ + { + entities: [ + { moduleName: 'hubspot' }, + { moduleName: 'hubspot' }, + { moduleName: 'slack' }, + ], + }, + ]); + + expect(result).toEqual([ + { integrationType: 'hubspot', count: 1 }, + { integrationType: 'slack', count: 1 }, + ]); + }); + + it('buckets credentials with no linked module under UNKNOWN_TYPE', () => { + const result = tallyActiveCredentialsByType([ + { entities: [] }, + { entities: [{ moduleName: null }] }, + {}, + ]); + + expect(result).toEqual([{ integrationType: UNKNOWN_TYPE, count: 3 }]); + }); + + it('returns an empty array for no credentials', () => { + expect(tallyActiveCredentialsByType([])).toEqual([]); + expect(tallyActiveCredentialsByType()).toEqual([]); + }); +}); diff --git a/packages/core/credential/repositories/credential-repository-documentdb.js b/packages/core/credential/repositories/credential-repository-documentdb.js index d79f88985..76cec105c 100644 --- a/packages/core/credential/repositories/credential-repository-documentdb.js +++ b/packages/core/credential/repositories/credential-repository-documentdb.js @@ -2,6 +2,7 @@ const { prisma } = require('../../database/prisma'); const { toObjectId, fromObjectId, + findManyDrained, findOne, insertOne, updateOne, @@ -10,6 +11,7 @@ const { const { CredentialRepositoryInterface, } = require('./credential-repository-interface'); +const { tallyActiveCredentialsByType } = require('./credential-active-type'); const { DocumentDBEncryptionService, } = require('../../database/documentdb-encryption-service'); @@ -238,6 +240,55 @@ class CredentialRepositoryDocumentDB extends CredentialRepositoryInterface { return this._mapCredential(decryptedCredential); } + /** + * Count credentials active since a timestamp, grouped by integration type. + * Projected raw reads only; the encrypted `data` is never fetched, so secrets are never decrypted for this read. + * @returns {Promise>} + */ + async countActiveByType({ since } = {}) { + const filter = {}; + // Coerce to Date: a raw string never matches the BSON date $gte (type bracketing). + if (since) filter.updatedAt = { $gte: new Date(since) }; + + // Drained: a deployment-wide scan must not truncate at DocumentDB's ~101-doc first batch. + const activeCredentials = await findManyDrained( + this.prisma, + 'Credential', + filter, + { projection: { _id: 1 } } + ); + + const credentialIds = activeCredentials + .map((doc) => toObjectId(doc._id)) + .filter(Boolean); + + const entities = credentialIds.length + ? await findManyDrained( + this.prisma, + 'Entity', + { credentialId: { $in: credentialIds } }, + { projection: { credentialId: 1, moduleName: 1 } } + ) + : []; + + const entitiesByCredential = new Map(); + for (const entity of entities) { + const key = fromObjectId(entity.credentialId); + if (!entitiesByCredential.has(key)) { + entitiesByCredential.set(key, []); + } + entitiesByCredential + .get(key) + .push({ moduleName: entity.moduleName }); + } + + const credentials = activeCredentials.map((doc) => ({ + entities: entitiesByCredential.get(fromObjectId(doc._id)) || [], + })); + + return tallyActiveCredentialsByType(credentials); + } + _buildIdentifierFilter(identifiers) { const filter = {}; if (identifiers._id || identifiers.id) { diff --git a/packages/core/credential/repositories/credential-repository-documentdb.test.js b/packages/core/credential/repositories/credential-repository-documentdb.test.js new file mode 100644 index 000000000..005618e8a --- /dev/null +++ b/packages/core/credential/repositories/credential-repository-documentdb.test.js @@ -0,0 +1,105 @@ +jest.mock('../../database/prisma', () => ({ + prisma: { $runCommandRaw: jest.fn() }, +})); +jest.mock('../../database/documentdb-encryption-service'); + +const { ObjectId } = require('bson'); +const { prisma } = require('../../database/prisma'); +const { + DocumentDBEncryptionService, +} = require('../../database/documentdb-encryption-service'); +const { + CredentialRepositoryDocumentDB, +} = require('./credential-repository-documentdb'); + +describe('CredentialRepositoryDocumentDB.countActiveByType', () => { + let repo; + let encryptionService; + + beforeEach(() => { + jest.clearAllMocks(); + encryptionService = { + encryptFields: jest.fn(), + decryptFields: jest.fn(), + }; + DocumentDBEncryptionService.mockImplementation(() => encryptionService); + repo = new CredentialRepositoryDocumentDB(); + }); + + it('filters updatedAt >= since, projects away secrets, and groups by related Entity.moduleName', async () => { + const credA = new ObjectId(); + const credB = new ObjectId(); + const credC = new ObjectId(); + const since = new Date('2026-06-01T00:00:00Z'); + + prisma.$runCommandRaw + // find active Credentials (projection excludes `data`) + .mockResolvedValueOnce({ + cursor: { + firstBatch: [ + { _id: credA }, + { _id: credB }, + { _id: credC }, + ], + }, + }) + // find Entities for those credential ids + .mockResolvedValueOnce({ + cursor: { + firstBatch: [ + { credentialId: credA, moduleName: 'hubspot' }, + { credentialId: credB, moduleName: 'hubspot' }, + // credC has no entity → bucketed as unknown + ], + }, + }); + + const result = await repo.countActiveByType({ since }); + + const credCmd = prisma.$runCommandRaw.mock.calls[0][0]; + expect(credCmd.find).toBe('Credential'); + expect(credCmd.filter).toEqual({ updatedAt: { $gte: since } }); + expect(credCmd.projection).toEqual({ _id: 1 }); + + const entityCmd = prisma.$runCommandRaw.mock.calls[1][0]; + expect(entityCmd.find).toBe('Entity'); + expect(entityCmd.filter.credentialId.$in).toHaveLength(3); + expect(entityCmd.projection).toEqual({ + credentialId: 1, + moduleName: 1, + }); + + // Neither read touched the encrypted `data`, so nothing is decrypted. + expect(encryptionService.decryptFields).not.toHaveBeenCalled(); + + expect(result).toEqual([ + { integrationType: 'hubspot', count: 2 }, + { integrationType: 'unknown', count: 1 }, + ]); + }); + + it('skips the entity query and returns [] when no credentials are active', async () => { + prisma.$runCommandRaw.mockResolvedValueOnce({ + cursor: { firstBatch: [] }, + }); + + const result = await repo.countActiveByType({ + since: new Date('2026-06-01T00:00:00Z'), + }); + + expect(prisma.$runCommandRaw).toHaveBeenCalledTimes(1); + expect(encryptionService.decryptFields).not.toHaveBeenCalled(); + expect(result).toEqual([]); + }); + + it('omits the updatedAt filter when since is not provided (count-all)', async () => { + prisma.$runCommandRaw.mockResolvedValueOnce({ + cursor: { firstBatch: [] }, + }); + + await repo.countActiveByType(); + + const credCmd = prisma.$runCommandRaw.mock.calls[0][0]; + expect(credCmd.filter).toEqual({}); + }); +}); diff --git a/packages/core/credential/repositories/credential-repository-interface.js b/packages/core/credential/repositories/credential-repository-interface.js index 5eeade4e0..25575156d 100644 --- a/packages/core/credential/repositories/credential-repository-interface.js +++ b/packages/core/credential/repositories/credential-repository-interface.js @@ -93,6 +93,21 @@ class CredentialRepositoryInterface { 'Method updateCredential must be implemented by subclass' ); } + + /** + * Count credentials active (updatedAt >= since) grouped by integration type. + * Reads ONLY non-encrypted fields — never the encrypted `data` JSON. + * + * @param {Object} params + * @param {Date} [params.since] - Lower bound on updatedAt + * @returns {Promise>} + * @abstract + */ + async countActiveByType(/* { since } */) { + throw new Error( + 'Method countActiveByType must be implemented by subclass' + ); + } } module.exports = { CredentialRepositoryInterface }; diff --git a/packages/core/credential/repositories/credential-repository-mongo.js b/packages/core/credential/repositories/credential-repository-mongo.js index dcf1ed8f9..cec35676c 100644 --- a/packages/core/credential/repositories/credential-repository-mongo.js +++ b/packages/core/credential/repositories/credential-repository-mongo.js @@ -2,6 +2,7 @@ const { prisma } = require('../../database/prisma'); const { CredentialRepositoryInterface, } = require('./credential-repository-interface'); +const { tallyActiveCredentialsByType } = require('./credential-active-type'); /** * MongoDB Credential Repository Adapter @@ -231,6 +232,30 @@ class CredentialRepositoryMongo extends CredentialRepositoryInterface { }; } + /** + * Count credentials active since a timestamp, grouped by integration type. + * + * @param {Object} params + * @param {Date} [params.since] - Lower bound on updatedAt + * @returns {Promise>} + */ + async countActiveByType({ since } = {}) { + const where = {}; + if (since) where.updatedAt = { gte: since }; + + // Type lives on the related Entity.moduleName, not on Credential. Select + // only that field (+ id) so the encrypted `data` JSON is never decrypted. + const credentials = await this.prisma.credential.findMany({ + where, + select: { + id: true, + entities: { select: { moduleName: true } }, + }, + }); + + return tallyActiveCredentialsByType(credentials); + } + /** * Convert identifiers to Prisma where clause * @private diff --git a/packages/core/credential/repositories/credential-repository-mongo.test.js b/packages/core/credential/repositories/credential-repository-mongo.test.js new file mode 100644 index 000000000..a4ee4b6f1 --- /dev/null +++ b/packages/core/credential/repositories/credential-repository-mongo.test.js @@ -0,0 +1,60 @@ +jest.mock('../../database/prisma', () => ({ + prisma: { + credential: { + findMany: jest.fn(), + }, + }, +})); + +const { prisma } = require('../../database/prisma'); +const { CredentialRepositoryMongo } = require('./credential-repository-mongo'); + +describe('CredentialRepositoryMongo.countActiveByType', () => { + let repo; + + beforeEach(() => { + jest.clearAllMocks(); + repo = new CredentialRepositoryMongo(); + }); + + it('filters updatedAt >= since and groups by the related Entity.moduleName', async () => { + prisma.credential.findMany.mockResolvedValue([ + { id: 'a', entities: [{ moduleName: 'hubspot' }] }, + { id: 'b', entities: [{ moduleName: 'salesforce' }] }, + { id: 'c', entities: [{ moduleName: 'salesforce' }] }, + ]); + const since = new Date('2026-06-01T00:00:00Z'); + + const result = await repo.countActiveByType({ since }); + + const arg = prisma.credential.findMany.mock.calls[0][0]; + expect(arg.where).toEqual({ updatedAt: { gte: since } }); + expect(result).toEqual([ + { integrationType: 'hubspot', count: 1 }, + { integrationType: 'salesforce', count: 2 }, + ]); + }); + + it('selects ONLY non-encrypted fields — never the encrypted `data`', async () => { + prisma.credential.findMany.mockResolvedValue([]); + + await repo.countActiveByType({ since: new Date() }); + + const arg = prisma.credential.findMany.mock.calls[0][0]; + expect(arg.select).toEqual({ + id: true, + entities: { select: { moduleName: true } }, + }); + expect(arg.select).not.toHaveProperty('data'); + expect(arg.include).toBeUndefined(); + }); + + it('omits the updatedAt filter when since is not provided (count-all)', async () => { + prisma.credential.findMany.mockResolvedValue([]); + + await repo.countActiveByType(); + + const arg = prisma.credential.findMany.mock.calls[0][0]; + expect(arg.where).toEqual({}); + }); +}); diff --git a/packages/core/credential/repositories/credential-repository-postgres.js b/packages/core/credential/repositories/credential-repository-postgres.js index 8c25a28dd..d76bfc07e 100644 --- a/packages/core/credential/repositories/credential-repository-postgres.js +++ b/packages/core/credential/repositories/credential-repository-postgres.js @@ -2,6 +2,7 @@ const { prisma } = require('../../database/prisma'); const { CredentialRepositoryInterface, } = require('./credential-repository-interface'); +const { tallyActiveCredentialsByType } = require('./credential-active-type'); /** * PostgreSQL Credential Repository Adapter @@ -248,6 +249,30 @@ class CredentialRepositoryPostgres extends CredentialRepositoryInterface { }; } + /** + * Count credentials active since a timestamp, grouped by integration type. + * + * @param {Object} params + * @param {Date} [params.since] - Lower bound on updatedAt + * @returns {Promise>} + */ + async countActiveByType({ since } = {}) { + const where = {}; + if (since) where.updatedAt = { gte: since }; + + // Select only the entity moduleName (+ id) so the encrypted `data` JSON + // is never read and the encryption extension has nothing to decrypt. + const credentials = await this.prisma.credential.findMany({ + where, + select: { + id: true, + entities: { select: { moduleName: true } }, + }, + }); + + return tallyActiveCredentialsByType(credentials); + } + /** * Convert identifiers to Prisma where clause (converting IDs to Int) * @private diff --git a/packages/core/credential/repositories/credential-repository-postgres.test.js b/packages/core/credential/repositories/credential-repository-postgres.test.js new file mode 100644 index 000000000..7a75ecd76 --- /dev/null +++ b/packages/core/credential/repositories/credential-repository-postgres.test.js @@ -0,0 +1,65 @@ +jest.mock('../../database/prisma', () => ({ + prisma: { + credential: { + findMany: jest.fn(), + }, + }, +})); + +const { prisma } = require('../../database/prisma'); +const { + CredentialRepositoryPostgres, +} = require('./credential-repository-postgres'); + +describe('CredentialRepositoryPostgres.countActiveByType', () => { + let repo; + + beforeEach(() => { + jest.clearAllMocks(); + repo = new CredentialRepositoryPostgres(); + }); + + it('filters updatedAt >= since and groups by the related Entity.moduleName', async () => { + prisma.credential.findMany.mockResolvedValue([ + { id: 1, entities: [{ moduleName: 'hubspot' }] }, + { id: 2, entities: [{ moduleName: 'hubspot' }] }, + { id: 3, entities: [{ moduleName: 'salesforce' }] }, + { id: 4, entities: [] }, + ]); + const since = new Date('2026-06-01T00:00:00Z'); + + const result = await repo.countActiveByType({ since }); + + const arg = prisma.credential.findMany.mock.calls[0][0]; + expect(arg.where).toEqual({ updatedAt: { gte: since } }); + expect(result).toEqual([ + { integrationType: 'hubspot', count: 2 }, + { integrationType: 'salesforce', count: 1 }, + { integrationType: 'unknown', count: 1 }, + ]); + }); + + it('selects ONLY non-encrypted fields — never the encrypted `data`', async () => { + prisma.credential.findMany.mockResolvedValue([]); + + await repo.countActiveByType({ since: new Date() }); + + const arg = prisma.credential.findMany.mock.calls[0][0]; + expect(arg.select).toEqual({ + id: true, + entities: { select: { moduleName: true } }, + }); + // The secret store must never be projected or included. + expect(arg.select).not.toHaveProperty('data'); + expect(arg.include).toBeUndefined(); + }); + + it('omits the updatedAt filter when since is not provided (count-all)', async () => { + prisma.credential.findMany.mockResolvedValue([]); + + await repo.countActiveByType(); + + const arg = prisma.credential.findMany.mock.calls[0][0]; + expect(arg.where).toEqual({}); + }); +}); diff --git a/packages/core/database/documentdb-utils.drain.test.js b/packages/core/database/documentdb-utils.drain.test.js new file mode 100644 index 000000000..3865e31ac --- /dev/null +++ b/packages/core/database/documentdb-utils.drain.test.js @@ -0,0 +1,97 @@ +/** + * Tests for the cursor-draining helpers hoisted into documentdb-utils. + * These back the deployment-wide report reads (integration/mapping/credential + * DocumentDB adapters) that must NOT truncate at the ~101-doc first batch. + */ +const { findManyDrained, aggregateDrained } = require('./documentdb-utils'); + +function clientReturning(...responses) { + const calls = []; + const $runCommandRaw = jest.fn(async (command) => { + calls.push(command); + return responses[calls.length - 1]; + }); + return { client: { $runCommandRaw }, calls }; +} + +describe('findManyDrained', () => { + it('drains multiple batches via getMore until the cursor closes', async () => { + const { client, calls } = clientReturning( + { cursor: { id: 7, firstBatch: [{ _id: 'a' }, { _id: 'b' }] } }, + { cursor: { id: 7, nextBatch: [{ _id: 'c' }] } }, + { cursor: { id: 0, nextBatch: [{ _id: 'd' }] } } + ); + + const docs = await findManyDrained(client, 'Credential', { + updatedAt: { $gte: new Date('2026-01-01') }, + }); + + expect(docs.map((d) => d._id)).toEqual(['a', 'b', 'c', 'd']); + expect(calls[0]).toMatchObject({ find: 'Credential', batchSize: 1000 }); + expect(calls[1]).toMatchObject({ getMore: 7, collection: 'Credential' }); + }); + + it('forwards a projection and stops on an empty batch', async () => { + const { client, calls } = clientReturning( + { cursor: { id: 9, firstBatch: [{ _id: 'a' }] } }, + { cursor: { id: 9, nextBatch: [] } } + ); + + const docs = await findManyDrained( + client, + 'Credential', + {}, + { projection: { _id: 1 } } + ); + + expect(docs).toHaveLength(1); + expect(calls[0].projection).toEqual({ _id: 1 }); + }); + + it('treats an extended-JSON {$numberLong} cursor id as open', async () => { + const { client } = clientReturning( + { + cursor: { + id: { $numberLong: '9007199254740993' }, + firstBatch: [{ _id: 'a' }], + }, + }, + { cursor: { id: { $numberLong: '0' }, nextBatch: [{ _id: 'b' }] } } + ); + + const docs = await findManyDrained(client, 'Credential', {}); + expect(docs).toHaveLength(2); + }); + + it('does not call getMore when the first batch closes the cursor', async () => { + const { client, calls } = clientReturning({ + cursor: { id: 0, firstBatch: [{ _id: 'only' }] }, + }); + + const docs = await findManyDrained(client, 'Credential', {}); + expect(docs).toHaveLength(1); + expect(calls).toHaveLength(1); + }); +}); + +describe('aggregateDrained', () => { + it('drains grouped aggregation results across batches', async () => { + const { client, calls } = clientReturning( + { cursor: { id: 3, firstBatch: [{ _id: '1', count: 2 }] } }, + { cursor: { id: 0, nextBatch: [{ _id: '2', count: 5 }] } } + ); + + const rows = await aggregateDrained(client, 'IntegrationMapping', [ + { $group: { _id: '$integrationId', count: { $sum: 1 } } }, + ]); + + expect(rows).toEqual([ + { _id: '1', count: 2 }, + { _id: '2', count: 5 }, + ]); + expect(calls[0]).toMatchObject({ + aggregate: 'IntegrationMapping', + cursor: { batchSize: 1000 }, + }); + }); +}); diff --git a/packages/core/database/documentdb-utils.js b/packages/core/database/documentdb-utils.js index 0aba93eb2..6f2991fc8 100644 --- a/packages/core/database/documentdb-utils.js +++ b/packages/core/database/documentdb-utils.js @@ -121,6 +121,60 @@ async function aggregate(client, collection, pipeline) { return result?.cursor?.firstBatch || []; } +// findMany/aggregate return ONLY the first batch (~101 docs); the drained variants below follow the cursor to completion so full scans don't silently truncate. +const DRAIN_BATCH_SIZE = 1000; +const MAX_DRAIN_BATCHES = 100000; + +function isCursorOpen(id) { + if (id === undefined || id === null) return false; + if (typeof id === 'number') return id !== 0; + if (typeof id === 'bigint') return id !== 0n; + // Extended JSON can surface a 64-bit cursor id as { $numberLong: "..." }. + if (typeof id === 'object' && id.$numberLong !== undefined) { + return id.$numberLong !== '0'; + } + return String(id) !== '0'; +} + +async function drainCursor(client, collection, firstResult) { + const cursor = firstResult?.cursor || {}; + const docs = [...(cursor.firstBatch || [])]; + let cursorId = cursor.id; + let batches = 0; + + while (isCursorOpen(cursorId) && batches < MAX_DRAIN_BATCHES) { + batches += 1; + const next = await client.$runCommandRaw({ + getMore: cursorId, + collection, + batchSize: DRAIN_BATCH_SIZE, + }); + const nextCursor = next?.cursor || {}; + const nextBatch = nextCursor.nextBatch || []; + docs.push(...nextBatch); + cursorId = nextCursor.id; + if (nextBatch.length === 0) break; + } + return docs; +} + +async function findManyDrained(client, collection, filter = {}, options = {}) { + const command = { find: collection, filter, batchSize: DRAIN_BATCH_SIZE }; + if (options.projection) command.projection = options.projection; + if (options.sort) command.sort = options.sort; + const first = await client.$runCommandRaw(command); + return drainCursor(client, collection, first); +} + +async function aggregateDrained(client, collection, pipeline) { + const first = await client.$runCommandRaw({ + aggregate: collection, + pipeline, + cursor: { batchSize: DRAIN_BATCH_SIZE }, + }); + return drainCursor(client, collection, first); +} + module.exports = { toObjectId, toObjectIdArray, @@ -132,5 +186,7 @@ module.exports = { deleteOne, deleteMany, aggregate, + findManyDrained, + aggregateDrained, }; diff --git a/packages/core/handlers/app-definition-loader.js b/packages/core/handlers/app-definition-loader.js index b53200947..724260dba 100644 --- a/packages/core/handlers/app-definition-loader.js +++ b/packages/core/handlers/app-definition-loader.js @@ -8,7 +8,7 @@ const { resolveTelemetryConfig } = require('../telemetry/telemetry-config'); * @function loadAppDefinition * @description Searches for the nearest backend package.json, loads the corresponding index.js file, * and extracts the application definition containing integrations and user configuration. - * @returns {{integrations: Array, userConfig: object | null, adminScripts: Array, admin: object, telemetry: object}} An object containing the application definition. + * @returns {{integrations: Array, userConfig: object | null, adminScripts: Array, reports: Array, admin: object, telemetry: object}} An object containing the application definition. * @throws {Error} Throws error if backend package.json cannot be found. * @throws {Error} Throws error if index.js file cannot be found in the backend directory. * @example @@ -34,6 +34,7 @@ function loadAppDefinition() { integrations = [], user: userConfig = null, adminScripts = [], + reports = [], admin = {}, } = appDefinition; @@ -58,7 +59,7 @@ function loadAppDefinition() { }; } - return { integrations, userConfig, adminScripts, admin, telemetry }; + return { integrations, userConfig, adminScripts, reports, admin, telemetry }; } module.exports = { diff --git a/packages/core/handlers/routers/reporting.js b/packages/core/handlers/routers/reporting.js deleted file mode 100644 index 8c0825d4f..000000000 --- a/packages/core/handlers/routers/reporting.js +++ /dev/null @@ -1,9 +0,0 @@ -const { createReportingRouter } = require('@friggframework/core'); -const { createAppHandler } = require('./../app-handler-helpers'); - -const router = createReportingRouter(); - -// true → eager-connect Prisma; the reporting endpoints read the DB. -const handler = createAppHandler('HTTP Event: Reporting', router, true); - -module.exports = { handler, router }; diff --git a/packages/core/index.js b/packages/core/index.js index c6aa12288..f5d2b2764 100644 --- a/packages/core/index.js +++ b/packages/core/index.js @@ -66,8 +66,10 @@ const { LoadIntegrationContextUseCase, } = require('./integrations/index'); const { - createReportingRouter, - createReportingRepository, + ReportBase, + IntegrationsReport, + BUILTIN_REPORTS, + createReportCommands, } = require('./reporting/index'); const { createTelemetry, @@ -144,8 +146,10 @@ module.exports = { GetProcess, // reporting - createReportingRouter, - createReportingRepository, + ReportBase, + IntegrationsReport, + BUILTIN_REPORTS, + createReportCommands, // telemetry createTelemetry, diff --git a/packages/core/integrations/repositories/integration-mapping-repository-documentdb.js b/packages/core/integrations/repositories/integration-mapping-repository-documentdb.js index 9b48bf17a..07b28bc75 100644 --- a/packages/core/integrations/repositories/integration-mapping-repository-documentdb.js +++ b/packages/core/integrations/repositories/integration-mapping-repository-documentdb.js @@ -8,6 +8,7 @@ const { updateOne, deleteOne, deleteMany, + aggregateDrained, } = require('../../database/documentdb-utils'); const { IntegrationMappingRepositoryInterface, @@ -15,6 +16,7 @@ const { const { DocumentDBEncryptionService, } = require('../../database/documentdb-encryption-service'); + class IntegrationMappingRepositoryDocumentDB extends IntegrationMappingRepositoryInterface { constructor() { super(); @@ -22,6 +24,27 @@ class IntegrationMappingRepositoryDocumentDB extends IntegrationMappingRepositor this.encryptionService = new DocumentDBEncryptionService(); } + /** + * integrationId is stored as a string in DocumentDB, so ids are matched as + * strings (an ObjectId $in would never match). Drains the grouped cursor so + * a deployment-wide count is not truncated at the first batch. + */ + async countByIntegrationIds(ids = []) { + const counts = new Map(); + if (!ids || ids.length === 0) return counts; + + const stringIds = ids.map(String); + const rows = await aggregateDrained(this.prisma, 'IntegrationMapping', [ + { $match: { integrationId: { $in: stringIds } } }, + { $group: { _id: '$integrationId', count: { $sum: 1 } } }, + ]); + + for (const row of rows) { + counts.set(String(row?._id), row?.count ?? 0); + } + return counts; + } + async findMappingBy(integrationId, sourceId) { const filter = this._compositeFilter(integrationId, sourceId); const doc = await findOne(this.prisma, 'IntegrationMapping', filter); diff --git a/packages/core/integrations/repositories/integration-mapping-repository-interface.js b/packages/core/integrations/repositories/integration-mapping-repository-interface.js index 3223249a0..4583eda3f 100644 --- a/packages/core/integrations/repositories/integration-mapping-repository-interface.js +++ b/packages/core/integrations/repositories/integration-mapping-repository-interface.js @@ -77,6 +77,20 @@ class IntegrationMappingRepositoryInterface { ); } + /** + * Count mappings grouped by integration id, for a bounded set of ids. + * Adapters must drain the full grouped result (a deployment can have more + * than one first-batch of distinct integration ids). + * + * @returns {Promise>} Map of integrationId (string) → count + * @abstract + */ + async countByIntegrationIds(ids) { + throw new Error( + 'Method countByIntegrationIds must be implemented by subclass' + ); + } + /** * Find mapping by ID * diff --git a/packages/core/integrations/repositories/integration-mapping-repository-mongo.js b/packages/core/integrations/repositories/integration-mapping-repository-mongo.js index 85220bedd..d24bdf8f1 100644 --- a/packages/core/integrations/repositories/integration-mapping-repository-mongo.js +++ b/packages/core/integrations/repositories/integration-mapping-repository-mongo.js @@ -133,6 +133,28 @@ class IntegrationMappingRepositoryMongo extends IntegrationMappingRepositoryInte }; } + /** + * Count mappings grouped by integration id for a bounded id set. + * + * @param {Array} ids - Integration ids + * @returns {Promise>} integrationId (string) → count + */ + async countByIntegrationIds(ids = []) { + const counts = new Map(); + if (!ids || ids.length === 0) return counts; + + const groups = await this.prisma.integrationMapping.groupBy({ + by: ['integrationId'], + where: { integrationId: { in: ids } }, + _count: { _all: true }, + }); + + for (const group of groups) { + counts.set(String(group.integrationId), group._count._all); + } + return counts; + } + /** * Find mapping by ID * @param {string} id - Mapping ID diff --git a/packages/core/integrations/repositories/integration-mapping-repository-postgres.js b/packages/core/integrations/repositories/integration-mapping-repository-postgres.js index 6562e6071..1e21e7245 100644 --- a/packages/core/integrations/repositories/integration-mapping-repository-postgres.js +++ b/packages/core/integrations/repositories/integration-mapping-repository-postgres.js @@ -2,6 +2,7 @@ const { prisma } = require('../../database/prisma'); const { IntegrationMappingRepositoryInterface, } = require('./integration-mapping-repository-interface'); +const { strictIntId } = require('./report-id'); /** * PostgreSQL Integration Mapping Repository Adapter @@ -188,6 +189,33 @@ class IntegrationMappingRepositoryPostgres extends IntegrationMappingRepositoryI }; } + /** + * Count mappings grouped by integration id for a bounded id set. + * + * @param {Array} ids - Integration ids + * @returns {Promise>} integrationId (string) → count + */ + async countByIntegrationIds(ids = []) { + const counts = new Map(); + if (!ids || ids.length === 0) return counts; + + // Strict (matches findAllForReport): reject partially-numeric ids instead of coercing. + const intIds = ids.map((id) => strictIntId(id)); + const groups = await this.prisma.integrationMapping.groupBy({ + by: ['integrationId'], + where: { integrationId: { in: intIds } }, + _count: { _all: true }, + }); + + for (const group of groups) { + counts.set( + this._intToString(group.integrationId), + group._count._all + ); + } + return counts; + } + /** * Find mapping by ID * @param {string} id - Mapping ID (string from application layer) diff --git a/packages/core/integrations/repositories/integration-mapping-repository-report.test.js b/packages/core/integrations/repositories/integration-mapping-repository-report.test.js new file mode 100644 index 000000000..eba07f2b0 --- /dev/null +++ b/packages/core/integrations/repositories/integration-mapping-repository-report.test.js @@ -0,0 +1,130 @@ +/** + * countByIntegrationIds tests for the integration-mapping repository adapters. + * + * ADR-010 folds the reporting subsystem's mapped-record count into the mapping + * repository so reports read it through the command layer. These cases preserve + * what the retired reporting-repository-*.test.js verified: grouped counts keyed + * by integration id, the empty-ids short-circuit, strict integer-id rejection + * (Postgres), string-id matching on DocumentDB, and the DocumentDB aggregate + * cursor drain. + */ + +const { + IntegrationMappingRepositoryPostgres, +} = require('./integration-mapping-repository-postgres'); +const { + IntegrationMappingRepositoryMongo, +} = require('./integration-mapping-repository-mongo'); +const { + IntegrationMappingRepositoryDocumentDB, +} = require('./integration-mapping-repository-documentdb'); + +describe('IntegrationMappingRepositoryPostgres.countByIntegrationIds', () => { + let repo; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { integrationMapping: { groupBy: jest.fn() } }; + repo = new IntegrationMappingRepositoryPostgres(); + repo.prisma = mockPrisma; + }); + + it('groups counts by integration id (ids converted to int, keys to string)', async () => { + mockPrisma.integrationMapping.groupBy.mockResolvedValue([ + { integrationId: 7, _count: { _all: 3 } }, + { integrationId: 9, _count: { _all: 1 } }, + ]); + + const counts = await repo.countByIntegrationIds(['7', '9']); + + expect(mockPrisma.integrationMapping.groupBy).toHaveBeenCalledWith({ + by: ['integrationId'], + where: { integrationId: { in: [7, 9] } }, + _count: { _all: true }, + }); + expect(counts.get('7')).toBe(3); + expect(counts.get('9')).toBe(1); + }); + + it('short-circuits to an empty map for no ids', async () => { + const counts = await repo.countByIntegrationIds([]); + expect(counts.size).toBe(0); + expect(mockPrisma.integrationMapping.groupBy).not.toHaveBeenCalled(); + }); + + it('rejects a non-integer id instead of silently coercing it', async () => { + await expect( + repo.countByIntegrationIds(['12abc']) + ).rejects.toThrow(/cannot be converted to integer/); + expect(mockPrisma.integrationMapping.groupBy).not.toHaveBeenCalled(); + }); +}); + +describe('IntegrationMappingRepositoryMongo.countByIntegrationIds', () => { + let repo; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { integrationMapping: { groupBy: jest.fn() } }; + repo = new IntegrationMappingRepositoryMongo(); + repo.prisma = mockPrisma; + }); + + it('matches string ids and keys the map by string integration id', async () => { + mockPrisma.integrationMapping.groupBy.mockResolvedValue([ + { integrationId: '507f1f77bcf86cd799439011', _count: { _all: 5 } }, + ]); + + const counts = await repo.countByIntegrationIds([ + '507f1f77bcf86cd799439011', + ]); + + expect(mockPrisma.integrationMapping.groupBy).toHaveBeenCalledWith({ + by: ['integrationId'], + where: { integrationId: { in: ['507f1f77bcf86cd799439011'] } }, + _count: { _all: true }, + }); + expect(counts.get('507f1f77bcf86cd799439011')).toBe(5); + }); +}); + +describe('IntegrationMappingRepositoryDocumentDB.countByIntegrationIds', () => { + let repo; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { $runCommandRaw: jest.fn() }; + repo = new IntegrationMappingRepositoryDocumentDB(); + repo.prisma = mockPrisma; + }); + + it('aggregates by string ids and drains the cursor across batches', async () => { + mockPrisma.$runCommandRaw + .mockResolvedValueOnce({ + cursor: { id: 5, firstBatch: [{ _id: '7', count: 3 }] }, + }) + .mockResolvedValueOnce({ + cursor: { id: 0, nextBatch: [{ _id: '9', count: 2 }] }, + }); + + const counts = await repo.countByIntegrationIds([7, 9]); + + const aggregateCall = mockPrisma.$runCommandRaw.mock.calls[0][0]; + expect(aggregateCall.aggregate).toBe('IntegrationMapping'); + expect(aggregateCall.pipeline[0]).toEqual({ + $match: { integrationId: { $in: ['7', '9'] } }, + }); + expect(mockPrisma.$runCommandRaw.mock.calls[1][0]).toMatchObject({ + getMore: 5, + collection: 'IntegrationMapping', + }); + expect(counts.get('7')).toBe(3); + expect(counts.get('9')).toBe(2); + }); + + it('short-circuits to an empty map for no ids', async () => { + const counts = await repo.countByIntegrationIds([]); + expect(counts.size).toBe(0); + expect(mockPrisma.$runCommandRaw).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/core/integrations/repositories/integration-repository-documentdb.js b/packages/core/integrations/repositories/integration-repository-documentdb.js index 199188800..705ba2f19 100644 --- a/packages/core/integrations/repositories/integration-repository-documentdb.js +++ b/packages/core/integrations/repositories/integration-repository-documentdb.js @@ -4,6 +4,7 @@ const { toObjectIdArray, fromObjectId, findMany, + findManyDrained, findOne, insertOne, updateOne, @@ -254,6 +255,43 @@ class IntegrationRepositoryDocumentDB extends IntegrationRepositoryInterface { return this._mapIntegration(updated); } + // Drain the full cursor so a deployment-wide report is never truncated. + async findAllForReport({ status, userId } = {}) { + const filter = {}; + if (status) filter.status = status; + if (userId !== undefined && userId !== null) { + const objectId = toObjectId(userId); + // Invalid userId means no matches — don't fall through to an unfiltered whole-deployment query. + if (!objectId) return []; + filter.userId = objectId; + } + + const docs = await findManyDrained(this.prisma, 'Integration', filter); + + return docs.map((doc) => { + const errors = this._extractReportErrors(doc); + return { + id: fromObjectId(doc?._id), + type: doc?.config?.type ?? null, + status: doc?.status ?? null, + userId: fromObjectId(doc?.userId) ?? null, + version: doc?.version ?? null, + errorCount: Array.isArray(errors) ? errors.length : 0, + moduleCount: Array.isArray(doc?.entityIds) + ? doc.entityIds.length + : 0, + createdAt: doc?.createdAt ?? null, + updatedAt: doc?.updatedAt ?? null, + }; + }); + } + + _extractReportErrors(doc) { + if (Array.isArray(doc?.errors)) return doc.errors; + if (Array.isArray(doc?.messages?.errors)) return doc.messages.errors; + return []; + } + _mapIntegration(doc) { const messages = this._extractMessages(doc); return { diff --git a/packages/core/integrations/repositories/integration-repository-interface.js b/packages/core/integrations/repositories/integration-repository-interface.js index a8e6a3717..fb74db55d 100644 --- a/packages/core/integrations/repositories/integration-repository-interface.js +++ b/packages/core/integrations/repositories/integration-repository-interface.js @@ -35,6 +35,22 @@ class IntegrationRepositoryInterface { throw new Error('Method findIntegrations must be implemented by subclass'); } + /** + * Find every integration in a report-shaped projection, optionally + * filtered by status and/or owning user. Adapters must drain the full + * result set (no first-batch truncation) since this powers a + * deployment-wide scan. + * + * @param {Object} [filter={}] + * @param {string} [filter.status] - Integration status + * @param {string|number} [filter.userId] - Owning user ID + * @returns {Promise>} + * @abstract + */ + async findAllForReport(filter = {}) { + throw new Error('Method findAllForReport must be implemented by subclass'); + } + /** * Delete integration by ID * diff --git a/packages/core/integrations/repositories/integration-repository-mongo.js b/packages/core/integrations/repositories/integration-repository-mongo.js index e65d6c1b4..ddcb12ff2 100644 --- a/packages/core/integrations/repositories/integration-repository-mongo.js +++ b/packages/core/integrations/repositories/integration-repository-mongo.js @@ -89,6 +89,42 @@ class IntegrationRepositoryMongo extends IntegrationRepositoryInterface { })); } + /** + * Find every integration in a report-shaped projection. + * + * type lives in config.type (a JSON path not portably groupable across + * DBs); it is left in the row for the caller to bucket. + * + * @param {Object} [filter={}] + * @param {string} [filter.status] - Integration status + * @param {string} [filter.userId] - Owning user ID (ObjectId as string) + * @returns {Promise} Report-shaped integration rows + */ + async findAllForReport({ status, userId } = {}) { + const where = {}; + if (status) where.status = status; + if (userId !== undefined && userId !== null) where.userId = userId; + + const integrations = await this.prisma.integration.findMany({ + where, + include: { entities: { select: { id: true } } }, + }); + + return integrations.map((integration) => ({ + id: integration.id, + type: integration.config?.type ?? null, + status: integration.status ?? null, + userId: integration.userId ?? null, + version: integration.version ?? null, + errorCount: Array.isArray(integration.errors) + ? integration.errors.length + : 0, + moduleCount: integration.entities?.length ?? 0, + createdAt: integration.createdAt ?? null, + updatedAt: integration.updatedAt ?? null, + })); + } + /** * Delete integration by ID * Replaces: IntegrationModel.deleteOne({ _id: integrationId }) diff --git a/packages/core/integrations/repositories/integration-repository-postgres.js b/packages/core/integrations/repositories/integration-repository-postgres.js index 3b29b7ba7..3296bff0f 100644 --- a/packages/core/integrations/repositories/integration-repository-postgres.js +++ b/packages/core/integrations/repositories/integration-repository-postgres.js @@ -3,6 +3,7 @@ const { IntegrationRepositoryInterface, } = require('./integration-repository-interface'); const { validateConfigPatch } = require('./config-patch-shared'); +const { strictIntId } = require('./report-id'); /** * PostgreSQL Integration Repository Adapter @@ -129,6 +130,45 @@ class IntegrationRepositoryPostgres extends IntegrationRepositoryInterface { }); } + /** + * Find every integration in a report-shaped projection. + * + * type lives in config.type (a JSON path not portably groupable across + * DBs); it is left in the row for the caller to bucket. + * + * @param {Object} [filter={}] + * @param {string} [filter.status] - Integration status + * @param {string|number} [filter.userId] - Owning user ID + * @returns {Promise} Report-shaped integration rows + */ + async findAllForReport({ status, userId } = {}) { + const where = {}; + if (status) where.status = status; + if (userId !== undefined && userId !== null) { + // Strict: parseInt would coerce '12abc'/'12.9' to 12 and read the wrong user. + where.userId = strictIntId(userId); + } + + const integrations = await this.prisma.integration.findMany({ + where, + include: { entities: { select: { id: true } } }, + }); + + return integrations.map((integration) => ({ + id: integration.id?.toString(), + type: integration.config?.type ?? null, + status: integration.status ?? null, + userId: integration.userId?.toString() ?? null, + version: integration.version ?? null, + errorCount: Array.isArray(integration.errors) + ? integration.errors.length + : 0, + moduleCount: integration.entities?.length ?? 0, + createdAt: integration.createdAt ?? null, + updatedAt: integration.updatedAt ?? null, + })); + } + /** * Delete integration by ID * diff --git a/packages/core/integrations/repositories/integration-repository-report.test.js b/packages/core/integrations/repositories/integration-repository-report.test.js new file mode 100644 index 000000000..3d50d6e2d --- /dev/null +++ b/packages/core/integrations/repositories/integration-repository-report.test.js @@ -0,0 +1,247 @@ +/** + * findAllForReport tests for the integration repository adapters. + * + * This is the report-shaped read that the reporting subsystem used to own in + * its own repository triad (packages/core/reporting/repositories). ADR-010 + * folds it into the integration repository so reports read cross-integration + * data through the same command/repository path as everything else. These + * cases preserve the behavior the retired reporting-repository-*.test.js files + * verified: where-clause building, errorCount/moduleCount derivation, strict + * id handling (Postgres), and the DocumentDB cursor drain + invalid-userId + * short-circuit. + */ + +const { + IntegrationRepositoryPostgres, +} = require('./integration-repository-postgres'); +const { + IntegrationRepositoryMongo, +} = require('./integration-repository-mongo'); +const { + IntegrationRepositoryDocumentDB, +} = require('./integration-repository-documentdb'); + +describe('IntegrationRepositoryPostgres.findAllForReport', () => { + let repo; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { integration: { findMany: jest.fn() } }; + repo = new IntegrationRepositoryPostgres(); + repo.prisma = mockPrisma; + }); + + it('filters by status and userId and maps the report projection', async () => { + mockPrisma.integration.findMany.mockResolvedValue([ + { + id: 7, + config: { type: 'attio' }, + status: 'ENABLED', + userId: 3, + version: '1.2.0', + errors: [{ message: 'a' }, { message: 'b' }], + entities: [{ id: 1 }, { id: 2 }, { id: 3 }], + createdAt: new Date('2026-01-01T00:00:00Z'), + updatedAt: new Date('2026-02-01T00:00:00Z'), + }, + ]); + + const rows = await repo.findAllForReport({ + status: 'ENABLED', + userId: '3', + }); + + expect(mockPrisma.integration.findMany).toHaveBeenCalledWith({ + where: { status: 'ENABLED', userId: 3 }, + include: { entities: { select: { id: true } } }, + }); + expect(rows).toEqual([ + { + id: '7', + type: 'attio', + status: 'ENABLED', + userId: '3', + version: '1.2.0', + errorCount: 2, + moduleCount: 3, + createdAt: new Date('2026-01-01T00:00:00Z'), + updatedAt: new Date('2026-02-01T00:00:00Z'), + }, + ]); + }); + + it('defaults type/errorCount/moduleCount when absent', async () => { + mockPrisma.integration.findMany.mockResolvedValue([ + { id: 9, config: {}, status: null, userId: null }, + ]); + + const [row] = await repo.findAllForReport(); + + expect(mockPrisma.integration.findMany).toHaveBeenCalledWith({ + where: {}, + include: { entities: { select: { id: true } } }, + }); + expect(row).toMatchObject({ + id: '9', + type: null, + errorCount: 0, + moduleCount: 0, + userId: null, + }); + }); + + it('rejects a non-integer userId instead of silently coercing it', async () => { + await expect( + repo.findAllForReport({ userId: '12abc' }) + ).rejects.toThrow(/cannot be converted to integer/); + expect(mockPrisma.integration.findMany).not.toHaveBeenCalled(); + }); +}); + +describe('IntegrationRepositoryMongo.findAllForReport', () => { + let repo; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { integration: { findMany: jest.fn() } }; + repo = new IntegrationRepositoryMongo(); + repo.prisma = mockPrisma; + }); + + it('uses the string userId directly and maps the report projection', async () => { + mockPrisma.integration.findMany.mockResolvedValue([ + { + id: '507f1f77bcf86cd799439011', + config: { type: 'hubspot' }, + status: 'ERROR', + userId: '507f191e810c19729de860ea', + version: '2.0.0', + errors: [{ message: 'boom' }], + entities: [{ id: 'e1' }], + createdAt: new Date('2026-03-01T00:00:00Z'), + updatedAt: new Date('2026-03-02T00:00:00Z'), + }, + ]); + + const rows = await repo.findAllForReport({ + userId: '507f191e810c19729de860ea', + }); + + expect(mockPrisma.integration.findMany).toHaveBeenCalledWith({ + where: { userId: '507f191e810c19729de860ea' }, + include: { entities: { select: { id: true } } }, + }); + expect(rows[0]).toEqual({ + id: '507f1f77bcf86cd799439011', + type: 'hubspot', + status: 'ERROR', + userId: '507f191e810c19729de860ea', + version: '2.0.0', + errorCount: 1, + moduleCount: 1, + createdAt: new Date('2026-03-01T00:00:00Z'), + updatedAt: new Date('2026-03-02T00:00:00Z'), + }); + }); +}); + +describe('IntegrationRepositoryDocumentDB.findAllForReport', () => { + let repo; + let mockPrisma; + + beforeEach(() => { + mockPrisma = { $runCommandRaw: jest.fn() }; + repo = new IntegrationRepositoryDocumentDB(); + repo.prisma = mockPrisma; + }); + + it('drains the cursor across batches (no first-batch truncation)', async () => { + const doc = (n) => ({ + _id: { $oid: `507f1f77bcf86cd7994390${n}` }, + config: { type: 'attio' }, + status: 'ENABLED', + userId: null, + entityIds: [{ $oid: 'e1' }], + errors: [], + }); + + mockPrisma.$runCommandRaw + .mockResolvedValueOnce({ + cursor: { id: 42, firstBatch: [doc('11'), doc('12')] }, + }) + .mockResolvedValueOnce({ + cursor: { id: 0, nextBatch: [doc('13')] }, + }); + + const rows = await repo.findAllForReport({ status: 'ENABLED' }); + + expect(rows).toHaveLength(3); + const firstCall = mockPrisma.$runCommandRaw.mock.calls[0][0]; + expect(firstCall).toMatchObject({ + find: 'Integration', + filter: { status: 'ENABLED' }, + }); + expect(mockPrisma.$runCommandRaw.mock.calls[1][0]).toMatchObject({ + getMore: 42, + collection: 'Integration', + }); + expect(rows[0]).toMatchObject({ + type: 'attio', + moduleCount: 1, + errorCount: 0, + }); + }); + + it('treats an extended-JSON {$numberLong} cursor id as open and drains it', async () => { + const doc = (n) => ({ + _id: { $oid: `507f1f77bcf86cd7994390${n}` }, + config: { type: 'attio' }, + entityIds: [], + errors: [], + }); + + mockPrisma.$runCommandRaw + .mockResolvedValueOnce({ + cursor: { id: { $numberLong: '9007199254740993' }, firstBatch: [doc('11')] }, + }) + .mockResolvedValueOnce({ + cursor: { id: { $numberLong: '0' }, nextBatch: [doc('12')] }, + }); + + const rows = await repo.findAllForReport(); + + expect(rows).toHaveLength(2); + expect(mockPrisma.$runCommandRaw.mock.calls[1][0]).toMatchObject({ + getMore: { $numberLong: '9007199254740993' }, + collection: 'Integration', + }); + }); + + it('returns [] for an invalid userId without querying', async () => { + const rows = await repo.findAllForReport({ userId: 'not-an-object-id' }); + + expect(rows).toEqual([]); + expect(mockPrisma.$runCommandRaw).not.toHaveBeenCalled(); + }); + + it('derives errorCount from messages.errors when top-level errors is absent', async () => { + mockPrisma.$runCommandRaw.mockResolvedValueOnce({ + cursor: { + id: 0, + firstBatch: [ + { + _id: { $oid: '507f1f77bcf86cd799439011' }, + config: { type: 'x' }, + messages: { errors: [{ m: 1 }, { m: 2 }] }, + entityIds: [], + }, + ], + }, + }); + + const [row] = await repo.findAllForReport(); + + expect(row.errorCount).toBe(2); + expect(row.moduleCount).toBe(0); + }); +}); diff --git a/packages/core/integrations/repositories/report-id.js b/packages/core/integrations/repositories/report-id.js new file mode 100644 index 000000000..47cf27eb1 --- /dev/null +++ b/packages/core/integrations/repositories/report-id.js @@ -0,0 +1,13 @@ +/** + * Coerces an id to an integer, rejecting partially-numeric input ('12abc', + * '12.9') that parseInt would silently truncate to 12 and read the wrong record. + */ +function strictIntId(id) { + const str = String(id).trim(); + if (!/^-?\d+$/.test(str)) { + throw new TypeError(`Invalid ID: ${id} cannot be converted to integer`); + } + return Number.parseInt(str, 10); +} + +module.exports = { strictIntId }; diff --git a/packages/core/package.json b/packages/core/package.json index c96b15efa..a53d951e5 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -6,8 +6,10 @@ "@aws-sdk/client-apigatewaymanagementapi": "^3.588.0", "@aws-sdk/client-kms": "^3.588.0", "@aws-sdk/client-lambda": "^3.714.0", + "@aws-sdk/client-s3": "^3.588.0", "@aws-sdk/client-sqs": "^3.588.0", "@aws-sdk/client-ssm": "^3.588.0", + "@aws-sdk/s3-request-presigner": "^3.588.0", "@hapi/boom": "^10.0.1", "@opentelemetry/api": "^1.9.1", "@opentelemetry/context-async-hooks": "^2.9.0", diff --git a/packages/core/reporting/README.md b/packages/core/reporting/README.md index 90e3c3149..16f2c8792 100644 --- a/packages/core/reporting/README.md +++ b/packages/core/reporting/README.md @@ -1,27 +1,94 @@ # Reporting -Read-only, deployment-wide reporting endpoints for `@friggframework/core`. -Gated by a dedicated reporting API key — **not** the per-user auth used by the -rest of the Management API. +Deployment-wide reports for `@friggframework/core`, modelled as **admin +operations** (ADR-010) — a sibling of the Admin Script Runner (ADR-005) on the +same primitives: one registry, the shared admin API key, sync/async execution, +the isolated admin execution store, and EventBridge scheduling. + +A report is a definition (`ReportBase`) whose output is its payload. **Core +ships built-in reports; adopters register their own** — both are discovered, +listed, and executed through the same runner. This package provides `ReportBase`, +the built-in reports, and `createReportCommands()`. The HTTP router, runner, and +executor live in `@friggframework/admin-scripts`; the infrastructure (queue, +executor Lambda, artifact bucket, scheduler) is generated by the devtools +`AdminScriptBuilder`. + +## Registering reports + +```js +// app definition — same shape as adminScripts +const Definition = { + name: 'my-app', + integrations: [HubSpotIntegration, SalesforceIntegration], + reports: [ConnectedAccountsActivity], // adopter-defined + admin: { includeBuiltinReports: true }, // + core built-ins (integrations, ...) +}; +``` + +A report extends `ReportBase` and reads only through the injected admin command +bundle (`frigg` / `this.context.commands`), never a repository directly: + +```js +const { ReportBase } = require('@friggframework/core'); + +class ConnectedAccountsActivity extends ReportBase { + static Definition = { + name: 'connected-accounts-activity', + version: '1.0.0', + runModes: ['snapshot', 'recorded', 'live'], // first is the default + inputSchema: { type: 'object', properties: { + windowDays: { type: 'integer', enum: [30, 60, 90], default: 30 } } }, + output: { format: 'json' }, // 'csv'|'pdf'|'zip' → artifact storage + schedule: { enabled: true, cron: 'cron(0 6 * * ? *)', mode: 'snapshot' }, // default mode only; activate via PUT + }; + + async execute(frigg, params) { + const since = daysAgo(params.windowDays ?? 30); + const byType = await frigg.credentials.countActiveByType({ since }); + return { windowDays: params.windowDays ?? 30, byType }; + } +} +``` ## Auth -Send the reporting key in the `x-frigg-reporting-api-key` header. It is validated -against the `REPORTING_API_KEY` environment variable. Missing or wrong key → `401`. +All report endpoints use the **shared admin API key** (ADR-005): the +`x-frigg-admin-api-key` header, validated against `ADMIN_API_KEY`. The dedicated +reporting key (`x-frigg-reporting-api-key` / `REPORTING_API_KEY`) is **retired**. + +## Run modes + +Each invocation picks a mode; persistence follows from it. All three run through +the one runner and the isolated admin execution store (`AdminScriptExecution`, +`type: 'REPORT'` — no user/integration FK, so a user-scoped query can never +return a report record). + +| Mode | Persists | Use it for | +| --- | --- | --- | +| `live` | nothing — computes and returns inline | cheap, always-fresh queries | +| `recorded` | an execution record (input + results + logs) | audit trail, async polling, expensive scans | +| `snapshot` | a recorded run tagged into a named series | trends over time | + +Non-JSON `output.format` (`csv`/`pdf`/`zip`) is written to artifact storage (S3, +private + signed URL) instead of inline; JSON stays inline. ## Endpoints | Method | Path | Description | | --- | --- | --- | -| `GET` | `/api/v2/reports` | Index of available reports. | -| `GET` | `/api/v2/reports/integrations` | Integrations report (see below). | +| `GET` | `/api/v2/reports` | List registered report definitions. | +| `GET` | `/api/v2/reports/:name` | Report definition detail (run modes, input schema, schedule). | +| `POST` | `/api/v2/reports/:name/run` | Run `{ mode, params }`. `live` → `200` inline; `recorded`/`snapshot` → `202 { executionId }` (queued). | +| `GET` | `/api/v2/reports/:name/snapshots?from=&to=` | A report's snapshot series (trend); each point carries a signed `artifactUrl` when the run produced a non-JSON artifact. | +| `GET` | `/api/v2/reports/executions/:id` | Fetch one report execution (guarded to `type: 'REPORT'`); a stored artifact is returned as a signed `results.artifactUrl`. | +| `GET`/`PUT`/`DELETE` | `/api/v2/reports/:name/schedule` | Manage a report's recurring schedule. | +| `GET` | `/api/v2/reports/integrations` | Deprecated back-compat for PR #607 — runs the built-in `integrations` report in `live` mode and returns its payload directly. Prefer `POST /:name/run`. | -### `GET /api/v2/reports/integrations` +## Built-in: `integrations` -Optional query params (all strings): `status` (an `IntegrationStatus`), `type` -(`config.type` slug), `userId`. - -## Response shape +Integrations by status and type, with per-type usage columns. Input query params +(all strings): `status` (an `IntegrationStatus`), `type` (`config.type` slug), +`userId`. Response shape (`schemaVersion: 1`): ```jsonc { @@ -37,7 +104,8 @@ Optional query params (all strings): `status` (an `IntegrationStatus`), `type` "type": "hubspot", "label": "HubSpot CRM", "total": 5, - "byStatus": { "ENABLED": 4, "ERROR": 1, "NEEDS_CONFIG": 0, "PROCESSING": 0, "DISABLED": 0 } + "byStatus": { "ENABLED": 4, "ERROR": 1, "NEEDS_CONFIG": 0, "PROCESSING": 0, "DISABLED": 0 }, + "usage": { "records.synced": 4200, "webhooks.received": 118 } } ], "typeLabels": { "hubspot": "HubSpot CRM" }, @@ -46,7 +114,7 @@ Optional query params (all strings): `status` (an `IntegrationStatus`), `type` } ``` -## Field reference +### Field reference - `schemaVersion` — contract version; branch on it. Additive changes do **not** bump it. - `filters` — echoes the applied filters (nulls when omitted). @@ -54,40 +122,33 @@ Optional query params (all strings): `status` (an `IntegrationStatus`), `type` - `metrics.byStatus` — counts keyed by `IntegrationStatus` value. - `metrics.byType[]` — per `type` breakdown: `{ type, label, total, byStatus, usage? }`. - `type` — the `config.type` slug. Integrations with no type bucket as `"unknown"`. - - `usage` — **additive** (ADR-011): present only when the deployment has the - usage store configured. A map of canonical counter → total for that type, - e.g. `{ "records.synced": 4200, "webhooks.received": 118, "api.requests": 9004 }` - (`0` when a type has no data). Read only from the Frigg usage store, never an - external APM; a usage-store read failure never fails the structural report. - Populated by the counters an integration opts into via `Definition.usage` — - see [`../telemetry/README.md`](../telemetry/README.md). - - `label` — human-readable name from the integration class's - `Definition.display.label`. Falls back to the `type` slug when no registered - class supplies a label (e.g. an integration that was removed, or run on an - older app that doesn't register it). Additive — `type` is unchanged. -- `metrics.typeLabels` — map of `type` slug → human-readable label, so callers can - label `integrations[].type` rows without bloating each row. Contains only types - whose registered class supplies a non-default `display.label` (classes still - carrying the IntegrationBase default `'Integration Name'` are excluded). -- `metrics.integrations[]` — lightweight per-integration rows (`id`, `type`, - `status`, `userId`, `version`, `moduleCount`, `errorCount`, `mappedRecordCount`, - `createdAt`, `updatedAt`). These rows carry `type` only — resolve display names - via `metrics.typeLabels`. - -## How labels are sourced - -`createReportingRouter()` loads the app's registered integration classes via -`loadAppDefinition()` and builds a `{ slug → label }` map from each class's -`Definition.display.label`. Loading is wrapped in try/catch, so reporting still -works (labels fall back to slugs) if the app definition can't be loaded. The -`ListIntegrationsReport` use case stays storage-agnostic — it just reads the -injected `typeLabels` map, so all database adapters (PostgreSQL, MongoDB, -DocumentDB) get labels with zero adapter changes. + - `usage` — **additive** (ADR-011): present only when the deployment has the usage + store configured. A map of canonical counter → total for that type (`0` when a + type has no data). Read only from the Frigg usage store; a usage-store read + failure never fails the structural report. Populated by the counters an + integration opts into via `Definition.usage` — see [`../telemetry/README.md`](../telemetry/README.md). + - `label` — human-readable name from the integration class's `Definition.display.label`, + falling back to the `type` slug when no registered class supplies one. +- `metrics.typeLabels` — map of `type` slug → label, so callers label + `integrations[].type` rows without bloating each row. +- `metrics.integrations[]` — lightweight per-integration rows (`id`, `type`, `status`, + `userId`, `version`, `moduleCount`, `errorCount`, `mappedRecordCount`, `createdAt`, + `updatedAt`). Resolve display names via `metrics.typeLabels`. + +The built-in report reads its data through the admin command bundle +(`frigg.integrations.listForReport`, `frigg.integrationMappings.countByIntegrationIds`, +`frigg.usage.getTotalsByDimension`) — the retired reporting repository triad's +logic now lives in the canonical integration/mapping repositories, so all +database adapters (PostgreSQL, MongoDB, DocumentDB) are served through one path. ## Caveats -- The response is sensitive (exposes integration types, counts, user IDs, versions). - Treat the reporting key as an admin secret. -- Read-only — no mutation endpoints. -- Labels only appear once the deployment runs a `@friggframework/core` version that - includes this field and the app registers the integration classes. +- Report output is sensitive (integration types, counts, user IDs, versions). + Treat the admin API key as an admin secret. +- The `integrations` back-compat alias is transitional; migrate to + `POST /api/v2/reports/integrations/run { "mode": "live" }`. +- Labels only appear once the app registers the integration classes. +- A `schedule` block in a report's Definition only supplies the default scheduled + **mode**. It does not create a recurring trigger on its own — activate the + schedule with `PUT /api/v2/reports/:name/schedule`; the DB schedule + (`ScriptSchedule`) is the single source of truth. diff --git a/packages/core/reporting/builtin-reports.js b/packages/core/reporting/builtin-reports.js new file mode 100644 index 000000000..bd621aa73 --- /dev/null +++ b/packages/core/reporting/builtin-reports.js @@ -0,0 +1,6 @@ +const { IntegrationsReport } = require('./reports/integrations-report'); + +// Registered by the admin-scripts bootstrap when the app definition sets `admin.includeBuiltinReports`. +const BUILTIN_REPORTS = [IntegrationsReport]; + +module.exports = { BUILTIN_REPORTS }; diff --git a/packages/core/reporting/index.js b/packages/core/reporting/index.js index 333ffa2e5..5c1a6241f 100644 --- a/packages/core/reporting/index.js +++ b/packages/core/reporting/index.js @@ -1,17 +1,13 @@ -const { createReportingRouter } = require('./reporting-router'); +const { ReportBase } = require('./report-base'); +const { IntegrationsReport } = require('./reports/integrations-report'); +const { BUILTIN_REPORTS } = require('./builtin-reports'); const { - createReportingRepository, - ReportingRepositoryMongo, - ReportingRepositoryPostgres, - ReportingRepositoryDocumentDB, -} = require('./repositories/reporting-repository-factory'); -const { ListIntegrationsReport } = require('./use-cases'); + createReportCommands, +} = require('../application/commands/report-commands'); module.exports = { - createReportingRouter, - createReportingRepository, - ReportingRepositoryMongo, - ReportingRepositoryPostgres, - ReportingRepositoryDocumentDB, - ListIntegrationsReport, + ReportBase, + IntegrationsReport, + BUILTIN_REPORTS, + createReportCommands, }; diff --git a/packages/core/reporting/report-base.js b/packages/core/reporting/report-base.js new file mode 100644 index 000000000..867309306 --- /dev/null +++ b/packages/core/reporting/report-base.js @@ -0,0 +1,49 @@ +/** + * ReportBase — a report is an admin operation whose output is its payload. + * + * A report Definition declares: + * - runModes: allowed modes; the first is the default. + * 'live' — compute and return inline; persist nothing. + * 'recorded' — persist an execution record (input + results + logs). + * 'snapshot' — a recorded run tagged as a point in a named series. + * - output.format: 'json' returns inline; 'csv'|'pdf'|'zip' go to artifact storage. + * - schedule: optional recurring run via EventBridge. + * + * execute(frigg, params): `frigg` is the admin command bundle + * (=== this.context.commands); do data reads through it, never a repository. + */ +class ReportBase { + static Definition = { + name: 'Report Name', + version: '0.0.0', + description: 'What this report computes', + source: 'USER_DEFINED', // 'BUILTIN' | 'USER_DEFINED' + + runModes: ['live', 'recorded', 'snapshot'], + inputSchema: null, + outputSchema: null, + output: { format: 'json' }, + schedule: { enabled: false, cron: null, mode: 'snapshot' }, + + config: { + timeout: 300000, + }, + + display: { + category: 'reporting', + icon: null, + }, + }; + + constructor(params = {}) { + this.context = params.context || null; + this.executionId = params.executionId || null; + this.integrationFactory = params.integrationFactory || null; + } + + async execute(frigg, params) { + throw new Error('ReportBase.execute() must be implemented by subclass'); + } +} + +module.exports = { ReportBase }; diff --git a/packages/core/reporting/reporting-router.js b/packages/core/reporting/reporting-router.js deleted file mode 100644 index cc1ef74ff..000000000 --- a/packages/core/reporting/reporting-router.js +++ /dev/null @@ -1,84 +0,0 @@ -const express = require('express'); -const catchAsyncError = require('express-async-handler'); -const { - createReportingRepository, -} = require('./repositories/reporting-repository-factory'); -const { - createUsageRepository, -} = require('../usage/repositories/usage-repository-factory'); -const { - ListIntegrationsReport, -} = require('./use-cases/list-integrations-report'); -const { loadAppDefinition } = require('../handlers/app-definition-loader'); - -// IntegrationBase.Definition default — skip it so the slug is used instead. -const PLACEHOLDER_DISPLAY_NAME = 'Integration Name'; - -// Map each integration's config.type slug to its human-readable display label. -// Wrapped so reporting still works if the app definition fails to load. -function buildTypeLabels() { - try { - const { integrations = [] } = loadAppDefinition(); - const labels = {}; - for (const IntegrationClass of integrations) { - const def = IntegrationClass?.Definition; - if (!def?.name) continue; - const label = def.display?.label; - if (label && label !== PLACEHOLDER_DISPLAY_NAME) { - labels[def.name] = label; - } - } - return labels; - } catch (error) { - console.error( - 'Reporting: failed to load integration labels:', - error.message - ); - return {}; - } -} - -function createReportingRouter() { - const reportingRepository = createReportingRepository(); - const usageRepository = createUsageRepository(); - const listIntegrationsReport = new ListIntegrationsReport({ - reportingRepository, - usageRepository, - typeLabels: buildTypeLabels(), - }); - - const router = express.Router(); - router.use(validateApiKey); - - router.get('/api/v2/reports', (_req, res) => { - res.json({ service: 'frigg-core-api', reports: ['integrations'] }); - }); - - router.get( - '/api/v2/reports/integrations', - catchAsyncError(async (req, res) => { - const { status, type, userId } = req.query; - res.json( - await listIntegrationsReport.execute({ status, type, userId }) - ); - }) - ); - - return router; -} - -function validateApiKey(req, res, next) { - const apiKey = req.headers['x-frigg-reporting-api-key']; - - if (!apiKey || apiKey !== process.env.REPORTING_API_KEY) { - console.error('Unauthorized access attempt to reporting endpoint'); - return res.status(401).json({ - status: 'error', - message: 'Unauthorized - x-frigg-reporting-api-key header required', - }); - } - - next(); -} - -module.exports = { createReportingRouter, validateApiKey }; diff --git a/packages/core/reporting/reporting-router.test.js b/packages/core/reporting/reporting-router.test.js deleted file mode 100644 index 1678d9948..000000000 --- a/packages/core/reporting/reporting-router.test.js +++ /dev/null @@ -1,152 +0,0 @@ -process.env.REPORTING_API_KEY = 'test-reporting-key'; - -jest.mock('./repositories/reporting-repository-factory', () => ({ - createReportingRepository: jest.fn(() => ({ - findIntegrationsForReport: jest.fn().mockResolvedValue([ - { - id: '1', - type: 'hubspot', - status: 'ENABLED', - userId: '3', - version: '1', - moduleCount: 1, - errorCount: 0, - createdAt: null, - updatedAt: null, - }, - ]), - countMappingsByIntegrationIds: jest - .fn() - .mockResolvedValue(new Map([['1', 5]])), - })), -})); - -jest.mock('../usage/repositories/usage-repository-factory', () => ({ - createUsageRepository: jest.fn(() => ({ - getTotalsByDimension: jest.fn().mockResolvedValue([]), - })), -})); - -jest.mock('../handlers/app-definition-loader', () => ({ - loadAppDefinition: jest.fn(() => ({ - integrations: [ - { Definition: { name: 'hubspot', display: { label: 'HubSpot CRM' } } }, - { Definition: { name: 'salesforce', display: { label: 'Salesforce' } } }, - ], - })), -})); - -const { createReportingRouter, validateApiKey } = require('./reporting-router'); - -const mockRes = () => { - const res = {}; - res.status = jest.fn(() => res); - res.json = jest.fn(() => res); - return res; -}; - -const findRouteHandler = (router, path) => { - for (const layer of router.stack) { - if (layer.route && layer.route.path === path) { - const stack = layer.route.stack; - return stack[stack.length - 1].handle; - } - } - return null; -}; - -describe('reporting-router', () => { - describe('validateApiKey', () => { - it('returns 401 when the reporting key is missing', () => { - const res = mockRes(); - const next = jest.fn(); - validateApiKey({ headers: {} }, res, next); - expect(res.status).toHaveBeenCalledWith(401); - expect(next).not.toHaveBeenCalled(); - }); - - it('returns 401 when the reporting key is wrong', () => { - const res = mockRes(); - const next = jest.fn(); - validateApiKey( - { headers: { 'x-frigg-reporting-api-key': 'nope' } }, - res, - next - ); - expect(res.status).toHaveBeenCalledWith(401); - expect(next).not.toHaveBeenCalled(); - }); - - it('calls next when the reporting key matches', () => { - const res = mockRes(); - const next = jest.fn(); - validateApiKey( - { headers: { 'x-frigg-reporting-api-key': 'test-reporting-key' } }, - res, - next - ); - expect(next).toHaveBeenCalled(); - expect(res.status).not.toHaveBeenCalled(); - }); - }); - - describe('routes', () => { - const router = createReportingRouter(); - - it('GET /api/v2/reports returns the index', async () => { - const handler = findRouteHandler(router, '/api/v2/reports'); - expect(handler).toBeTruthy(); - const res = mockRes(); - await handler({ query: {} }, res, jest.fn()); - expect(res.json).toHaveBeenCalledWith({ - service: 'frigg-core-api', - reports: ['integrations'], - }); - }); - - it('GET /api/v2/reports/integrations returns the versioned envelope', async () => { - const handler = findRouteHandler( - router, - '/api/v2/reports/integrations' - ); - expect(handler).toBeTruthy(); - const res = mockRes(); - await handler({ query: {} }, res, jest.fn()); - expect(res.json).toHaveBeenCalled(); - const body = res.json.mock.calls[0][0]; - expect(body.schemaVersion).toBe(1); - expect(body.metrics.total).toBe(1); - expect(body.metrics.integrations[0].mappedRecordCount).toBe(5); - }); - - const integrationsHandler = () => - findRouteHandler(router, '/api/v2/reports/integrations'); - - it('forwards query params to the use case and echoes filters', async () => { - const res = mockRes(); - await integrationsHandler()( - { query: { status: 'ENABLED', type: '', userId: '7' } }, - res, - jest.fn() - ); - const body = res.json.mock.calls[0][0]; - expect(body.filters).toEqual({ - status: 'ENABLED', - type: null, - userId: '7', - }); - }); - - it('labels byType entries from the loaded integration definitions', async () => { - const res = mockRes(); - await integrationsHandler()({ query: {} }, res, jest.fn()); - const body = res.json.mock.calls[0][0]; - const hub = body.metrics.byType.find((t) => t.type === 'hubspot'); - expect(hub.label).toBe('HubSpot CRM'); - expect(body.metrics.typeLabels).toEqual({ - hubspot: 'HubSpot CRM', - salesforce: 'Salesforce', - }); - }); - }); -}); diff --git a/packages/core/reporting/use-cases/list-integrations-report.js b/packages/core/reporting/reports/integrations-report.js similarity index 63% rename from packages/core/reporting/use-cases/list-integrations-report.js rename to packages/core/reporting/reports/integrations-report.js index acc935405..2d2636a38 100644 --- a/packages/core/reporting/use-cases/list-integrations-report.js +++ b/packages/core/reporting/reports/integrations-report.js @@ -1,11 +1,11 @@ -const Boom = require('@hapi/boom'); +const { ReportBase } = require('../report-base'); const { CANONICAL_COUNTERS } = require('../../telemetry/canonical-counters'); +const { loadAppDefinition } = require('../../handlers/app-definition-loader'); const SCHEMA_VERSION = 1; const SERVICE = 'frigg-core-api'; -// Seeded so every known status appears (even at 0); unknown values added to -// the schema later are still counted dynamically. +// Seeded so every known status appears in output even at count 0. const KNOWN_STATUSES = [ 'IN_CREATION', 'ENABLED', @@ -16,33 +16,37 @@ const KNOWN_STATUSES = [ 'DISABLED', ]; -class ListIntegrationsReport { - constructor({ - reportingRepository, - usageRepository, - typeLabels = {}, - } = {}) { - if (!reportingRepository) { - throw new Error('reportingRepository is required'); - } - if (!usageRepository) { - throw new Error('usageRepository is required'); - } - this.reportingRepository = reportingRepository; - this.usageRepository = usageRepository; - this.typeLabels = typeLabels; - } +// IntegrationBase.Definition default — skip it so the slug is used instead. +const PLACEHOLDER_DISPLAY_NAME = 'Integration Name'; + +class IntegrationsReport extends ReportBase { + static Definition = { + name: 'integrations', + version: '1.0.0', + description: 'Integrations by status and type, with per-type usage counts', + source: 'BUILTIN', + runModes: ['live', 'recorded', 'snapshot'], + inputSchema: { + type: 'object', + additionalProperties: false, + properties: { + status: { type: 'string' }, + type: { type: 'string' }, + userId: { type: 'string' }, + }, + }, + output: { format: 'json' }, + schedule: { enabled: false, cron: null, mode: 'snapshot' }, + display: { category: 'reporting', icon: null }, + }; - async execute(query = {}) { - const { status, type, userId } = this._validateQuery(query); + async execute(frigg, params = {}) { + const { status, type, userId } = this._validateQuery(params); + const typeLabels = buildTypeLabels(); - const rows = await this.reportingRepository.findIntegrationsForReport({ - status, - userId, - }); + const rows = await frigg.integrations.listForReport({ status, userId }); - // type lives in config.type (a JSON path not portably groupable across - // DBs), so it is filtered here rather than in the repository query. + // type lives in config.type, a JSON path not portably groupable across DBs, so filter here not in the query. const filtered = type === undefined ? rows @@ -50,7 +54,7 @@ class ListIntegrationsReport { const ids = filtered.map((row) => row.id); const mappingCounts = ids.length - ? await this.reportingRepository.countMappingsByIntegrationIds(ids) + ? await frigg.integrationMappings.countByIntegrationIds(ids) : new Map(); const integrations = filtered.map((row) => ({ @@ -76,8 +80,7 @@ class ListIntegrationsReport { if (!byTypeMap.has(integration.type)) { byTypeMap.set(integration.type, { type: integration.type, - label: - this.typeLabels[integration.type] || integration.type, + label: typeLabels[integration.type] || integration.type, total: 0, byStatus: emptyStatusCounts(), }); @@ -87,7 +90,7 @@ class ListIntegrationsReport { bucket.byStatus[statusKey] = (bucket.byStatus[statusKey] ?? 0) + 1; } - await this._attachUsageColumns(byTypeMap); + await this._attachUsageColumns(byTypeMap, frigg); return { schemaVersion: SCHEMA_VERSION, @@ -102,18 +105,18 @@ class ListIntegrationsReport { total: integrations.length, byStatus, byType: Array.from(byTypeMap.values()), - typeLabels: { ...this.typeLabels }, + typeLabels: { ...typeLabels }, integrations, }, }; } - async _attachUsageColumns(byTypeMap) { + async _attachUsageColumns(byTypeMap, frigg) { const metrics = Object.keys(CANONICAL_COUNTERS); try { const totalsByMetric = await Promise.all( metrics.map(async (metric) => { - const totals = await this.usageRepository.getTotalsByDimension({ + const totals = await frigg.usage.getTotalsByDimension({ metric, groupBy: 'integrationType', }); @@ -146,7 +149,7 @@ class ListIntegrationsReport { value !== null && typeof value !== 'string' ) { - throw Boom.badRequest( + throw invalidInput( `Invalid query parameter '${key}': expected a string` ); } @@ -158,7 +161,7 @@ class ListIntegrationsReport { userId: normalize(userId), }; if (normalized.status && !KNOWN_STATUSES.includes(normalized.status)) { - throw Boom.badRequest( + throw invalidInput( `Invalid status '${ normalized.status }'. Expected one of: ${KNOWN_STATUSES.join(', ')}` @@ -168,6 +171,13 @@ class ListIntegrationsReport { } } +// INVALID_INPUT keeps the report protocol-agnostic; the runner/router map it to a 400. +function invalidInput(message) { + const error = new Error(message); + error.code = 'INVALID_INPUT'; + return error; +} + function emptyStatusCounts() { return KNOWN_STATUSES.reduce((acc, status) => { acc[status] = 0; @@ -186,4 +196,26 @@ function toIso(value) { return String(value); } -module.exports = { ListIntegrationsReport, SCHEMA_VERSION }; +function buildTypeLabels() { + try { + const { integrations = [] } = loadAppDefinition(); + const labels = {}; + for (const IntegrationClass of integrations) { + const def = IntegrationClass?.Definition; + if (!def?.name) continue; + const label = def.display?.label; + if (label && label !== PLACEHOLDER_DISPLAY_NAME) { + labels[def.name] = label; + } + } + return labels; + } catch (error) { + console.error( + 'Reporting: failed to load integration labels:', + error.message + ); + return {}; + } +} + +module.exports = { IntegrationsReport, SCHEMA_VERSION }; diff --git a/packages/core/reporting/reports/integrations-report.test.js b/packages/core/reporting/reports/integrations-report.test.js new file mode 100644 index 000000000..50afcfee7 --- /dev/null +++ b/packages/core/reporting/reports/integrations-report.test.js @@ -0,0 +1,138 @@ +const { IntegrationsReport, SCHEMA_VERSION } = require('./integrations-report'); + +// A fake admin command bundle — reports read only through this surface. Mirrors +// what the runner injects as `frigg` (context.commands) at runtime. +function makeFrigg({ rows = [], mappingCounts = new Map(), usage = [] } = {}) { + return { + integrations: { listForReport: jest.fn().mockResolvedValue(rows) }, + integrationMappings: { + countByIntegrationIds: jest.fn().mockResolvedValue(mappingCounts), + }, + usage: { getTotalsByDimension: jest.fn().mockResolvedValue(usage) }, + }; +} + +describe('IntegrationsReport (built-in)', () => { + it('is a BUILTIN report named "integrations" with live as the default mode', () => { + const def = IntegrationsReport.Definition; + expect(def.name).toBe('integrations'); + expect(def.source).toBe('BUILTIN'); + expect(def.runModes[0]).toBe('live'); + expect(def.output.format).toBe('json'); + }); + + it('builds the schemaVersion:1 shape: totals, byStatus, byType, mappedRecordCount', async () => { + const rows = [ + { + id: '1', + type: 'attio', + status: 'ENABLED', + userId: 'u1', + version: '1.0.0', + moduleCount: 2, + errorCount: 0, + createdAt: new Date('2026-01-01T00:00:00Z'), + updatedAt: new Date('2026-01-02T00:00:00Z'), + }, + { + id: '2', + type: 'attio', + status: 'ERROR', + userId: 'u2', + version: '1.0.0', + moduleCount: 1, + errorCount: 3, + createdAt: new Date('2026-01-03T00:00:00Z'), + updatedAt: new Date('2026-01-04T00:00:00Z'), + }, + ]; + const frigg = makeFrigg({ + rows, + mappingCounts: new Map([['1', 10]]), + }); + + const report = new IntegrationsReport(); + const result = await report.execute(frigg, {}); + + expect(result.schemaVersion).toBe(SCHEMA_VERSION); + expect(result.service).toBe('frigg-core-api'); + expect(result.metrics.total).toBe(2); + expect(result.metrics.byStatus.ENABLED).toBe(1); + expect(result.metrics.byStatus.ERROR).toBe(1); + + const attio = result.metrics.byType.find((b) => b.type === 'attio'); + expect(attio.total).toBe(2); + expect(attio.byStatus.ENABLED).toBe(1); + expect(attio.byStatus.ERROR).toBe(1); + + const first = result.metrics.integrations.find((i) => i.id === '1'); + expect(first.mappedRecordCount).toBe(10); + expect(first.createdAt).toBe('2026-01-01T00:00:00.000Z'); + const second = result.metrics.integrations.find((i) => i.id === '2'); + expect(second.mappedRecordCount).toBe(0); + + // countByIntegrationIds is called only with the filtered ids + expect( + frigg.integrationMappings.countByIntegrationIds + ).toHaveBeenCalledWith(['1', '2']); + }); + + it('filters by type in the report and echoes filters', async () => { + const rows = [ + { id: '1', type: 'attio', status: 'ENABLED' }, + { id: '2', type: 'hubspot', status: 'ENABLED' }, + ]; + const frigg = makeFrigg({ rows, mappingCounts: new Map() }); + + const report = new IntegrationsReport(); + const result = await report.execute(frigg, { type: 'attio' }); + + expect(result.metrics.total).toBe(1); + expect(result.metrics.integrations[0].type).toBe('attio'); + expect(result.filters).toEqual({ + status: null, + type: 'attio', + userId: null, + }); + }); + + it('attaches per-type usage columns from the usage command', async () => { + const frigg = makeFrigg({ + rows: [{ id: '1', type: 'attio', status: 'ENABLED' }], + usage: [{ integrationType: 'attio', value: 42 }], + }); + + const report = new IntegrationsReport(); + const result = await report.execute(frigg, {}); + + const attio = result.metrics.byType.find((b) => b.type === 'attio'); + expect(attio.usage).toBeDefined(); + // every canonical counter uses the same mocked totals here + expect(Object.values(attio.usage).every((v) => v === 42)).toBe(true); + }); + + it('still returns the report when the usage store is unavailable', async () => { + const frigg = makeFrigg({ + rows: [{ id: '1', type: 'attio', status: 'ENABLED' }], + }); + frigg.usage.getTotalsByDimension.mockRejectedValue( + new Error('usage store down') + ); + + const report = new IntegrationsReport(); + const result = await report.execute(frigg, {}); + + expect(result.metrics.total).toBe(1); + const attio = result.metrics.byType.find((b) => b.type === 'attio'); + expect(attio.usage).toBeUndefined(); + }); + + it('rejects an unknown status with an INVALID_INPUT error (no HTTP coupling)', async () => { + const report = new IntegrationsReport(); + // The report is protocol-agnostic: it throws a coded error, not a Boom + // HTTP error. The runner/router maps INVALID_INPUT to a 400. + await expect( + report.execute(makeFrigg(), { status: 'NOPE' }) + ).rejects.toMatchObject({ code: 'INVALID_INPUT' }); + }); +}); diff --git a/packages/core/reporting/repositories/reporting-repository-documentdb.js b/packages/core/reporting/repositories/reporting-repository-documentdb.js deleted file mode 100644 index 10f55e839..000000000 --- a/packages/core/reporting/repositories/reporting-repository-documentdb.js +++ /dev/null @@ -1,127 +0,0 @@ -const { prisma } = require('../../database/prisma'); -const { toObjectId, fromObjectId } = require('../../database/documentdb-utils'); -const { - ReportingRepositoryInterface, -} = require('./reporting-repository-interface'); - -const DRAIN_BATCH_SIZE = 1000; -const MAX_BATCHES = 100000; - -// Drains cursors via getMore rather than reusing documentdb-utils.findMany/ -// aggregate, which return only the first batch (~101 docs) and would silently -// truncate a deployment-wide report. -class ReportingRepositoryDocumentDB extends ReportingRepositoryInterface { - constructor() { - super(); - this.prisma = prisma; - } - - async findIntegrationsForReport({ status, userId } = {}) { - const filter = {}; - if (status) filter.status = status; - if (userId !== undefined && userId !== null) { - const objectId = toObjectId(userId); - // An invalid userId means no matches — must not fall through to an - // unfiltered query that returns the whole deployment. - if (!objectId) return []; - filter.userId = objectId; - } - - const docs = await this._findDrained('Integration', filter); - - return docs.map((doc) => { - const errors = this._extractErrors(doc); - return { - id: fromObjectId(doc?._id), - type: doc?.config?.type ?? null, - status: doc?.status ?? null, - userId: fromObjectId(doc?.userId) ?? null, - version: doc?.version ?? null, - errorCount: Array.isArray(errors) ? errors.length : 0, - moduleCount: Array.isArray(doc?.entityIds) - ? doc.entityIds.length - : 0, - createdAt: doc?.createdAt ?? null, - updatedAt: doc?.updatedAt ?? null, - }; - }); - } - - async countMappingsByIntegrationIds(ids = []) { - const counts = new Map(); - if (!ids || ids.length === 0) return counts; - - // IntegrationMapping.integrationId is stored as a string in DocumentDB, - // so match by string — an ObjectId $in would never match (always 0). - const stringIds = ids.map(String); - - const rows = await this._aggregateDrained('IntegrationMapping', [ - { $match: { integrationId: { $in: stringIds } } }, - { $group: { _id: '$integrationId', count: { $sum: 1 } } }, - ]); - - for (const row of rows) { - counts.set(String(row?._id), row?.count ?? 0); - } - return counts; - } - - async _findDrained(collection, filter) { - const first = await this.prisma.$runCommandRaw({ - find: collection, - filter, - batchSize: DRAIN_BATCH_SIZE, - }); - return this._drain(collection, first); - } - - async _aggregateDrained(collection, pipeline) { - const first = await this.prisma.$runCommandRaw({ - aggregate: collection, - pipeline, - cursor: { batchSize: DRAIN_BATCH_SIZE }, - }); - return this._drain(collection, first); - } - - async _drain(collection, firstResult) { - const cursor = firstResult?.cursor || {}; - const docs = [...(cursor.firstBatch || [])]; - let cursorId = cursor.id; - let batches = 0; - - while (this._cursorOpen(cursorId) && batches < MAX_BATCHES) { - batches += 1; - const next = await this.prisma.$runCommandRaw({ - getMore: cursorId, - collection, - batchSize: DRAIN_BATCH_SIZE, - }); - const nextCursor = next?.cursor || {}; - const nextBatch = nextCursor.nextBatch || []; - docs.push(...nextBatch); - cursorId = nextCursor.id; - if (nextBatch.length === 0) break; - } - return docs; - } - - _cursorOpen(id) { - if (id === undefined || id === null) return false; - if (typeof id === 'number') return id !== 0; - if (typeof id === 'bigint') return id !== 0n; - // Extended JSON can surface a 64-bit cursor id as { $numberLong: "..." }. - if (typeof id === 'object' && id.$numberLong !== undefined) { - return id.$numberLong !== '0'; - } - return String(id) !== '0'; - } - - _extractErrors(doc) { - if (Array.isArray(doc?.errors)) return doc.errors; - if (Array.isArray(doc?.messages?.errors)) return doc.messages.errors; - return []; - } -} - -module.exports = { ReportingRepositoryDocumentDB }; diff --git a/packages/core/reporting/repositories/reporting-repository-documentdb.test.js b/packages/core/reporting/repositories/reporting-repository-documentdb.test.js deleted file mode 100644 index 86b17e634..000000000 --- a/packages/core/reporting/repositories/reporting-repository-documentdb.test.js +++ /dev/null @@ -1,106 +0,0 @@ -const mockRunCommandRaw = jest.fn(); - -jest.mock('../../database/prisma', () => ({ - prisma: { $runCommandRaw: mockRunCommandRaw }, -})); -jest.mock('../../database/documentdb-utils', () => ({ - toObjectId: jest.fn((v) => (v == null || v === '' ? undefined : `oid:${v}`)), - fromObjectId: jest.fn((v) => - typeof v === 'string' && v.startsWith('oid:') ? v.slice(4) : v - ), -})); - -const { - ReportingRepositoryDocumentDB, -} = require('./reporting-repository-documentdb'); - -// Single-batch cursor result (id 0 → exhausted, no getMore). -const singleBatch = (firstBatch) => ({ cursor: { firstBatch, id: 0 } }); - -describe('ReportingRepositoryDocumentDB', () => { - let repo; - - beforeEach(() => { - jest.clearAllMocks(); - repo = new ReportingRepositoryDocumentDB(); - }); - - it('filters by status/userId and derives moduleCount from entityIds', async () => { - mockRunCommandRaw.mockResolvedValueOnce( - singleBatch([ - { - _id: 'i1', - config: { type: 'hubspot' }, - status: 'ENABLED', - userId: 'u1', - version: '1', - errors: [{ title: 'boom' }], - entityIds: ['e1', 'e2', 'e3'], - createdAt: null, - updatedAt: null, - }, - ]) - ); - - const rows = await repo.findIntegrationsForReport({ - status: 'ENABLED', - userId: 'u1', - }); - - const command = mockRunCommandRaw.mock.calls[0][0]; - expect(command.find).toBe('Integration'); - expect(command.filter).toEqual({ status: 'ENABLED', userId: 'oid:u1' }); - expect(rows[0]).toMatchObject({ - id: 'i1', - type: 'hubspot', - moduleCount: 3, - errorCount: 1, - }); - }); - - it('returns an empty list (does not query) when userId is not a valid id', async () => { - const rows = await repo.findIntegrationsForReport({ userId: '' }); - expect(rows).toEqual([]); - expect(mockRunCommandRaw).not.toHaveBeenCalled(); - }); - - it('drains the cursor across multiple batches (does not stop at firstBatch)', async () => { - mockRunCommandRaw - .mockResolvedValueOnce({ - cursor: { - firstBatch: [{ _id: 'i1', entityIds: [] }], - id: { $numberLong: '42' }, - }, - }) - .mockResolvedValueOnce({ - cursor: { nextBatch: [{ _id: 'i2', entityIds: [] }], id: 0 }, - }); - - const rows = await repo.findIntegrationsForReport({}); - - expect(rows.map((r) => r.id)).toEqual(['i1', 'i2']); - // first command = find, second command = getMore on the open cursor - expect(mockRunCommandRaw.mock.calls[0][0].find).toBe('Integration'); - const getMore = mockRunCommandRaw.mock.calls[1][0]; - expect(getMore.getMore).toEqual({ $numberLong: '42' }); - expect(getMore.collection).toBe('Integration'); - }); - - it('counts mappings by matching integrationId as a STRING via aggregation', async () => { - mockRunCommandRaw.mockResolvedValueOnce( - singleBatch([{ _id: 'i1', count: 9 }]) - ); - - const counts = await repo.countMappingsByIntegrationIds(['i1']); - - const command = mockRunCommandRaw.mock.calls[0][0]; - expect(command.aggregate).toBe('IntegrationMapping'); - expect(command.pipeline[0]).toEqual({ - $match: { integrationId: { $in: ['i1'] } }, - }); - expect(command.pipeline[1]).toEqual({ - $group: { _id: '$integrationId', count: { $sum: 1 } }, - }); - expect(counts.get('i1')).toBe(9); - }); -}); diff --git a/packages/core/reporting/repositories/reporting-repository-factory.js b/packages/core/reporting/repositories/reporting-repository-factory.js deleted file mode 100644 index 107a9aeb8..000000000 --- a/packages/core/reporting/repositories/reporting-repository-factory.js +++ /dev/null @@ -1,35 +0,0 @@ -const { ReportingRepositoryMongo } = require('./reporting-repository-mongo'); -const { - ReportingRepositoryPostgres, -} = require('./reporting-repository-postgres'); -const { - ReportingRepositoryDocumentDB, -} = require('./reporting-repository-documentdb'); -const config = require('../../database/config'); - -function createReportingRepository() { - const dbType = config.DB_TYPE; - - switch (dbType) { - case 'mongodb': - return new ReportingRepositoryMongo(); - - case 'postgresql': - return new ReportingRepositoryPostgres(); - - case 'documentdb': - return new ReportingRepositoryDocumentDB(); - - default: - throw new Error( - `Unsupported database type: ${dbType}. Supported values: 'mongodb', 'documentdb', 'postgresql'` - ); - } -} - -module.exports = { - createReportingRepository, - ReportingRepositoryMongo, - ReportingRepositoryPostgres, - ReportingRepositoryDocumentDB, -}; diff --git a/packages/core/reporting/repositories/reporting-repository-interface.js b/packages/core/reporting/repositories/reporting-repository-interface.js deleted file mode 100644 index b314f44c9..000000000 --- a/packages/core/reporting/repositories/reporting-repository-interface.js +++ /dev/null @@ -1,16 +0,0 @@ -class ReportingRepositoryInterface { - // returns: [{ id, type, status, userId, version, errorCount, moduleCount, createdAt, updatedAt }] - async findIntegrationsForReport(filter) { - throw new Error( - 'Method findIntegrationsForReport must be implemented by subclass' - ); - } - - async countMappingsByIntegrationIds(ids) { - throw new Error( - 'Method countMappingsByIntegrationIds must be implemented by subclass' - ); - } -} - -module.exports = { ReportingRepositoryInterface }; diff --git a/packages/core/reporting/repositories/reporting-repository-interface.test.js b/packages/core/reporting/repositories/reporting-repository-interface.test.js deleted file mode 100644 index d4f11c9b0..000000000 --- a/packages/core/reporting/repositories/reporting-repository-interface.test.js +++ /dev/null @@ -1,19 +0,0 @@ -const { - ReportingRepositoryInterface, -} = require('./reporting-repository-interface'); - -describe('ReportingRepositoryInterface', () => { - const repo = new ReportingRepositoryInterface(); - - it('findIntegrationsForReport is abstract', async () => { - await expect(repo.findIntegrationsForReport({})).rejects.toThrow( - /must be implemented by subclass/ - ); - }); - - it('countMappingsByIntegrationIds is abstract', async () => { - await expect(repo.countMappingsByIntegrationIds([])).rejects.toThrow( - /must be implemented by subclass/ - ); - }); -}); diff --git a/packages/core/reporting/repositories/reporting-repository-mongo.js b/packages/core/reporting/repositories/reporting-repository-mongo.js deleted file mode 100644 index fa1054c96..000000000 --- a/packages/core/reporting/repositories/reporting-repository-mongo.js +++ /dev/null @@ -1,54 +0,0 @@ -const { prisma } = require('../../database/prisma'); -const { - ReportingRepositoryInterface, -} = require('./reporting-repository-interface'); - -class ReportingRepositoryMongo extends ReportingRepositoryInterface { - constructor() { - super(); - this.prisma = prisma; - } - - async findIntegrationsForReport({ status, userId } = {}) { - const where = {}; - if (status) where.status = status; - if (userId !== undefined && userId !== null) where.userId = userId; - - const integrations = await this.prisma.integration.findMany({ - where, - include: { entities: { select: { id: true } } }, - }); - - return integrations.map((integration) => ({ - id: integration.id, - type: integration.config?.type ?? null, - status: integration.status ?? null, - userId: integration.userId ?? null, - version: integration.version ?? null, - errorCount: Array.isArray(integration.errors) - ? integration.errors.length - : 0, - moduleCount: integration.entities?.length ?? 0, - createdAt: integration.createdAt ?? null, - updatedAt: integration.updatedAt ?? null, - })); - } - - async countMappingsByIntegrationIds(ids = []) { - const counts = new Map(); - if (!ids || ids.length === 0) return counts; - - const groups = await this.prisma.integrationMapping.groupBy({ - by: ['integrationId'], - where: { integrationId: { in: ids } }, - _count: { _all: true }, - }); - - for (const group of groups) { - counts.set(group.integrationId, group._count._all); - } - return counts; - } -} - -module.exports = { ReportingRepositoryMongo }; diff --git a/packages/core/reporting/repositories/reporting-repository-mongo.test.js b/packages/core/reporting/repositories/reporting-repository-mongo.test.js deleted file mode 100644 index f2257946c..000000000 --- a/packages/core/reporting/repositories/reporting-repository-mongo.test.js +++ /dev/null @@ -1,64 +0,0 @@ -jest.mock('../../database/prisma', () => ({ - prisma: { - integration: { findMany: jest.fn() }, - integrationMapping: { groupBy: jest.fn() }, - }, -})); - -const { prisma } = require('../../database/prisma'); -const { ReportingRepositoryMongo } = require('./reporting-repository-mongo'); - -describe('ReportingRepositoryMongo', () => { - let repo; - - beforeEach(() => { - jest.clearAllMocks(); - repo = new ReportingRepositoryMongo(); - }); - - it('uses string ids in the where clause and maps rows', async () => { - prisma.integration.findMany.mockResolvedValue([ - { - id: 'abc', - config: { type: 'salesforce' }, - status: 'ERROR', - userId: 'u1', - version: '1', - errors: [{}, {}], - entities: [{ id: 'e1' }], - createdAt: null, - updatedAt: null, - }, - ]); - - const rows = await repo.findIntegrationsForReport({ - status: 'ERROR', - userId: 'u1', - }); - - expect(prisma.integration.findMany.mock.calls[0][0].where).toEqual({ - status: 'ERROR', - userId: 'u1', - }); - expect(rows[0]).toMatchObject({ - id: 'abc', - type: 'salesforce', - status: 'ERROR', - moduleCount: 1, - errorCount: 2, - }); - }); - - it('keys the mapping-count map by string id', async () => { - prisma.integrationMapping.groupBy.mockResolvedValue([ - { integrationId: 'abc', _count: { _all: 7 } }, - ]); - - const counts = await repo.countMappingsByIntegrationIds(['abc']); - - expect(prisma.integrationMapping.groupBy.mock.calls[0][0].where).toEqual( - { integrationId: { in: ['abc'] } } - ); - expect(counts.get('abc')).toBe(7); - }); -}); diff --git a/packages/core/reporting/repositories/reporting-repository-postgres.js b/packages/core/reporting/repositories/reporting-repository-postgres.js deleted file mode 100644 index edd96e6a9..000000000 --- a/packages/core/reporting/repositories/reporting-repository-postgres.js +++ /dev/null @@ -1,70 +0,0 @@ -const { prisma } = require('../../database/prisma'); -const { - ReportingRepositoryInterface, -} = require('./reporting-repository-interface'); - -class ReportingRepositoryPostgres extends ReportingRepositoryInterface { - constructor() { - super(); - this.prisma = prisma; - } - - async findIntegrationsForReport({ status, userId } = {}) { - const where = {}; - if (status) where.status = status; - if (userId !== undefined && userId !== null) { - where.userId = this._convertId(userId); - } - - const integrations = await this.prisma.integration.findMany({ - where, - include: { entities: { select: { id: true } } }, - }); - - return integrations.map((integration) => ({ - id: integration.id?.toString(), - type: integration.config?.type ?? null, - status: integration.status ?? null, - userId: integration.userId?.toString() ?? null, - version: integration.version ?? null, - errorCount: Array.isArray(integration.errors) - ? integration.errors.length - : 0, - moduleCount: integration.entities?.length ?? 0, - createdAt: integration.createdAt ?? null, - updatedAt: integration.updatedAt ?? null, - })); - } - - async countMappingsByIntegrationIds(ids = []) { - const counts = new Map(); - if (!ids || ids.length === 0) return counts; - - const intIds = ids.map((id) => this._convertId(id)); - const groups = await this.prisma.integrationMapping.groupBy({ - by: ['integrationId'], - where: { integrationId: { in: intIds } }, - _count: { _all: true }, - }); - - for (const group of groups) { - counts.set(group.integrationId?.toString(), group._count._all); - } - return counts; - } - - _convertId(id) { - if (id === null || id === undefined) return id; - // Reject anything that isn't an exact integer — parseInt would coerce - // '12abc'/'12.9' to 12 and return the wrong record. - const str = String(id).trim(); - if (!/^-?\d+$/.test(str)) { - throw new TypeError( - `Invalid ID: ${id} cannot be converted to integer` - ); - } - return Number.parseInt(str, 10); - } -} - -module.exports = { ReportingRepositoryPostgres }; diff --git a/packages/core/reporting/repositories/reporting-repository-postgres.test.js b/packages/core/reporting/repositories/reporting-repository-postgres.test.js deleted file mode 100644 index c02f863eb..000000000 --- a/packages/core/reporting/repositories/reporting-repository-postgres.test.js +++ /dev/null @@ -1,109 +0,0 @@ -jest.mock('../../database/prisma', () => ({ - prisma: { - integration: { findMany: jest.fn() }, - integrationMapping: { groupBy: jest.fn() }, - }, -})); - -const { prisma } = require('../../database/prisma'); -const { - ReportingRepositoryPostgres, -} = require('./reporting-repository-postgres'); - -describe('ReportingRepositoryPostgres', () => { - let repo; - - beforeEach(() => { - jest.clearAllMocks(); - repo = new ReportingRepositoryPostgres(); - }); - - it('applies status/userId to the where clause and maps rows (only scalar + entity ids)', async () => { - prisma.integration.findMany.mockResolvedValue([ - { - id: 17, - config: { type: 'hubspot' }, - status: 'ENABLED', - userId: 3, - version: '1.0.0', - errors: [], - entities: [{ id: 1 }, { id: 2 }], - createdAt: new Date('2026-01-01T00:00:00Z'), - updatedAt: new Date('2026-01-02T00:00:00Z'), - }, - ]); - - const rows = await repo.findIntegrationsForReport({ - status: 'ENABLED', - userId: '3', - }); - - const callArg = prisma.integration.findMany.mock.calls[0][0]; - expect(callArg.where).toEqual({ status: 'ENABLED', userId: 3 }); - expect(callArg.include).toEqual({ entities: { select: { id: true } } }); - expect(rows[0]).toMatchObject({ - id: '17', - type: 'hubspot', - status: 'ENABLED', - userId: '3', - moduleCount: 2, - errorCount: 0, - }); - }); - - it('counts errors from the errors array', async () => { - prisma.integration.findMany.mockResolvedValue([ - { - id: 1, - config: { type: 'x' }, - status: 'ERROR', - errors: [{ title: 'a' }, { title: 'b' }, { title: 'c' }], - entities: [], - }, - ]); - const rows = await repo.findIntegrationsForReport({}); - expect(rows[0].errorCount).toBe(3); - expect(rows[0].moduleCount).toBe(0); - }); - - it('groups mapping counts by integrationId into a Map keyed by string id', async () => { - prisma.integrationMapping.groupBy.mockResolvedValue([ - { integrationId: 17, _count: { _all: 412 } }, - ]); - - const counts = await repo.countMappingsByIntegrationIds(['17']); - - const arg = prisma.integrationMapping.groupBy.mock.calls[0][0]; - expect(arg.by).toEqual(['integrationId']); - expect(arg.where).toEqual({ integrationId: { in: [17] } }); - expect(counts.get('17')).toBe(412); - }); - - it('returns an empty map and skips the query for no ids', async () => { - const counts = await repo.countMappingsByIntegrationIds([]); - expect(counts.size).toBe(0); - expect(prisma.integrationMapping.groupBy).not.toHaveBeenCalled(); - }); - - it('throws on a non-numeric userId filter', async () => { - await expect( - repo.findIntegrationsForReport({ userId: 'abc' }) - ).rejects.toThrow(/Invalid ID/); - }); - - it('rejects partially-numeric / decimal userId instead of coercing it', async () => { - await expect( - repo.findIntegrationsForReport({ userId: '12abc' }) - ).rejects.toThrow(/Invalid ID/); - await expect( - repo.findIntegrationsForReport({ userId: '12.9' }) - ).rejects.toThrow(/Invalid ID/); - expect(prisma.integration.findMany).not.toHaveBeenCalled(); - }); - - it('throws on a non-numeric id in countMappingsByIntegrationIds', async () => { - await expect( - repo.countMappingsByIntegrationIds(['abc']) - ).rejects.toThrow(/Invalid ID/); - }); -}); diff --git a/packages/core/reporting/use-cases/index.js b/packages/core/reporting/use-cases/index.js deleted file mode 100644 index f8ce71064..000000000 --- a/packages/core/reporting/use-cases/index.js +++ /dev/null @@ -1,6 +0,0 @@ -const { - ListIntegrationsReport, - SCHEMA_VERSION, -} = require('./list-integrations-report'); - -module.exports = { ListIntegrationsReport, SCHEMA_VERSION }; diff --git a/packages/core/reporting/use-cases/list-integrations-report.test.js b/packages/core/reporting/use-cases/list-integrations-report.test.js deleted file mode 100644 index c39bb397f..000000000 --- a/packages/core/reporting/use-cases/list-integrations-report.test.js +++ /dev/null @@ -1,435 +0,0 @@ -const { ListIntegrationsReport } = require('./list-integrations-report'); - -const makeRepo = (rows, mappingCounts = new Map()) => ({ - findIntegrationsForReport: jest.fn().mockResolvedValue(rows), - countMappingsByIntegrationIds: jest.fn().mockResolvedValue(mappingCounts), -}); - -const noUsage = () => ({ - getTotalsByDimension: jest.fn().mockResolvedValue([]), -}); - -describe('ListIntegrationsReport — usage columns', () => { - const rows = [ - { id: '1', type: 'hubspot', status: 'ENABLED', userId: 'u1' }, - { id: '2', type: 'salesforce', status: 'ENABLED', userId: 'u2' }, - ]; - - it('enriches byType with usage columns read from the usage store', async () => { - const reportingRepository = makeRepo(rows); - const usageRepository = { - getTotalsByDimension: jest.fn(async ({ metric }) => { - if (metric === 'records.synced') { - return [{ integrationType: 'hubspot', value: 42 }]; - } - return []; - }), - }; - const useCase = new ListIntegrationsReport({ - reportingRepository, - usageRepository, - }); - - const result = await useCase.execute(); - - const hubspot = result.metrics.byType.find((t) => t.type === 'hubspot'); - const salesforce = result.metrics.byType.find( - (t) => t.type === 'salesforce' - ); - expect(hubspot.usage['records.synced']).toBe(42); - // absent usage reads as 0, not undefined - expect(salesforce.usage['records.synced']).toBe(0); - expect(hubspot.usage['webhooks.received']).toBe(0); - expect(usageRepository.getTotalsByDimension).toHaveBeenCalledWith( - expect.objectContaining({ - metric: 'records.synced', - groupBy: 'integrationType', - }) - ); - }); - - it('never lets a usage-store failure break the structural report', async () => { - const usageRepository = { - getTotalsByDimension: jest - .fn() - .mockRejectedValue(new Error('usage db down')), - }; - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository, - }); - - const result = await useCase.execute(); - expect(result.metrics.total).toBe(2); - // structural report intact; usage simply absent/empty - }); -}); - -describe('ListIntegrationsReport', () => { - it('requires a reportingRepository', () => { - expect(() => new ListIntegrationsReport({})).toThrow( - /reportingRepository is required/ - ); - }); - - it('requires a usageRepository', () => { - expect( - () => - new ListIntegrationsReport({ - reportingRepository: makeRepo([]), - }) - ).toThrow(/usageRepository is required/); - }); - - it('rejects an unknown status with a 400 (Boom) error', async () => { - const repo = makeRepo([]); - const useCase = new ListIntegrationsReport({ - reportingRepository: repo, - usageRepository: noUsage(), - }); - await expect( - useCase.execute({ status: 'BOGUS' }) - ).rejects.toMatchObject({ - isBoom: true, - output: { statusCode: 400 }, - }); - expect(repo.findIntegrationsForReport).not.toHaveBeenCalled(); - }); - - it('rejects a non-string query param with a 400 (Boom) error', async () => { - const repo = makeRepo([]); - const useCase = new ListIntegrationsReport({ - reportingRepository: repo, - usageRepository: noUsage(), - }); - await expect( - useCase.execute({ userId: { $oid: 'x' } }) - ).rejects.toMatchObject({ isBoom: true, output: { statusCode: 400 } }); - expect(repo.findIntegrationsForReport).not.toHaveBeenCalled(); - }); - - it('builds the versioned envelope with totals, byStatus, byType and per-row counts', async () => { - const rows = [ - { - id: '1', - type: 'hubspot', - status: 'ENABLED', - userId: '3', - version: '1.0.0', - moduleCount: 2, - errorCount: 0, - createdAt: new Date('2026-01-01T00:00:00Z'), - updatedAt: new Date('2026-01-02T00:00:00Z'), - }, - { - id: '2', - type: 'hubspot', - status: 'ERROR', - userId: '3', - version: '1.0.0', - moduleCount: 1, - errorCount: 3, - }, - { - id: '3', - type: 'salesforce', - status: 'ENABLED', - userId: '4', - version: '2.0.0', - moduleCount: 1, - errorCount: 0, - }, - ]; - const counts = new Map([ - ['1', 412], - ['2', 5], - ]); - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows, counts), - usageRepository: noUsage(), - }); - - const out = await useCase.execute({}); - - expect(out.schemaVersion).toBe(1); - expect(out.service).toBe('frigg-core-api'); - expect(typeof out.generatedAt).toBe('string'); - expect(out.metrics.total).toBe(3); - expect(out.metrics.byStatus).toEqual({ - ENABLED: 2, - ERROR: 1, - NEEDS_CONFIG: 0, - PROCESSING: 0, - IN_CREATION: 0, - IN_DELETION: 0, - DISABLED: 0, - }); - - const hub = out.metrics.byType.find((t) => t.type === 'hubspot'); - expect(hub.total).toBe(2); - expect(hub.byStatus).toEqual({ - ENABLED: 1, - ERROR: 1, - NEEDS_CONFIG: 0, - PROCESSING: 0, - IN_CREATION: 0, - IN_DELETION: 0, - DISABLED: 0, - }); - - const row1 = out.metrics.integrations.find((i) => i.id === '1'); - expect(row1.mappedRecordCount).toBe(412); - expect(row1.moduleCount).toBe(2); - expect(row1.createdAt).toBe('2026-01-01T00:00:00.000Z'); - - const row3 = out.metrics.integrations.find((i) => i.id === '3'); - expect(row3.mappedRecordCount).toBe(0); - expect(row3.createdAt).toBeNull(); - }); - - it('filters by type in the use-case and only counts mappings for matching ids', async () => { - const rows = [ - { - id: '1', - type: 'hubspot', - status: 'ENABLED', - moduleCount: 1, - errorCount: 0, - }, - { - id: '2', - type: 'salesforce', - status: 'ENABLED', - moduleCount: 1, - errorCount: 0, - }, - ]; - const repo = makeRepo(rows, new Map([['1', 10]])); - const useCase = new ListIntegrationsReport({ - reportingRepository: repo, - usageRepository: noUsage(), - }); - - const out = await useCase.execute({ type: 'hubspot' }); - - expect(out.metrics.total).toBe(1); - expect(out.metrics.integrations[0].id).toBe('1'); - expect(repo.countMappingsByIntegrationIds).toHaveBeenCalledWith(['1']); - expect(out.filters.type).toBe('hubspot'); - }); - - it('passes status/userId to the repository, echoes filters, and skips mapping count when empty', async () => { - const repo = makeRepo([]); - const useCase = new ListIntegrationsReport({ - reportingRepository: repo, - usageRepository: noUsage(), - }); - - const out = await useCase.execute({ status: 'ERROR', userId: '7' }); - - expect(repo.findIntegrationsForReport).toHaveBeenCalledWith({ - status: 'ERROR', - userId: '7', - }); - expect(out.metrics.total).toBe(0); - expect(out.filters).toEqual({ - status: 'ERROR', - type: null, - userId: '7', - }); - expect(repo.countMappingsByIntegrationIds).not.toHaveBeenCalled(); - }); - - it('buckets null type as "unknown"', async () => { - const rows = [ - { - id: '1', - type: null, - status: 'ENABLED', - moduleCount: 0, - errorCount: 0, - }, - ]; - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - }); - - const out = await useCase.execute({}); - - expect(out.metrics.integrations[0].type).toBe('unknown'); - expect(out.metrics.byType[0].type).toBe('unknown'); - }); - - it('picks up an unknown status value dynamically (new enum member)', async () => { - const rows = [ - { - id: '1', - type: 'x', - status: 'ARCHIVED', - moduleCount: 0, - errorCount: 0, - }, - ]; - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - }); - - const out = await useCase.execute({}); - - expect(out.metrics.byStatus.ARCHIVED).toBe(1); - expect(out.metrics.byStatus.ENABLED).toBe(0); - }); - - it('buckets a missing status under UNKNOWN, never a literal "null" key', async () => { - const rows = [ - { id: '1', type: 'x', status: null, moduleCount: 0, errorCount: 0 }, - ]; - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - }); - - const out = await useCase.execute({}); - - expect(out.metrics.byStatus.UNKNOWN).toBe(1); - expect(out.metrics.byStatus).not.toHaveProperty('null'); - expect(out.metrics.byType[0].byStatus.UNKNOWN).toBe(1); - expect(out.metrics.byType[0].byStatus).not.toHaveProperty('null'); - }); - - it('asserts byType buckets are independent across types', async () => { - const rows = [ - { - id: '1', - type: 'hubspot', - status: 'ENABLED', - moduleCount: 1, - errorCount: 0, - }, - { - id: '2', - type: 'salesforce', - status: 'ERROR', - moduleCount: 1, - errorCount: 1, - }, - ]; - const out = await new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - }).execute({}); - - expect(out.metrics.byType).toHaveLength(2); - const sf = out.metrics.byType.find((t) => t.type === 'salesforce'); - expect(sf.total).toBe(1); - expect(sf.byStatus).toEqual({ - ENABLED: 0, - ERROR: 1, - NEEDS_CONFIG: 0, - PROCESSING: 0, - IN_CREATION: 0, - IN_DELETION: 0, - DISABLED: 0, - }); - }); - - it('normalizes timestamps from Date, extended-JSON {$date}, string, and null', async () => { - const rows = [ - { - id: '1', - type: 'x', - status: 'ENABLED', - moduleCount: 0, - errorCount: 0, - createdAt: { $date: '2026-03-04T05:06:07Z' }, - updatedAt: { $date: 'not-a-date' }, - }, - { - id: '2', - type: 'x', - status: 'ENABLED', - moduleCount: 0, - errorCount: 0, - createdAt: '2026-03-04T05:06:07.000Z', - updatedAt: null, - }, - ]; - const out = await new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - }).execute({}); - - const r1 = out.metrics.integrations.find((i) => i.id === '1'); - expect(r1.createdAt).toBe('2026-03-04T05:06:07.000Z'); - expect(r1.updatedAt).toBeNull(); - const r2 = out.metrics.integrations.find((i) => i.id === '2'); - expect(r2.createdAt).toBe('2026-03-04T05:06:07.000Z'); - expect(r2.updatedAt).toBeNull(); - }); - - it('labels byType buckets from the typeLabels map and exposes the map', async () => { - const rows = [ - { - id: '1', - type: 'hubspot', - status: 'ENABLED', - moduleCount: 1, - errorCount: 0, - }, - { - id: '2', - type: 'salesforce', - status: 'ENABLED', - moduleCount: 1, - errorCount: 0, - }, - ]; - const typeLabels = { hubspot: 'HubSpot CRM', salesforce: 'Salesforce' }; - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - typeLabels, - }); - - const out = await useCase.execute({}); - - const hub = out.metrics.byType.find((t) => t.type === 'hubspot'); - expect(hub.label).toBe('HubSpot CRM'); - const sf = out.metrics.byType.find((t) => t.type === 'salesforce'); - expect(sf.label).toBe('Salesforce'); - expect(out.metrics.typeLabels).toEqual(typeLabels); - }); - - it('falls back to the slug when a type has no label, and defaults typeLabels to {}', async () => { - const rows = [ - { - id: '1', - type: 'hubspot', - status: 'ENABLED', - moduleCount: 1, - errorCount: 0, - }, - { - id: '2', - type: null, - status: 'ENABLED', - moduleCount: 0, - errorCount: 0, - }, - ]; - const useCase = new ListIntegrationsReport({ - reportingRepository: makeRepo(rows), - usageRepository: noUsage(), - }); - - const out = await useCase.execute({}); - - const hub = out.metrics.byType.find((t) => t.type === 'hubspot'); - expect(hub.label).toBe('hubspot'); - const unknown = out.metrics.byType.find((t) => t.type === 'unknown'); - expect(unknown.label).toBe('unknown'); - expect(out.metrics.typeLabels).toEqual({}); - }); -}); diff --git a/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.js b/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.js index 07df6595f..e5e0d2860 100644 --- a/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.js +++ b/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.js @@ -23,7 +23,15 @@ class AdminScriptBuilder extends InfrastructureBuilder { } shouldExecute(appDefinition) { - return Array.isArray(appDefinition.adminScripts) && appDefinition.adminScripts.length > 0; + const hasScripts = + Array.isArray(appDefinition.adminScripts) && + appDefinition.adminScripts.length > 0; + const hasReports = + Array.isArray(appDefinition.reports) && + appDefinition.reports.length > 0; + const hasBuiltinReports = + appDefinition.admin?.includeBuiltinReports === true; + return hasScripts || hasReports || hasBuiltinReports; } getDependencies() { @@ -33,31 +41,53 @@ class AdminScriptBuilder extends InfrastructureBuilder { validate(appDefinition) { const result = new ValidationResult(); - if (!appDefinition.adminScripts) { - return result; // Not an error, just no scripts - } - - if (!Array.isArray(appDefinition.adminScripts)) { - result.addError('adminScripts must be an array'); - return result; + if (appDefinition.adminScripts !== undefined) { + if (!Array.isArray(appDefinition.adminScripts)) { + result.addError('adminScripts must be an array'); + } else { + appDefinition.adminScripts.forEach((script, index) => { + if (!script?.Definition?.name) { + result.addError(`Admin script at index ${index} is missing Definition or name`); + } + }); + } } - // Validate each script - appDefinition.adminScripts.forEach((script, index) => { - if (!script?.Definition?.name) { - result.addError(`Admin script at index ${index} is missing Definition or name`); + if (appDefinition.reports !== undefined) { + if (!Array.isArray(appDefinition.reports)) { + result.addError('reports must be an array'); + } else { + appDefinition.reports.forEach((report, index) => { + if (!report?.Definition?.name) { + result.addError(`Report at index ${index} is missing Definition or name`); + } + }); } - }); + } return result; } async build(appDefinition, discoveredResources) { - console.log(`\n[${this.name}] Configuring admin scripts...`); - console.log(` Processing ${appDefinition.adminScripts.length} scripts...`); + console.log(`\n[${this.name}] Configuring admin operations...`); const usePrismaLayer = appDefinition.usePrismaLambdaLayer !== false; const adminConfig = appDefinition.admin || {}; + const adminScripts = Array.isArray(appDefinition.adminScripts) + ? appDefinition.adminScripts + : []; + const reports = Array.isArray(appDefinition.reports) + ? appDefinition.reports + : []; + const hasReports = + reports.length > 0 || adminConfig.includeBuiltinReports === true; + + // Only non-JSON report output is stored in S3, so provision the bucket + // only for that. Built-in reports emit JSON, so they don't trigger it. + const reportsNeedArtifacts = reports.some((report) => { + const format = report?.Definition?.output?.format; + return Boolean(format) && format !== 'json'; + }); const result = { functions: {}, @@ -67,27 +97,45 @@ class AdminScriptBuilder extends InfrastructureBuilder { iamStatements: [], }; - // Create admin script queue - this.createAdminScriptQueue(result, appDefinition); - - // Create Lambda function for script execution - this.createScriptExecutorFunction(appDefinition, result, usePrismaLayer); + if (adminScripts.length > 0) { + console.log(` Processing ${adminScripts.length} scripts...`); + this.createAdminScriptQueue(result, appDefinition); + this.createScriptExecutorFunction(appDefinition, result, usePrismaLayer); + this.createAdminScriptRoutes(appDefinition, result, usePrismaLayer); - // Create API routes for script management - this.createAdminScriptRoutes(appDefinition, result, usePrismaLayer); + adminScripts.forEach(script => { + const name = script.Definition?.name || 'unknown'; + console.log(` ✓ Registered script: ${name}`); + }); + } - // Phase 2: Create EventBridge Scheduler resources - if (adminConfig.enableScheduling) { - this.createSchedulerResources(appDefinition, result); + if (hasReports) { + this.createReportQueue(result, appDefinition); + this.createReportExecutorFunction(appDefinition, result, usePrismaLayer); + this.createReportRoutes(appDefinition, result, usePrismaLayer); + if (reportsNeedArtifacts) { + this.createReportArtifactBucket(result, appDefinition); + } + reports.forEach(report => { + const name = report.Definition?.name || 'unknown'; + console.log(` ✓ Registered report: ${name}`); + }); + if (adminConfig.includeBuiltinReports) { + console.log(' ✓ Built-in reports enabled'); + } } - // Log registered scripts - appDefinition.adminScripts.forEach(script => { - const name = script.Definition?.name || 'unknown'; - console.log(` ✓ Registered: ${name}`); - }); + if ( + adminConfig.enableScheduling && + (adminScripts.length > 0 || hasReports) + ) { + this.createSchedulerResources(appDefinition, result, { + scriptsPresent: adminScripts.length > 0, + hasReports, + }); + } - console.log(`[${this.name}] ✅ Admin script configuration completed`); + console.log(`[${this.name}] ✅ Admin operations configuration completed`); return result; } @@ -193,6 +241,151 @@ class AdminScriptBuilder extends InfrastructureBuilder { console.log(' ✓ Created adminScriptRouter function'); } + createReportRoutes(appDefinition, result, usePrismaLayer) { + result.functions.reportRouter = { + handler: 'node_modules/@friggframework/admin-scripts/src/infrastructure/report-router.handler', + skipEsbuild: true, + package: this.skipEsbuildPackageConfig(appDefinition, usePrismaLayer), + ...(usePrismaLayer && { layers: [{ Ref: 'PrismaLambdaLayer' }] }), + timeout: 30, + events: [ + { httpApi: { path: '/api/v2/reports', method: 'GET' } }, + // Definition detail, snapshots, executions, schedule, back-compat alias + { httpApi: { path: '/api/v2/reports/{proxy+}', method: 'GET' } }, + // Run a report ({name}/run) + { httpApi: { path: '/api/v2/reports/{proxy+}', method: 'POST' } }, + // Schedule management (PUT/DELETE {name}/schedule) + { httpApi: { path: '/api/v2/reports/{proxy+}', method: 'PUT' } }, + { httpApi: { path: '/api/v2/reports/{proxy+}', method: 'DELETE' } }, + ], + }; + console.log(' ✓ Created reportRouter function'); + } + + createReportQueue(result, appDefinition) { + result.resources.ReportQueue = { + Type: 'AWS::SQS::Queue', + Properties: { + QueueName: '${self:service}-${self:provider.stage}-ReportQueue', + MessageRetentionPeriod: 86400, // 1 day + VisibilityTimeout: 900, // 15 minutes (Lambda max) + RedrivePolicy: { + maxReceiveCount: 3, + deadLetterTargetArn: { + 'Fn::GetAtt': ['InternalErrorQueue', 'Arn'], + }, + }, + }, + }; + + if (isScopedEnvironmentActive(appDefinition)) { + // Only the report functions read this queue URL + result.functionEnvironments = result.functionEnvironments || {}; + for (const fnName of ['reportRouter', 'reportExecutor']) { + result.functionEnvironments[fnName] = { + ...result.functionEnvironments[fnName], + REPORT_QUEUE_URL: { Ref: 'ReportQueue' }, + }; + } + } else { + result.environment.REPORT_QUEUE_URL = { Ref: 'ReportQueue' }; + } + + // The report router enqueues async recorded/snapshot runs. The base + // role's wildcard does not cover this queue's name, so grant + // SendMessage explicitly. + result.iamStatements.push({ + Effect: 'Allow', + Action: [ + 'sqs:SendMessage', + 'sqs:SendMessageBatch', + 'sqs:GetQueueUrl', + 'sqs:GetQueueAttributes', + ], + Resource: { 'Fn::GetAtt': ['ReportQueue', 'Arn'] }, + }); + + console.log(' ✓ Created ReportQueue'); + } + + createReportExecutorFunction(appDefinition, result, usePrismaLayer) { + result.functions.reportExecutor = { + handler: 'node_modules/@friggframework/admin-scripts/src/infrastructure/report-executor-handler.handler', + skipEsbuild: true, + package: this.skipEsbuildPackageConfig(appDefinition, usePrismaLayer), + ...(usePrismaLayer && { layers: [{ Ref: 'PrismaLambdaLayer' }] }), + timeout: 900, // 15 minutes max + memorySize: 1024, + events: [ + { + sqs: { + arn: { 'Fn::GetAtt': ['ReportQueue', 'Arn'] }, + batchSize: 1, + }, + }, + ], + }; + console.log(' ✓ Created reportExecutor function'); + } + + // Non-JSON report output; the router mints short-lived presigned URLs for reads. + createReportArtifactBucket(result, appDefinition) { + result.resources.ReportArtifactBucket = { + Type: 'AWS::S3::Bucket', + Properties: { + BucketName: + '${self:service}-${self:provider.stage}-report-artifacts', + BucketEncryption: { + ServerSideEncryptionConfiguration: [ + { + ServerSideEncryptionByDefault: { + SSEAlgorithm: 'AES256', + }, + }, + ], + }, + PublicAccessBlockConfiguration: { + BlockPublicAcls: true, + BlockPublicPolicy: true, + IgnorePublicAcls: true, + RestrictPublicBuckets: true, + }, + }, + }; + + if (isScopedEnvironmentActive(appDefinition)) { + result.functionEnvironments = result.functionEnvironments || {}; + for (const fnName of ['reportRouter', 'reportExecutor']) { + result.functionEnvironments[fnName] = { + ...result.functionEnvironments[fnName], + REPORT_ARTIFACT_BUCKET: { Ref: 'ReportArtifactBucket' }, + }; + } + } else { + result.environment.REPORT_ARTIFACT_BUCKET = { + Ref: 'ReportArtifactBucket', + }; + } + + // Executor writes and router reads (via presign); scope to this bucket's objects. + result.iamStatements.push({ + Effect: 'Allow', + Action: ['s3:PutObject', 's3:GetObject'], + Resource: { + 'Fn::Sub': [ + '${BucketArn}/*', + { + BucketArn: { + 'Fn::GetAtt': ['ReportArtifactBucket', 'Arn'], + }, + }, + ], + }, + }); + + console.log(' ✓ Created ReportArtifactBucket'); + } + // Without this, the skipEsbuild functions package the whole node_modules // closure (aws-sdk, Prisma, dev deps) and blow past Lambda's 250 MB limit. // Mirrors the exclusions the framework's other node_modules handlers use. @@ -248,13 +441,25 @@ class AdminScriptBuilder extends InfrastructureBuilder { }; } - createSchedulerResources(appDefinition, result) { - // Constructed ARN, not Fn::GetAtt: a GetAtt edge to the executor closes a + createSchedulerResources( + appDefinition, + result, + { scriptsPresent = true, hasReports = false } = {} + ) { + // Constructed ARNs, not Fn::GetAtt: a GetAtt edge to an executor closes a // CloudFormation circular dependency via the shared Lambda execution role. - const executorArn = { - 'Fn::Sub': - 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${self:service}-${self:provider.stage}-adminScriptExecutor', - }; + const fnArn = (logicalName) => ({ + 'Fn::Sub': `arn:aws:lambda:\${AWS::Region}:\${AWS::AccountId}:function:\${self:service}-\${self:provider.stage}-${logicalName}`, + }); + const scriptExecutorArn = fnArn('adminScriptExecutor'); + const reportExecutorArn = fnArn('reportExecutor'); + + const invokeResources = [ + ...(scriptsPresent ? [scriptExecutorArn] : []), + ...(hasReports ? [reportExecutorArn] : []), + ]; + const invokeResource = + invokeResources.length === 1 ? invokeResources[0] : invokeResources; // Create IAM role for EventBridge Scheduler result.resources.AdminScriptSchedulerRole = { @@ -276,7 +481,7 @@ class AdminScriptBuilder extends InfrastructureBuilder { Statement: [{ Effect: 'Allow', Action: 'lambda:InvokeFunction', - Resource: executorArn, + Resource: invokeResource, }], }, }], @@ -293,18 +498,35 @@ class AdminScriptBuilder extends InfrastructureBuilder { // Router-scoped, not shared provider env. Two reasons: broadcasting the // resource references to every function creates CloudFormation circular - // deps; and SCHEDULER_PROVIDER='aws' is only valid for the admin-script - // adapter (the router's sole consumer) — core's scheduler factory, used - // by integration Lambdas, rejects 'aws', so it must not leak app-wide. - result.functions.adminScriptRouter.environment = { - ...(result.functions.adminScriptRouter.environment || {}), - SCHEDULER_PROVIDER: 'aws', - SCHEDULER_ROLE_ARN: { - 'Fn::GetAtt': ['AdminScriptSchedulerRole', 'Arn'], - }, - ADMIN_SCRIPT_SCHEDULE_GROUP: { Ref: 'AdminScriptScheduleGroup' }, - ADMIN_SCRIPT_EXECUTOR_LAMBDA_ARN: executorArn, - }; + // deps; and SCHEDULER_PROVIDER='aws' is only valid for the admin-scripts + // adapter (the routers are its sole consumers) — core's scheduler + // factory, used by integration Lambdas, rejects 'aws', so it must not + // leak app-wide. + if (scriptsPresent) { + result.functions.adminScriptRouter.environment = { + ...(result.functions.adminScriptRouter.environment || {}), + SCHEDULER_PROVIDER: 'aws', + SCHEDULER_ROLE_ARN: { + 'Fn::GetAtt': ['AdminScriptSchedulerRole', 'Arn'], + }, + ADMIN_SCRIPT_SCHEDULE_GROUP: { Ref: 'AdminScriptScheduleGroup' }, + ADMIN_SCRIPT_EXECUTOR_LAMBDA_ARN: scriptExecutorArn, + }; + } + + // Report schedules reuse the shared role/group but must target the + // report executor, not the script one (REPORT_EXECUTOR_LAMBDA_ARN). + if (hasReports) { + result.functions.reportRouter.environment = { + ...(result.functions.reportRouter.environment || {}), + SCHEDULER_PROVIDER: 'aws', + SCHEDULER_ROLE_ARN: { + 'Fn::GetAtt': ['AdminScriptSchedulerRole', 'Arn'], + }, + REPORT_SCHEDULE_GROUP: { Ref: 'AdminScriptScheduleGroup' }, + REPORT_EXECUTOR_LAMBDA_ARN: reportExecutorArn, + }; + } // The router manages schedules through the AWS scheduler adapter, so it // needs scheduler:* on this group plus iam:PassRole for the role it hands diff --git a/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.test.js b/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.test.js index 652de085a..b0c17039f 100644 --- a/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.test.js +++ b/packages/devtools/infrastructure/domains/admin-scripts/admin-script-builder.test.js @@ -620,6 +620,319 @@ describe('AdminScriptBuilder', () => { }); }); + describe('reports (ReportQueue + reportExecutor)', () => { + it('creates ReportQueue + reportExecutor and wires REPORT_QUEUE_URL when reports are present', async () => { + const appDefinition = { + reports: [{ Definition: { name: 'my-report', version: '1.0.0' } }], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + // Queue + expect(result.resources.ReportQueue).toBeDefined(); + expect(result.resources.ReportQueue.Type).toBe('AWS::SQS::Queue'); + expect( + result.resources.ReportQueue.Properties.MessageRetentionPeriod + ).toBe(86400); + expect( + result.resources.ReportQueue.Properties.VisibilityTimeout + ).toBe(900); + expect( + result.resources.ReportQueue.Properties.RedrivePolicy + ).toEqual({ + maxReceiveCount: 3, + deadLetterTargetArn: { + 'Fn::GetAtt': ['InternalErrorQueue', 'Arn'], + }, + }); + + // Executor function + expect(result.functions.reportExecutor).toBeDefined(); + expect(result.functions.reportExecutor.handler).toBe( + 'node_modules/@friggframework/admin-scripts/src/infrastructure/report-executor-handler.handler' + ); + expect(result.functions.reportExecutor.timeout).toBe(900); + expect(result.functions.reportExecutor.memorySize).toBe(1024); + expect(result.functions.reportExecutor.events).toEqual([ + { + sqs: { + arn: { 'Fn::GetAtt': ['ReportQueue', 'Arn'] }, + batchSize: 1, + }, + }, + ]); + expect(result.functions.reportExecutor.skipEsbuild).toBe(true); + expect(result.functions.reportExecutor.layers).toEqual([ + { Ref: 'PrismaLambdaLayer' }, + ]); + + // Env wired app-wide (scoped flag off) + expect(result.environment.REPORT_QUEUE_URL).toEqual({ + Ref: 'ReportQueue', + }); + + // IAM SendMessage grant on the queue Arn + const grant = result.iamStatements.find( + (s) => + Array.isArray(s.Action) && + s.Action.includes('sqs:SendMessage') && + s.Resource && + s.Resource['Fn::GetAtt'] && + s.Resource['Fn::GetAtt'][0] === 'ReportQueue' + ); + expect(grant).toBeDefined(); + expect(grant.Action).toContain('sqs:SendMessageBatch'); + }); + + it('scopes REPORT_QUEUE_URL to reportRouter + reportExecutor when scopedEnvironment is on', async () => { + const appDefinition = { + lambda: { scopedEnvironment: true }, + reports: [{ Definition: { name: 'my-report', version: '1.0.0' } }], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + expect(result.environment.REPORT_QUEUE_URL).toBeUndefined(); + for (const fnName of ['reportRouter', 'reportExecutor']) { + expect( + result.functionEnvironments[fnName].REPORT_QUEUE_URL + ).toEqual({ Ref: 'ReportQueue' }); + } + }); + + it('creates ReportQueue + reportExecutor when only builtin reports are enabled', async () => { + const appDefinition = { + admin: { includeBuiltinReports: true }, + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + expect(result.resources.ReportQueue).toBeDefined(); + expect(result.functions.reportExecutor).toBeDefined(); + }); + + it('does NOT create ReportQueue or reportExecutor when only adminScripts are present', async () => { + const appDefinition = { + adminScripts: [{ Definition: { name: 'test-script' } }], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + expect(result.resources.ReportQueue).toBeUndefined(); + expect(result.functions.reportExecutor).toBeUndefined(); + expect(result.functions.reportRouter).toBeUndefined(); + expect(result.environment.REPORT_QUEUE_URL).toBeUndefined(); + }); + }); + + describe('report artifacts (ReportArtifactBucket for non-JSON output)', () => { + it('provisions a private encrypted bucket + IAM + env when a report emits non-JSON', async () => { + const appDefinition = { + reports: [ + { + Definition: { + name: 'sales-csv', + version: '1.0.0', + output: { format: 'csv' }, + }, + }, + ], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + const bucket = result.resources.ReportArtifactBucket; + expect(bucket).toBeDefined(); + expect(bucket.Type).toBe('AWS::S3::Bucket'); + expect( + bucket.Properties.BucketEncryption + .ServerSideEncryptionConfiguration[0] + .ServerSideEncryptionByDefault.SSEAlgorithm + ).toBe('AES256'); + expect(bucket.Properties.PublicAccessBlockConfiguration).toEqual({ + BlockPublicAcls: true, + BlockPublicPolicy: true, + IgnorePublicAcls: true, + RestrictPublicBuckets: true, + }); + + // Env wired app-wide (scoped flag off). + expect(result.environment.REPORT_ARTIFACT_BUCKET).toEqual({ + Ref: 'ReportArtifactBucket', + }); + + // IAM: object-level Put/Get scoped to the bucket keys. + const grant = result.iamStatements.find( + (s) => + Array.isArray(s.Action) && + s.Action.includes('s3:PutObject') + ); + expect(grant).toBeDefined(); + expect(grant.Action).toContain('s3:GetObject'); + expect(grant.Resource['Fn::Sub'][0]).toBe('${BucketArn}/*'); + expect(grant.Resource['Fn::Sub'][1]).toEqual({ + BucketArn: { 'Fn::GetAtt': ['ReportArtifactBucket', 'Arn'] }, + }); + }); + + it('does NOT provision the bucket for JSON-only reports', async () => { + const appDefinition = { + reports: [ + { + Definition: { + name: 'json-report', + version: '1.0.0', + output: { format: 'json' }, + }, + }, + // No output field defaults to JSON. + { Definition: { name: 'plain', version: '1.0.0' } }, + ], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + expect(result.resources.ReportArtifactBucket).toBeUndefined(); + expect(result.environment.REPORT_ARTIFACT_BUCKET).toBeUndefined(); + const grant = result.iamStatements.find( + (s) => + Array.isArray(s.Action) && + s.Action.includes('s3:PutObject') + ); + expect(grant).toBeUndefined(); + }); + + it('scopes REPORT_ARTIFACT_BUCKET to report functions when scopedEnvironment is on', async () => { + const appDefinition = { + lambda: { scopedEnvironment: true }, + reports: [ + { + Definition: { + name: 'sales-csv', + version: '1.0.0', + output: { format: 'csv' }, + }, + }, + ], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + expect(result.environment.REPORT_ARTIFACT_BUCKET).toBeUndefined(); + for (const fnName of ['reportRouter', 'reportExecutor']) { + expect( + result.functionEnvironments[fnName].REPORT_ARTIFACT_BUCKET + ).toEqual({ Ref: 'ReportArtifactBucket' }); + } + }); + }); + + describe('report scheduling (enableScheduling && reports)', () => { + it('wires the report scheduler env onto reportRouter targeting the report executor', async () => { + const appDefinition = { + reports: [{ Definition: { name: 'my-report', version: '1.0.0' } }], + admin: { enableScheduling: true }, + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + // Shared scheduler role + group are created even without scripts. + expect(result.resources.AdminScriptSchedulerRole).toBeDefined(); + expect(result.resources.AdminScriptScheduleGroup).toBeDefined(); + + const routerEnv = result.functions.reportRouter.environment; + expect(routerEnv.SCHEDULER_PROVIDER).toBe('aws'); + expect(routerEnv.SCHEDULER_ROLE_ARN).toEqual({ + 'Fn::GetAtt': ['AdminScriptSchedulerRole', 'Arn'], + }); + expect(routerEnv.REPORT_SCHEDULE_GROUP).toEqual({ + Ref: 'AdminScriptScheduleGroup', + }); + // Constructed ARN (Fn::Sub), not Fn::GetAtt — avoids a circular dep. + expect(routerEnv.REPORT_EXECUTOR_LAMBDA_ARN).toEqual({ + 'Fn::Sub': + 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${self:service}-${self:provider.stage}-reportExecutor', + }); + + // Must NOT leak onto the shared provider env. + expect(result.environment.SCHEDULER_PROVIDER).toBeUndefined(); + expect(result.environment.REPORT_EXECUTOR_LAMBDA_ARN).toBeUndefined(); + }); + + it('grants the scheduler role invoke on the report executor (reports-only)', async () => { + const appDefinition = { + reports: [{ Definition: { name: 'my-report', version: '1.0.0' } }], + admin: { enableScheduling: true }, + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + const statement = + result.resources.AdminScriptSchedulerRole.Properties.Policies[0] + .PolicyDocument.Statement[0]; + // Only reports present -> single Resource, the report executor ARN. + expect(statement.Resource).toEqual({ + 'Fn::Sub': + 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${self:service}-${self:provider.stage}-reportExecutor', + }); + + // scheduler:* + iam:PassRole grants present. + const schedulerGrant = result.iamStatements.find( + (s) => + Array.isArray(s.Action) && + s.Action.includes('scheduler:CreateSchedule') + ); + expect(schedulerGrant).toBeDefined(); + }); + + it('lets the scheduler role invoke BOTH executors when scripts and reports coexist', async () => { + const appDefinition = { + adminScripts: [{ Definition: { name: 'test-script' } }], + reports: [{ Definition: { name: 'my-report', version: '1.0.0' } }], + admin: { enableScheduling: true }, + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + const statement = + result.resources.AdminScriptSchedulerRole.Properties.Policies[0] + .PolicyDocument.Statement[0]; + expect(statement.Resource).toEqual([ + { + 'Fn::Sub': + 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${self:service}-${self:provider.stage}-adminScriptExecutor', + }, + { + 'Fn::Sub': + 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${self:service}-${self:provider.stage}-reportExecutor', + }, + ]); + + // Both routers get their own scheduler env. + expect( + result.functions.adminScriptRouter.environment + .ADMIN_SCRIPT_EXECUTOR_LAMBDA_ARN + ).toBeDefined(); + expect( + result.functions.reportRouter.environment + .REPORT_EXECUTOR_LAMBDA_ARN + ).toBeDefined(); + }); + + it('does NOT wire report scheduler env when enableScheduling is off', async () => { + const appDefinition = { + reports: [{ Definition: { name: 'my-report', version: '1.0.0' } }], + }; + + const result = await adminScriptBuilder.build(appDefinition, {}); + + expect(result.resources.AdminScriptSchedulerRole).toBeUndefined(); + expect( + result.functions.reportRouter.environment + ).toBeUndefined(); + }); + }); + describe('getName()', () => { it('should return AdminScriptBuilder', () => { expect(adminScriptBuilder.getName()).toBe('AdminScriptBuilder'); diff --git a/packages/devtools/infrastructure/domains/shared/utilities/base-definition-factory.js b/packages/devtools/infrastructure/domains/shared/utilities/base-definition-factory.js index 210b40d5e..3e318a1c6 100644 --- a/packages/devtools/infrastructure/domains/shared/utilities/base-definition-factory.js +++ b/packages/devtools/infrastructure/domains/shared/utilities/base-definition-factory.js @@ -311,16 +311,7 @@ function createBaseDefinition( { httpApi: { path: '/health/{proxy+}', method: 'GET' } }, ], }, - reporting: { - handler: 'node_modules/@friggframework/core/handlers/routers/reporting.handler', - ...(usePrismaLayer && { layers: [{ Ref: 'PrismaLambdaLayer' }] }), - skipEsbuild: true, // Handlers in node_modules don't need bundling - package: skipEsbuildPackageConfig, - events: [ - { httpApi: { path: '/api/v2/reports', method: 'GET' } }, - { httpApi: { path: '/api/v2/reports/{proxy+}', method: 'GET' } }, - ], - }, + // Reporting is an admin operation (ADR-010): the report router runs on the admin-scripts Lambda, not as a standalone function here. // Note: dbMigrate removed - MigrationBuilder now handles migration infrastructure // See: packages/devtools/infrastructure/domains/database/migration-builder.js },